please keep in mind that not all people who stutters think the same thing and one person's response might not be appropriate for another. we are all human beings though, so "Hey man, sorry I made that insensitive comment about forgetting your number. I'll try to be more sensitive in future." should do it.