Speak your mind. But keep in mind that it sounds like they are trying to be nice, in their own weird way. e.g. "That's nice of you to say, but, NO, you really don't wish you had a stutter. My stutter has been one of the harder things to deal with in my life. I'm not looking for sympathy. Just trust me, you don't want a speech impediment." And before you say anything to them, imagine you're talking to your favorite person in the world. Best friend, grandparent... you get the idea. Adjust your tone and approach as if you were speaking to that loved one. Whatever you say, it's going to come down to how you say it, more than what you say. And be selective. Sometimes it's OK to seem like a jerk, depending who it is.