You're right, we absolutely care more about it than everyone else and that is the point. It isn't the perception of others that is the primary concern. What causes distress is the stutter itself. It inhibits the ability to freely, at will, say what one wants to say, when they want to say it. The opinions of others are secondary. Your words have good intentions, but it's invalidating what people truly feel about their stutter.