It's infuriating, and it diminishes stuttering as a serious topic. I have also been told that it is endearing. People who don't stutter do not have the right or capacity to tell people who have stuttered since they could speak not to worry, because *they* think it is charming or whatever the hell. These people would likely not say the same thing about any other disability.