I think it has a lot to do with how society interprets stuttering - if you look at any character that stutters in any fictional story, not one of them *just stutters* (that I know about, anyway). It's always to emphasize another lack, whether it be social ineptitude, a lack of emotional control, or just stupidity. Stuttering seems to imply that to a lot of people, so it can definitely impact confidence if you're constantly broadcasting the message that something's wrong with you.