Trying not to sugar coat anything, but it does and that's what does worry stutterers. Stuttering is usually associated with nervousness, fear and insecurity. Bold, confident, coherent speakers are usually looked upon as if they believe in their message so well they're able to assertively convey it. It's also unprofessional for people to disregard your speech as professional because of your stutter. If you feel that your stutter is having an impact on your speeches, then you could start off your speech by claiming that you have a stutter in a proud, bold tone. It should relieve some of the confusion.