Definitely annoying. Especially when people automatically assume you're nervous because you're stuttering, unaware that you just stutter, nervous or not.