I had those same feelings. Basically the only people who are annoyed are ass holes and the only people who are inconvenienced are so self centered I don't give a heck what they think. Friends, coworkers, partners, etc should all understand. If the occasional coworker is more pointed in their criticism then you can drop some truth on them in a nice way. I've done that a time or two in the past. I had a professor ask me if I forgot my name when I was stuttering. I stayed after class and talked to him. That was spicey haha