Not wrong at all, I agree. I just like to acknowledge I'm slightly inconveniencing others due to a condition not many people understand or know how to react to. Now, if the other person is being a dick about it, HELL NO don't apologize. Take longer, actually. And let them know they're in the wrong.