Reminding him that you have a disability will put him in his place. Being the bigger person by genuinely and honestly explaining, will make him feel guilty, and that's a lot better than cursing him out or quietly ignoring it. I'd ask to speak with him at some point, and give him the facts. Tell him why you have problems speaking, and if he isn't apologetic and understanding, then consider finding another place to work. That's my best advice for a situation like this. Don't put up with it if he continues or doesn't care.