Totally inappropriate and insensitive. Either he has a problem with your stutter or he is completely oblivious to how inappropriate this is. Did you interview with him? Did your company know you stuttered when they hired you? If so, this is extra not ok. I know you said you're non-confrontational but if you don't address this now you won't address it later. It'll only be more difficult letting this go on and then trying to bring it up. I don't want you to leave it until things get so stressful you want to quit. If you don't feel comfortable talking to your boss directly, is there someone else you can talk to?