He sounds insensitive in my opinion. I've never been in your position but you could maybe have a serious discussion with him? I've realised once you educate people on stuttering they start to understand it better and gradually accept it can be uncontrollable sometimes.