It sounds like you are trying to hide the fact that you stutter, wich builds up a lot of pressure. I can't speak for you, but for me it would make me stutter worse. Tell him that you have a mild stutter, or make it obvious, so you don't have to worry about he might notice.