Don't let it annoy you, just let them know. Usually people are very apologetic, it's not like they encounter stutterers on a daily basis and have a "correct" impulse response ready to go