I work on the phone and experience this a lot. People are just thoughtless and ignorant. Over the years I’ve learnt to call people out on it, not be rude or even upset but politely let them know that I stutter. People are usually mortified and apologise profusely. I just want people to be more mindful when speaking to strangers, especially on the phone when you can’t see them. I would suggest calling up and speaking to him or the manager about it, leaving a review or sending an email. Good luck!