So I work with somebody else that stutters...

So I work with somebody else that stutters... And I hate it, which makes me feel like a super asshole. I have a mild stutter, most people don't even realize I stutter unless I talk to them for awhile, and then i'll occasionally hang up, but my new coworker has a pretty severe stutter. I think the worst part is that his stutter triggers my stutter. And I pick up his stutter pattern or mannerisms, stuff I never used to do. I mean, he's a good co-worker and I like him, but I hate talking to him because he makes me stutter, then makes me feel like an asshole for hating to talk to him. I've never told him or acted like I hated talking to him, and I'd never, ever treat him differently or badly because of his stutter (and actually defended him when some other coworkers talked shit about him behind his back to me before they knew I stuttered too), but it really bugs me. ... Is anybody else going through this kind of thing? Is this feeling unusual?