You don’t need to justify the way you feel only your actions that should be justified. As a stutterer myself, I don’t like listening to other stutterers for the same reasons you listed but when I do, I treat them the way that I want to be treated. Just being a good listener and giving them the time they need to talk. It’s not how we feel is what makes us a good person, it’s how we act.