You get to choose whether your opinion or their opinion is worth more. I think you need to take a lighter approach on your speech. Being hard on yourself makes it worse. It sounds like you have friends that are willing to invite you places so people care for you, just think about that next time you feel overwhelmed or stressed about stuttering. Have you tried attending speech therapy or working on it in your own time?