The more you socialise the more comfortable it becomes and when we're relaxed in friendly company we stutter less. Just like you'd have no problem hanging out with a blind or deaf person or someone in a wheelchair or someone just learning your language, others don't have a problem with your stutter. Genuine people look through speech barriers to see the person underneath. If you are being invited to socialise it's because people want to get to know you. Why not give them the chance to see the real you, warts and all? That's what friendship is all about.