Covert stutterers have a huge mental and emotional burden. I don't worry about other people hearing me stutter because I can't hide it and I don't stress too much about most speaking situations. That's different from covert stutterers. If you don't want to keep doing this and 'come out of the stuttering closet' you can try opening up to those you feel most comfortable and even stuttering a little around them.