That's the hand we were dealt. My stuttering doesn't get worse with anxiety, but it does for a lot of people. Mine just naturally fluctuates in a seemingly random way. For me, what changed my life was realizing how little impact stuttering actually had on me. It was all in my head. I was the only one bothered by it. Sure, it might take longer to say something, or I might feel embarrassed in certain situations, but that's not so bad. It sure isn't killing me or hurting me physically. From there, exposing myself and years of stuttering in front of people has made it no longer affect me mentally. Good luck.