I don't know if it'll help but I do think if you don't have to rely on medication you never have to go off it or be without it. My stutter is massively affected by my anxiety levels and I manage it through meditation. Even when I don't think I'm particularly stressed or anxious but I notice my stutter is worse than usual. I man's sure I take ten minutes a day to just sit in silence. Close my eyes and focus on breathing slowly. It's not an immediate reduction like you're hoping to get from medication but it always improves over a couple of day. It might not be right for your situation but maybe try working on ways of reducing anxiety without medication first.