It's the same with me, I can't utter a single letter in some situations, and in other situations I don't stutter. In my experience, it's because of 'conditions / triggers' I set to myself. So basically, you can try out this to analyze your condition or trigger: whenever you stutter, observe your mindset to make the unconscious more conscious and ask yourself 'does this or this thought reinforce justifying / anticipating stuttering?' You said 'without being nervous', in my opinion, justifying and anticipating compulsion is mostly not from anxiety in my experience compared to non-anxiety triggers.