Stuttering is terribly varied. We learn to expect the unexpected, so to say. Have you tried managing your stutter with speech therapy? Maybe there are some things you're (not) doing when you're relaxed that work against you.