You might be true. But I never stutter when I choose to speak when no one's around. That surely says a lot. You're always a bit nervous even if you think you're truly comfortable and that's why we stutter even if we're with close relatives/friends etc. You state that we don't want to embarrass ourselves or feel out of control. Those aren't physical issues but more psychological. There might be different reasons why different people stutter, but in my experience I have 99% fluency if I'm feeling extremely confident for whatever reason it may be and 100% fluency if I'm practicing a speech by myself.