I don't think it's primarily tension/anxiety. It's about the processing demand of different speaking situations. Some situations are far less complex - e.g. speaking alone. With other people you introduce social dynamics and unpredictability which makes it much more demanding.