Do people really care about stuttering?

Do people really care about stuttering? I've heard many times that people don't care about stuttering. If one is smart and interesting enough adequate people don't notice, even on a date. However I see the opposite in reality - even if a person is warned and fine about my stutter before we meet they seem a bit irritated and tired after. I have middle-severe stuttering. Is it just my luck or does stutter really scares people away unless it's mild state?