Do people care about stutter?

Do people care about stutter? I've heard many times that people don't care about stuttering. If one is smart and interesting enough adequate people don't notice, even on a date. However I see the opposite in reality - even if a person is warned and fine about my stutter before we meet they seem a bit irritated and tired after. Is it just my luck or does stutter really scares people away unless it's mild state?