It really depends. In my experience, most people do care, at least a little. I always tell people I stutter when I meet them. Even then, when I'm still new to people, they look weirded out/concerned when I stutter. After a while they usually get used to it.