No you should never worry about what other people think about you. Think of them as being irrelevant which they should be. Life is too short and difficult as it is so why waste time on a bunch of idiot people.