It's up to how you feel about it as to whether it's OK or not. I'd just tell him you know he's not trying to be offensive but you don't like it and I'm sure he'll stop.