Personally, I'd bring it up only if it happens again. It doesn't sound like he made the remark with malicious intent. He may genuinely somehow not know. If it's just the two of you and he makes another remark, I'd bring it up lightly. If it's with a group, I'd wait until you two are alone to bring it up.