The you are not being hypocritical. But… the actual way to decrease the negative emotional impact on him is to NOT be upset about it. It’s a weird reversal thing, but the less he thinks you are uncomfortable by it, the more secure he will be.