Yeah, my partner and I have been told this as well. I think it mostly stems from a jealousy that when one suffers through raising children, we all must. It is threatening to others to go against the status quo. But it isn't selfish. Like you, I think having kids is selfish these days.