I don't think you get to make a black and white, general argument about this. How about this: if a person raises and cares for a chicken, giving it a charmed life it would have otherwise never had, but takes and eats its unfertilised eggs, then that's not morally wrong.
It's just not as obvious as people think, and your first sentence is a naive oversimplification and a great example of the kind of lazy argument I'm talking about. But I don't want to get into it with my friends since it's such a touchy subject, and I'll never get a decent conversation about it online.