Not everyone who is a landlord is born into wealth. Someone born into poverty can also be a landlord.
By your logic, grocery stores are the same. They don't grow the food. They don't invent new food. There's nothing wrong with grocery stores either.
The main reason landlords are hated is jealousy. People hate those who have something they don't. Especially when landlords worked for what they have and the ones who are jealous didn't -- they want to be handed things for free without contributing. Look at the old parable of the ant and the grasshopper.