Supply of homes is not dynamic like a factory, it can't respond quickly to prices. Investors will grumble about whatever they want, but the alternative is a time consuming and expensive sale.
Even then those homes don't disappear . If they are removed from the rental market they will just end up sold to the very group of people who were renting them, thus reducing demand. Rents would not move much, although there will probably be more real estate sales than usual as the rental market sheds the lazier real estate investors.