Not everywhere. Many places its much more sustainable to make clothes from the animals you are eating and it makes sure that you aren't wasting any of the life you've taken that you need to survive.
Growing cattle has also had a massive impact on the environment. And you often need more land for animal based materials because you both need land for the animals and the land to grow food for the animals. With cotton at least you just need land for the cotton.