Recognising mass-immigrantion as non-ideal can be valid if coming from a place of compassion. But with this perspective, mass-immigration is seen as a symptom of wider socioeconomic problems (or non-societal factors such as natural disasters), not as a problem in itself that needs to be "fixed" by sending immigrants "back home".
Furthermore, seeing immigration as a cause for socioeconomic problems only comes from a place of racism, ascribing negative expectations to people according to their country of origin / culture / ethnicity. It is clear that you stand with this camp from how you phrased what you think "the left" thinks:
"Immigration is good from any country in the world and if you have any reservations what so ever you're racist".
It implies that a person's country of origin plays a factor in whether or not they can be considered a "good" immigrant. That's racist.