Humans are social animals, you're the odd one out here from a social perspective, not that you're not entitled to that choice but choices have consequences.
I'd suggest just ignoring them. You aren't going to find a better work environment anywhere else unless you literally have no coworkers.