So it is the level of "privilege" that does or does not allow the commission of -isms then. The better off the target is, the more acceptable discrimination is? That is also a very Western perspective. It would be ok to tell Muslims in the Middle East that terrorism is their responsibility because their country's power structure does put Islam firmly above others?
This "some animals are more equal than others" stuff is moral equivocating. If something is wrong if done to a group that isn't "in power", then it is also wrong to do it to the group "in power". This isn't a zero sum game. We don't have to weight the guilt by association for a black man when compared with a white man because systemic racism competes with systemic patriarchy. If you do think that the immutable characteristics a person is born with are the most important things about them, I would encourage you to self interrogate how messed up that is.