I'd agree with that, but calling it privilege is a bad name. Because how do you implement equality when dealing with privilege? You take from the privileged to level the playing field. So when you apply that to being privileged because you aren't being discriminated against, the solution is to remove that privilege? So... do more discrimination so everyone is equal?
When you instead identify those that are oppressed and those that are not, the solution to equality is to remove the oppression. So when applied to our situation, remove the discrimination so everyone is equally not discriminated against.