That's why philanthropy needs to be reigned in. It's just such people using money (that would've gone to taxes otherwise) dictating how society should be run, whether in academia, the arts, leisure or even policing.
If they paid this money in taxes, we'd all get a say what happens to the money (I know, that's mostly in theory, in reality the rich get to say what happens with that money too).