The problem for the Fermi paradox is that there's no reason to do stuff like that before we start colonizing other solar systems.
Also, how do you destroy a civilization that has a Dyson swarm already? That's not exactly an easy task, and if you insist on remaining stealthy yourself it's nigh impossible.