If using a somewhat modern distro, this isn't an issue anymore (unless you run a really old OpenGL game).
I run my PC in this way with little to no performance degradation: monitors go to my motherboard (r5 2400g CPU with vega11 iGPU) and games use my RX 5700XT without having to do anything at all... Pretty smart handling tbh