Well for the current generation consoles they're both x86-64 CPUs with only a single set of GDDR6 memory shared across the CPU and GPU so I'm not sure if you have such a penalty anymore
It’s not that unified memory can’t be created, but it’s not the architecture of a PC, where peripheral cards communicate over the PCI bus, with great penalties to touch RAM.
Are there any tests showing the difference in memory access of x86-64 CPUs with iGPUs compared to ARM chips?