There's one practical thing. Routers have had years to optimize IPv4 routing, which has to be redone for IPv6. Same with networking stacks in general.
In theory, IPv6 should be faster by not having to do bullshit like CGNAT. There's every reason to think it'll match that advantage if we just make it happen.