I can speak at least for rootless podman, I spent some hours on it and different ways I tried all ended in permission issues.
I gave up on trying to do it properly and just set the permissions of the /dev/dri device to 666, so that my podman container can use the gpu for transcoding.
Part of the issue with the container images that I tried is that they create a new user with whatever uid:gid I pass to the container, and so even if my nonroot user is part of the render group, the new user inside the container is not and so it can't write to the /dev/dri/renderD128 (gpu), and so transcode wouldn't work.
That's where I left the troubleshooting at cause it was being a headache