Basically anything will work if you pop an Intel A380 in there and set up hardware encoding.
Honestly, a decently fast CPU with QuickSync will work just fine without a GPU. Something like a mini PC with an Intel N100 would work great.
There’s also no reason to use an NVMe RAID. Either just buy a big NVMe or use a HDD raid. Either way, have a backup solution if that’s what you’re going for, cause RAID is not backup.
As far as BSD, I have no idea if that will work. I guess if it runs Docker, you can use the Jellyfin Docker image. What makes you want to use it over Linux?