Depending on your overall OS and sessions setup, your distro install may not be tweaked properly for Xfce, which still doesn't have Wayland support last I checked. So unless you made sure to clear out all the other global configs that could impact the GUI session, you'll probably have some issues unless you switch to an Xfce catered distro.