Xwrapper not always working on PI3

I have a minimal raspberry os install with xfce on, using xorg-xserver-legacy, I have set anybody as Xwrapper configuration and I can run startx from systemd and ssh fine on raspberry 4; I am using the exact same install (same sd) for raspberry 3 too and all startx sometimes (decided at boot) doesnt work complaining about permissioms etc... (Like legacy was not installed or not configured) Putting need_root in the wrapper seems to solve the issue, I'd like to avoid that tho; I'd like to understand what makes it work all the time under pi4 and not pi3

Update, works in both ONLY with need root yes, just the PI4 auto detect for root requirement works, while in pi3 chooses sometimes no (maybe depending on order of some startup processes)