Pi 5 USB Boot is clunky

Flashing USB Boot SD card does nothing and it still boots from SD Card, remove the SD Card with only USB and the Pi idles, going into raspi-config on the sd card setting Boot Order to usb and then rebooting finally boots to the USB, if I reboot, it still boots into the usb and works flawlessly but the moment I shut it down and start it again, it will start to boot into the SD Card or idle without it.... Latest raspi-config and bootloader and checked wtih 2 usb drives, even USB 2.0, used rpi imager and official img, balena etcher, ubuntu server and raspbian, is anybody else having this issue or is it just that I am dumb?

You shouldn't need a usb drive, if your microSD is formatted correctly and you've used 'NOOBS' to image the OS onto the card, it should be able to boot just from your microSD. Keep in mind that the microSD acts as the pi's boot drive so taking it out will result in the idling you keep experiencing.

I've never tried using a USB to boot a pi, so to be honest I'm not sure how you'd go about it or if it's even possible. I wish I could give you more advice but I'm a beginner as well, and don't know anything more than the default method 😞

https://www.raspberrypi.com/documentation/computers/raspberry-pi-5.html#usb-boot-and-power-supplies

You can check the USB devices power consumption with lsusb -v and to filer out the result you wan't lsusb -v | grep 'MaxPower\|Vendor'

I found USB3.0 to be extremely slow compared to the default SD card for no particular reason

The file read/write speed is better on the USB, but general usability is 10x better on the SD card

This is on the Pi4 at least

my USB SSD refused to boot

it got into the booting screen but kept aborting the connection and retrying or something like that

Maybe it's a power issue there, no clue really

They are generally slower than SSD and M.2, so I'm not surprised that the system is slow. Power can't be direct, my m.2 doesn't have its own power supply and runs without problems. It's more likely to be the flash drive itself.

The flash drive is a pretty good one

and I tried 2 flash drives and both were terrible performance

USB Flash drives isn't fast. Invest in a SSD, more reliable than a SD and faster than both a USB flash and a SD card.