Tiny linux/bare kernel for DE0/D10 is needed

I plan to create some opensource project for DE10-nano board. And i want to get all that wide range of devices/filesystems supported by linux kernel. Because CyclonV SoC has bug in FPGA2SDRAM bridge, it’s necessary to reboot HPS part after FPGA bitstream loading to make sure FPGA2SDRAM bridge is working. That’s why i need to use some very tiny linux or bare kernel with minimal initrd in ram - it should boot very fast to keep user experience in a good level.
what i would like to get from linux:

  1. USB storage
  2. USB HUB.HID: keyboard, mouse, many different gamepads and joysticks
  4. Network as a client is planing, but not mandatory at this stage. But it would be good to have very basic anonymous ftp server using existing ethernet port to exchange files.

So, may be something tiny like DD-WRT or similar embedded linux would be suitable. Most important is booting time.

Lifecycle is this way:

  1. load Linux (from mmc) -> load FPGA -> ready.
  2. By user request: Choose another FPGA bitstream while using current one: Load FPGA -> reboot Linux -> ready.

While first one is not required very fast loading since it’s initial turn on the device, the loading time on second scenario is more critical.

I’ve thought i can use bare U-boot, but after study the code i found it has very bad USB support. It doesn’t support HID devices on DWC2 USB host used on socfpga. So, it will require a lot of time to add HID support.

I know there are some embedded configuration exist, but i need specifically for socfpga.

I’ve already built Ubuntu minimal and Debian minimal versions - they both require too much time to load. I expect a total loading time not more than 3-5 seconds. Is it possible?