FPGA only partially accessible in u-boot

Boot
21

Great effort @BrianM. I may take a look at it if I get the time to see if it would work for the Arria 10, although we may be too far down the road of the alternative workaround.

22

I found some typos in the file “HOW_TO_BOOT_NVME.TXT” and learned a few more things. I updated it and rather than re-uploading, I’ll just include it here:

These instructions are for linux versioned tools. If you have Windows you will have to adapt them.

These instructions may be incomplete.

First: Build the PCIe and HPS FPGA image.

  1. Run quartus in the top directory
    cd CV_NVME_BOOT_EXAMPLE
    quartus &

  2. File → Open Project: file name = nvme_boot.qpf

  3. Select your Cyclone V part by choosing the Assignments → Device.

    This is going to break things. It is your responsibility to fix the pin outputs for clocks and reset.

    This design uses a pecl low jitter 100 MHz source clock. If you use something else you will have more work to do
    in the top level verilog and in the Platform Designer qsys opened in the next steps

  4. Tools → Platform Designer

  5. Open file name = nvme_boot_sys.qsys

  6. You have to modify the configuration of the HPS to match your hardware. If you have a working FPGA image,
    you can open another instance of Platform Designer (qsys-edit from the command line), open your qsys and hand
    copy all your parameters over to the nvme_boot_sys design. Please note: Do not change the Address mapping of the
    PCIe root port until you are sure it works and you know what you are doing. There is a hack in the u-boot source
    code to manufacture the correct CPU address for NVME BAR0 which will have to be manually adjusted to get u-boot’s
    NVMe driver working. To find that code grep -R “HACK HACK” in the u-boot source code folder.

    For your first attempt you should change as little as possible because changing even a character in the device tree can
    result in non-working hardware and trust me, when you are working things the other way and you have five things wrong,
    figuring out what is wrong and how to fix it is the most complicated game of whack-a-mole ever. I created this design
    so you won’t have to go through what I went through. Change as little as possible until you have it working,
    then change only one thing at a time.

    You have been warned: if you change more than one thing at a time you will risk orders of magnitude harder debugging.

  7. Once you are satisfied with the HPS configuration, save the design.

    (note: when you are ready to integrate thr PCIe controller with your hardware I recommend you pull your hardware into
    this design instead of the other way around. I think it will save time and effort. Of course your hardware has to be
    a “user created” Platform Designer module to accomplish that. This works really well for me.)

  8. After you save the qsys back with your HPS configuration, Choose menu Generate → Generate HDL.
    Ensure you do not generate simulation models. That is a huge waste of time.

  9. Exit Platform Designer.

  10. Back in Quartus press the right pointing blue triangle run button to start FPGA compilation.

  11. It should take only 5 to 10 minutes to sythesize, place and route.

  12. There are an ungodly number of warnings. I went through them all so you – probably – won’t have to.

  13. If you get errors, you have to fix them.

  14. When SPR is done, check the timing to make sure you met timing. If you didn’t, it’s probably my fault.
    This design runs the Root Port MM access to DRAM at 125 MHz. It’s crazy, but it saves a lot of modules.
    I should note that you cannot add fifo depth or change the interface configurations at all and meet timing
    on the part I’m using. This is the only configuration I have found that works at 125 MHz. If you can’t get
    it to meet timing at 125 MHz you will have to create another clock with a PLL at around 50-60 MHz (I tried
    100, no dice). Then you’ll have to add clock crossing bridges (BY HAND, the auto inserters will not meet
    timing). If you look at the example PCIe root port Altera provides you’ll see how they use the MM clock
    crossing bridges.

    Please note: The extra hardware found in the Atera example, SGDMA and 256K SRAM, are not used
    by the drivers included in the linux kernel. There is no reason to add them, they just waste space.

  15. When SPR is done and timing is met (note: in this design unspecified paths are not failures.
    That is Altera / intel’s fault for not properly specifying their own soft and hard module timing.)

    File → Convert Programming Files.
    When the window opens, select Programming file type: Raw Binary File (.rbf)
    Set Mode: 8 or 16 bit parallel.
    Set the output file name to output_files/nvme_boot.rbf
    Set the SOF Data to output_files/nvme_boot.sof
    Click the Generate Button in the lower right.

  16. At the shell prompt:
    cd output_files
    lzma -k nvme_boot.rbf (you can’t use xz)
    This produces a 284KB-ish file called, “nvme_boot.rbf.lzma” which will be used to bootstrap
    the pcie hardware in FPGA from u-boot. You may be able to compress more with -e -9, but pay
    attention to decompressor memory requirements.

Clone & patch v2020.10 of u-boot-socfpga:

  1. cd software
  2. mkdir bootloader
  3. cd bootloader
  4. git clone GitHub - altera-opensource/u-boot-socfpga: Official Intel SOCFPGA U-Boot repository. Note: (1) A "RC" labeled branch is for internal active development use and customer early access without official customer support. (2) Latest stable branch (no RC labeled) is strongly recommended for development and production use outside of Intel. (3) See doc/README.socfpga for Quartus and Device support.
  5. cd u-boot-socfpga-socfpga_v2020.10
  6. patch -p1 < …/u-boot-socfpga-socfpga_v2020.10.diff

Generate the BSP:

  1. cd CV_NVME_BOOT_EXAMPLE

  2. /opt/intelFPGA/20.1/embedded/embedded_command_shell.sh
    (note: you will have to enter the path to your version of quartus embedded)

    You will see this:

    Intel FPGA Embedded Command Shell

    Version 20.1 [Build 711]

  3. bsp-create-settings --jdi output_files/nvme_boot.jdi --type spl --bsp-dir software/bootloader
    –preloader-settings-dir hps_isw_handoff/nvme_boot_sys_hps_0 --settings software/bootloader/settings.bsp

    (you might need to create a destination directory with mkdir -p)

Configure u-boot spl
10. cd software/bootloader/u-boot-socfpga-socfpga_v2020.10
11. ./arch/arm/mach-socfpga/qts-filter.sh cyclone5 …/…/…/ …/ ./board/altera/cyclone5-socdk/qts/

Compile u-boot
12. cp …/nvme_config_backup .config
13. !!! make sure you have your gcc cross-compiler setup !!!
14. make -j [number of threads on your PC]

Fetch & patch the kernel:
(note: I’m pointing to the latest 5.11 kernel because 5.12 did not work for some reason: some ext4 nvme driver issue I suspect.)
(also: I tested with 5.11. If 5.11.21 does not work for some reason, try 5.11)

  1. cd software
  2. mkdir kernel
  3. cd kernel
  4. wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.21.tar.xz
  5. tar xf linux-5.11.21.tar.xz
  6. cd linux-5.11.21
  7. patch -p1 < …/linux-5.11.diff

Compile the kernel
8. make -j [number of threads on your PC]
(This takes a while)
9. make LOADADDR=0x01000000 uImage
(NVMe boot uses uImage)

  1. You’ll also need to build a linux bootable filesystem. We use Yocto.
    You have to figure that out on your own. I don’t know how to do it.

Now you have everything you need to boot from NVMe.

Important files:

CV_NVME_BOOT_EXAMPLE/software/kernel/linux-5.11.21/arch/arm/boot/uImage:
linux kernel image used by u-boot to start linux.

CV_NVME_BOOT_EXAMPLE/software/kernel/linux-5.11.21/arch/arm/boot/dts/socfpga_cyclone5_socdk.dtb
kernel device tree used by this example, loaded by u-boot, utilized by linux.

CV_NVME_BOOT_EXAMPLE/software/bootloader/u-boot-socfpga-socfpga_v2020.10/u-boot-with-spl.sfp
u-boot image that is flashed to location 0x0000000 of the boot QSPI.

CV_NVME_BOOT_EXAMPLE/software/bootloader/u-boot-socfpga-socfpga_v2020.10/arch/arm/dts/socfpga_cyclone5_socdk.dts
device tree source used by u-boot.

CV_NVME_BOOT_EXAMPLE/software/bootloader/u-boot-socfpga-socfpga_v2020.10/arch/arm/dts/socfpga_cyclone5_socdk-u-boot.dts
device tree source used by u-boot spl.

CV_NVME_BOOT_EXAMPLE/output_files/nvme_boot.rbf.lzma
This compressed FPGA image is stored at the 1 Megabyte point, offset: 0x100000, of the QSPI flash device.

This u-boot configuration assumes a 2MB QSPI Flash device. If you have a different size, you’ll need to modify
u-boot’s configuration and ensure that the SF driver supports your device.

QSPI Memory Map:
Address: Length :Description
0x000000:0x0E0000:896 KB for spl(256 KB) + u-boot
0x0E0000:0x010000:64 KB for u-boot environment backup
0x0F0000:0x010000:64 KB for u-boot environment primary
0x100000:0x080000:512 KB for fpga lzma compressed image
0x180000:0x080000:Free space could be used to store another FPGA image for redundancy.

How to program the QSPI with the images:

  1. Copy nvme_boot.rbf.lzma and u-boot-with-spl.sfp onto an SDCard. You can boot an existing u-boot from your
    QSPI or change the bootstrap configuration to boot from SDCard. The version of u-boot you run must be able
    to read the SDCard and write the QSPI device.

  2. Format your NVMe SSD as a single ext4 partition and rsync -avr your root file system onto it.

  3. Also copy uImage, your full fpga rbf file, and either your dtb for the kernel or the
    kernel socfpga_cyclone5_socdk.dtb listed above to the root folder of the NVMe SSD.

  4. Boot your board from SD or QSPI. Obviously, you’ll need a serial cable to control u-boot.

The rest of this document assumes you put the u-boot sfp and fpga lzma files on to an SDCard partition 1.

  1. Flash the QSPI with the new version of u-boot:
    sf probe
    load mmc 0:1 0x08000000 u-boot-with-spl.sfp
    sf update 0x08000000 0x000000 $filesize

  2. Flash the QSPI with the compressed FPGA image:
    sf probe
    load mmc 0:1 0x08000000 nvme_boot.rbf.lzma
    sf update 0x08000000 0x100000 $filesize
    PLEASE TAKE NOTE of the length of the compressed FPGA image, you need it later. Mine is: 0x46eb3.
    (note: I have put the fpga image at the 1MB boundary half way through the device, YMMV)

  3. Set your jumpers to boot from QSPI (if you haven’t already).

  4. Power cycle your board.

If you got all your settings correct, the NVMe supporting u-boot will start.
If you have issues, it’s time to debug.

When it boots you should see something similar to this:

U-Boot SPL 2020.10 (May 14 2021 - 11:28:55 -0700)
DDRCAL: Scrubbing ECC RAM (1024 MiB).
DDRCAL: SDRAM-ECC initialized success with 608 ms
Trying to boot from SPI

U-Boot 2020.10 (May 14 2021 - 11:28:55 -0700)

CPU: Altera SoCFPGA Platform
FPGA: Altera Cyclone V, SE/A5 or SX/C5 or ST/D5, version 0x0
BOOT: QSPI Flash (3.0V)
Watchdog enabled
DRAM: 1 GiB
MMC: dwmmc0@ff704000: 0
Loading Environment from SPIFlash… SF: Detected is25lp016 with page size 256 Bytes, erase size 64 KiB, total 2 MiB
OK
In: serial
Out: serial
Err: serial
Model: Altera SOCFPGA Cyclone V SoC Development Kit
Net: eth0: ethernet@ff702000
=>

At this point you are ready to boot. These are the command you need.
You will eventually make these part of the boot environment variables.

  1. sf probe
  2. sf read 0x08000000 0x100000 0x46eb3 ← Use the length of your compressed FPGA here, not this number!
  3. lzmadec 0x08000000 0x09000000 → note the decompressed length of your FPGA
  4. fpga load 0 0x09000000 7007204 ← Use the length of your uncompressed FPGA rbf file here!
  5. bridge enable
  6. pci
  7. nvme scan
  8. load nvme 0:1 0x07000000 socfpga_cyclone5_socdk.dtb
  9. load nvme 0:1 $loadaddr uImage

If you want to load your full FPGA image, now is the time to do it. It looks something like this:
(if you want, you can boot the stripped down FPGA image by jumping to step 13.)
10. load nvme 0:1 0x08000000 soc_system.rbf
11. fpga load 0 0x08000000 $filesize
12. bridge enable
13. setenv bootargs root=/dev/nvme0n1p1 rw rootwait earlyprintk console=ttyS0,115200n8
14. bootm $loadaddr - 0x07000000

Linux should boot from your SSD. It is pretty verbose it will tell you what went wrong.
Look at the error messages if you have problems.

Final comments:

  1. I suspect it is possible to get all this to work with the extlinux.conf file. However,
    the FPGA cannot be reloaded until all accesses to the NVMe are done. After you reload
    the FPGA, the NVMe and it’s driver will crash. linux re-initializes it which is why
    we can get away with booting linux.

  2. My hacks to u-boot-socfpga-socfpga-2020.10 are extensive. When u-boot-socfpga-socfpga-2021.04
    is released, the issues with pcie mapping will go away. There are still issues with clearing
    and invalidating dcache. I’m hoping to work with the u-boot guys on that at some point.

  3. Pay particular attention to the kernel socfpga.dtsi. Getting all that to work was a difficult
    task because documentation is quite lacking.

23

Hi…I boot from QSPI->eMCC and my eMCC simply has a filesystem that I duplicate the boot.scr and image.ub onto. I realize you need to compose the documents at a particular location that u-boot needs to peruse from yet I don’t know where you set that up.

24

I don’t understand your question, could you rephrase it?

Also, sorry for the delay. I have to figure out how to turn on notifications for this forum.