[solved] NVMe / PCIe storage on Arria10

I’m dealing with exactly the same issues as you. I’ve got everything working except interrupts. See this thread: https://community.intel.com/t5/Intel-Quartus-Prime-Software/Bug-Quartus-Pro-20-1-1-Cyclone-V-utilizing-PCIe-example-from-16/m-p/1254729#M68055

A little summary:

  1. You need to include the FPGA2DRAM modules in the dts file. I suspect you have that since you are up and running but getting errors.
  2. I think the read errors you are getting have to do with the way the altera pci driver maps the root port into the DRAM. I think it puts everything at 0x0, right at the beginning of DRAM. Not a good spot. It does not appear to be movable, although some guys got other things to work a few years ago, see this thread: Altera PCIe driver issue with SSD devices

I solved it like this:
Add this to your dts top:

reserved-memory {
#address-cells = <1>;
#size-cells = <1>;
ranges;

// 2 MiB reserved for PCI Express DMA
pcidma1@0 {
reg = <0x00000000 0x00200000>;
// Note: this is the maximum you can reserve with kernel defaults and relocating the kernel doesn't seem to work with ARM ARCH.
no-map;
};
};

This reserves the first 2MB of DRAM. The kernel complains:

ERROR: reserving fdt memory region failed (addr=0 size=200000)

however, it still works, as shown in ‘cat /proc/iomem’:

root@cyclone5:/mnt/test# cat /proc/iomem  
00200000-3fffffff : System RAM <- Notice first 2MB is not used...
 00c00000-00cabe7f : Kernel data
c0000000-c01fffff : pcie@000000000
 c0000000-c00fffff : PCI Bus 0000:01
   c0000000-c0003fff : 0000:01:00.0
     c0000000-c0003fff : nvme
ff200000-ff20007f : ff200080.msi vector_slave
ff200080-ff20008f : ff200080.msi csr
ff220000-ff223fff : c0000000.pcie Cra
ff700000-ff701fff : ff700000.ethernet ethernet@ff700000
ff702000-ff703fff : ff702000.ethernet ethernet@ff702000
ff704000-ff704fff : ff704000.dwmmc0 dwmmc0@ff704000
ff705000-ff705fff : ff705000.spi spi@ff705000
ff706000-ff706fff : ff706000.fpgamgr fpgamgr@ff706000
ff709000-ff709fff : ff709000.gpio gpio@ff709000
ffa00000-ffa00fff : ff705000.spi spi@ff705000
ffb90000-ffb90003 : ff706000.fpgamgr fpgamgr@ff706000
ffc02000-ffc0201f : serial
ffc03000-ffc0301f : serial
ffc04000-ffc04fff : ffc04000.i2c i2c@ffc04000
ffd02000-ffd02fff : ffd02000.watchdog watchdog@ffd02000
ffd05000-ffd05fff : rstmgr
ffe01000-ffe01fff : pdma@ffe01000
ffff0000-ffffffff : ffff0000.sram

Next, make the Txs bus on the Avalon-MM PCIe hard macro, 2MB big by setting it to 32 bits wide with a 1MB address width and two address pages. (This might work with 1MB in 64 bit mode, I will verify that later to shrink the reserved memory area once I figure out my latest issue)