The question follows the issues I was seeing from SignalTap when debugging the master ports to a Altera DMA core mentioned in my recent post (Cyclone 5 F2SDRAM issue - #5 by jackfrye11)
I do run that command in my script with a bit mask of 0xffffffff. The interface does not seem to be enabled.
bridge enable 0xffffffff
Where is the uboot device tree blob files located? In u-boot-socfpga source or linux-socfpga? Is it the same blob that is used for the kernel? Where are the files for Arria10? Did you have to change anything in those files?
Where can I find an example of the FPGA2HPS SDRAM node for the device tree? The other interfaces are listed in the decompiled dts I am looking at, but not the node for FPGA2HPS. I think I might need to add it manually.
I had similar issue but “bridge enable” fixed it for me. IIRC from looking at the u-boot source, the bridge command only enables bridges for which the u-boot device tree has the “hps_fpgabridge<x>” entry present with the “init-val = <1>;” set. As this is in the handoff dtsi file generated from the Quartus project I suspect you may be missing something in there which causes that entry to be created.
With the new bootloader flow, how can you change the bootloader device tree? Is there some file in u-boot-socfpga that will allow me to do that? I think I already did this for the kernel one. How can I make one for u-boot or tell u-boot where my kernel one is, which includes the setting you have listed above
Hi @jackfrye11
The process will somewhat depend on what board you are using and what version of u-boot you have. We are using u-boot 2020.04, in that release all the supported board device trees are in uboot/arch/arm/dts. In there you should find your boards dts file if its supported, so for example if you have an Arria 10 SoCDK board there are a number of file which starts socfpga_arria10_socdk_*.dts. There are several for the SoCDK board as it has different boot methods, NAND, QSPI, SD, etc.
So depending on which boot method is applicable, and the board as they don’t all follow the same structure, there should be a *_handoff.dtsi file. This is the file you need to replace with the device tree file generated from the firmware projects hps_isw_handoff folder. So again using the Arria 10 SoCDK as an example, when you build u-boot it should compile all the Arria 10 device trees (assuming you have selected the Arria 10 SoCDK as the Altera SOCFPGA board in the u-boot config), generating equivalent .dtb files.
The device tree files are hierarchical so it starts with say the socfpga_arria10_socdk_nand.dts file, this includes various common dtsi files such as the socfpga_arria10_socdk.dtsi which describe features of all SoCDK board, which in turn include other files like socfpga_arria10.dtsi, etc. as well as the handoff file, socfpga_arria10_handoff.dtsi, which contains the project specific SoC FPGA differences layered on top of the common definitions.
Now you may have to do things slightly different depending on your board, from earlier info it looks like you are using a Cyclone V, I assume the process is broadly the same. Like I said some Altera board device tree files have been quite nicely broken out ad structures, others not so.
I am seeing same behavior. Do I need it in kernel device tree too? I have it in there but it seems like it is not registering. I see nothing in dmesg about it.
I have it registering in the kernel. Still same failing behavior (cmd ports never ready). What does your u-boot device tree node look like? Here is the node I have in the kernel and u-boot device tree
So did you manually add that to the socfpga.dtsi as part of the uboot code?
My uboot dtsi segment is quite a bit different. The handoff file generated from the hps_isw_handoff firmware folder contains entries as below for the three SDRAM bridges (numbers differ):
However as this is just an incremental dts file some of the info you have, such as the bridge address, will be defined in a common file, again caveat this with the above is for an Arria 10. One thing that is different is the init-val has a hyphen not an underscore.
The main concern I have is that you manually added that entry rather than having it generated from the firmware project, assuming a Cyclone V follows the same flow as an Arria 10.
It seems for Arria10 that bsp-editor generates either the device tree itself or a handoff file for u-boot to generate the device tree depending on what version of the Intel SoC tools you are using. CycloneV does not have as extensive support.
According to the wiki, “All custom user settings must be done directly in U-Boot (device tree, configuration and source code).”
I will play around with the “init_val” vs “init-val” change and let you know if I see anything. I too worry about exactly how the sdram interface node is being fit into the larger DT structure and if that is being done properly. Does u-boot use the same device tree compile as Linux? Perhaps I can find a way to compile and decompile the u-boot device tree to flatten it out, then add the node, then recompile with dtc, telling u-boot-socfpga to use my newer recompiled version of the device tree.
I’ve had a quick look at the Cyclone V dts files in uboot and the technical reference manual. It looks like the Cyclone V is quite a bit different to the Arria 10 and as such I’m not sure I can suggest anything else of any help.
During the quick look I tried to find the F2H_SDRAM registers in the reference manual and failed, so I have no idea whether they need to be set up in the same way as the Arria 10, i.e. where any bridge enable bits might be. I’m not even sure that the socfpga-fpga2sdram-bridge device type in the dts file is even appropriate to the Cyclone V. If I’m reading the manual right the Cyclone V has up to six paths from the FPGA to SDRAM, the Arria 10 only has three.
You may need to dig into the uboot source code and trace through the bridge enable command and see how this applies to a Cyclone V, this may then give you a clue as to what is needed in the device tree.
Ok. I dug through the u-boot code. It seem the command “bridge” has no concept of the fpga2sdram bridge. If you look at the code and macro, it appears only concerned with the other three bridges.
I am not sure if there are other things I need to do in u-boot to make sure the processor and the FPGA communicate with the SDRAM in a “responsible” way such that one hitting the memory doesn’t mess up the state of the other (in this case the FPGA manipulating the L3 interconnect such that processor cannot use it)
It looks like this is where the Arria 10 and Cyclone V diverge, with bridge implementation significantly different. For Arria 10 the do_bridge_enable() ultimately calls the socfpga_reset_deassert_bridges_handoff() method which in turn loops over all the bridges defined by the bridge_cfg_tbl[] and checks for the init-val in the u-boot handoff dts file. That config table includes H2F, LWH2F, F2H, as well as the three F2SDR.
The code in misc_gen5.c, which I presume is used by Cyclone V, is significantly different, ultimately calling socfpga_bridges_set_handoff_regs() from reset_manager_gen5.c. It does appear that in that method only the H2F, LWH2F and F2H bridges are handled.
Looks like you may have to write to the appropriate reset register to enable the SDRAM bridges yourself.
OK so you have an Altera mSGDMA instantiated in the fabric, reading and writing to HPS DRAM. That is broadly what we have, albeit separate DMA engines are used to copy data from the HPS DRAM to the FPGA fabric and data from the fabric to the HPS DRAM and it works fine.
This may be an obvious question but are you taking into account that once in Linux the DRAM is virtualised and you need to perform a virtual to physical lookup in order to provide the fabric DMA engine with a valid address. This applied to both the read from and write to DRAM. Any mistake there could result in your data being written back to a piece or memory in use with the kernel, hence the crashes.
Any DMA transfer will also need to account for the memory paging as a result of the physical to virtual mapping. Data in user space buffers may span multiple physical pages, which are not guaranteed to be contiguous unless you allocate such memory and stage your data there before passing to the DMA engine. Data does not need to be copied to a contiguous buffer if you lock down all pages associated with the user space buffer and then set the DMA engine up with multiple scatter-gather descriptors.
I don’t think it has SG capability, so I’m not writing that metadata to some FPGA RAM or anything that complicated.
For allocation of memory, this is the important code in my driver.
#define BUFFER_SIZE 0x1000
typedef struct tx_dma_buf {
struct cdev cdev;
volatile phys_addr_t phys_addr_tx;
volatile unsigned int *virt_addr_tx;
volatile unsigned int *dma_regs;
dev_t dev_node;
struct class *class;
} tx_dma_buf_t;
typedef struct rx_dma_buf {
struct cdev cdev;
volatile phys_addr_t phys_addr_rx;
volatile unsigned int *virt_addr_rx;
dev_t dev_node;
struct class *class;
} rx_dma_buf_t;
tx_dma_buf_t *tx_dma_buf;
rx_dma_buf_t *rx_dma_buf;
static int __init my_init(void)
{
int err;
tx_dma_buf = (tx_dma_buf_t *) kmalloc(sizeof(tx_dma_buf_t));
rx_dma_buf = (rx_dma_buf_t *) kmalloc(sizeof(rx_dma_buf_t));
// Create TX and RX buffers by letting the kernel choose some physcial address and give us virtual pointer
// Use kmalloc with correct flags to ensure memory is contiguous
tx_dma_buf->virt_addr_tx = (unsigned int *)kmalloc(BUFFER_SIZE, GFP_KERNEL);
rx_dma_buf->virt_addr_rx = (unsigned int *)kmalloc(BUFFER_SIZE, GFP_KERNEL);
// Get the physcial addresses for programming the DMA using virt_to_phys()
tx_dma_buf->phys_addr_tx = virt_to_phys( (volatile void *)tx_dma_buf->virt_addr_tx);
rx_dma_buf->phys_addr_rx = virt_to_phys( (volatile void *)rx_dma_buf->virt_addr_rx);
printk( KERN_INFO "tx_dma_buf->virt_addr_tx: %x\n", tx_dma_buf->virt_addr_tx);
printk( KERN_INFO "rx_dma_buf->virt_addr_rx: %x\n", rx_dma_buf->virt_addr_rx);
printk( KERN_INFO "tx_dma_buf->phys_addr_tx: %x\n", tx_dma_buf->phys_addr_tx);
printk( KERN_INFO "rx_dma_buf->phys_addr_rx: %x\n", rx_dma_buf->phys_addr_rx);
...
}
Now the question would be whether the kmalloc is kosher or if I should be using dma_alloc_coherent(). I’ve done kmalloc on Xilinx platform before.
Also worth noting, the TX physical address printed when I “insmod” the driver and the init runs matches what I am seeing in the SignalTap (which makes sense since that is what I write to the register in the application code.)
Based on the RTL generated from Platform Designer, which shows how the AXI signals are mapped onto the raw interfaces, it seems like SDRAM interface is happy with the address commands, but never wants to RVALID-up and start sending read data.
I’m not familiar with that particular DMA core you are using, that said the FPGA side of things is not my speciality, I work more on the software side.
The code snippet seems OK. kmalloc() allocates contiguous physical memory so can be used as a source or destination for a DMA transaction. I believe the main difference between kmalloc() and dma_alloc_coherent(), which is what my driver uses, is that the latter automatically handles cache coherency for you. With kmalloc() you would need to manually flush the buffer before the DMA engine transmits the data, or invalidate the buffer after a DMA receive before the processor accesses it. However that would not stop it working, which I think is the issue you are seeing, it would however potentially show invalid data.
I created a design with an address space extender IP with a 64K window that would allow me to write to different 64K areas of the SDRAM via a register that would set base address for a given 64K area. I tried doing a simple write via HPS_AXI_LW (which I proved worked in Signal Tap), through this IP and to the SDRAM interface. I did this in u-boot console and when I initiated the transaction, u-boot froze and then rebooted. This is definitely a HW/FW problem. For reference, I had GPIO at address 0x0 in the FPGA LW BASE address space.
I am reading about SDRAM clocking. I think some of the blocks are supposed to be set up by u-boot via settings pll_config.h, which is created by running qts_filter.sh on you Quartus project. I think your device has the same flow that includes qts_filter. Do you mind sending me your pll_config.h file when you get a chance?
I switched to the 20.1 versions of the embedded_command_shell.sh/u-boot-socfpga source and now FPGA2SDRAM interface works.
I used:
intelFPGA_lite v20.1 of embedded_command_shell.sh
u-boot-socfpga branch origin/socfpga_v2020.07
Couple of notes
u-boot-socfpga 2020.07 will automatically scan for u-boot.scr (compiled u-boot script)
The image I was using was still generated from intelFPGA_lite v19.1 (wasn’t forced to upgrade F/W to get it to work)
