FPGA to SDRAM - Cyclone V - Can write but cannot read

As the title says, I’ve enabled the FPGA2SDRAM interface in my HPS and have written an Avalon Memory Mapped Host/Master to write to a specific memory location.

It’s a very simple code, based off the template in platform designer and it works fine:

`timescale 1 ps / 1 ps
module mem_write (
        output logic [31:0]  avm_m0_address,     // avm_m0.address
        input  logic        avm_m0_waitrequest, //       .waitrequest
        output logic        avm_m0_write,       //       .write
        output logic [31:0]  avm_m0_writedata,   //       .writedata
        input  logic        clk,          //  clock.clk
        input  logic        reset,         //  reset.reset
        input  logic [31:0] in_address
    );

    reg [31:0] cnt;

    assign avm_m0_address = in_address;

    assign avm_m0_write = 1'b1;

    assign avm_m0_writedata = {30'd0, cnt[26], cnt[25]};
    
    always_ff @(posedge clk) begin
      cnt <= cnt + 1'b1;
    end

endmodule

I hardcoded in_address in my top level design to be 32'h20000000 (512MB).

This works fine and I can see it cycling through 0-3 at 0x20000000 at regular intervals with the command:

busybox devmem 0x20000000

But when I try to do the reverse, i.e. read from the same memory location, linux hangs.

Here is my code:

`timescale 1 ps / 1 ps
module mem_read (
        output logic [31:0]  avm_m0_address,     // avm_m0.address
        output logic        avm_m0_read,        //       .read
        input  logic        avm_m0_waitrequest, //       .waitrequest
        input  logic [31:0] avm_m0_readdata,    //       .readdata
        input  logic        clk,          //  clock.clk
        input  logic        reset,         //  reset.reset
        input  logic [31:0] in_address,
        output logic [31:0] out_data
    );  

    reg [31:0] readdata;

    assign avm_m0_address = in_address;
    assign out_data = readdata;
    assign avm_m0_read = 1'b1;

    always_ff @(posedge clk) begin
      if (reset) begin
        readdata <= 32'd0;
      end else begin
        if (avm_m0_waitrequest) begin
          readdata <= readdata;
        end else begin
          readdata <= avm_m0_readdata;
        end 
      end 
    end 
endmodule

Is there something wrong with my code?

I’ve set up the FPGA2SDRAM in HPS to be bi-directional and 256 bits.

Here is a screenshot of platform designer:

If I change the read bit to always 0 as follows:

assign avm_m0_read = 1'b0;

linux doesn’t hang.

I changed the code to have the read bit set to 1 after a slight delay and linux seems to hang exactly after the delay.

At this point, I’m thinking there is something wrong in the configuration that prevents the interconnect from reading. Not sure why writing is permitted though.

I finally got this working. It looks like the reason was indeed something to do with the HPS registers.

When using the HPS and adding some peripherals, we need to modify the registers and set some startup macros in the bootloader so that they are set in a correct order. This is explained for cyclone V devices at this link:

Unfortunately, the example seems to be for [cyclone V soc dev kit] (Cyclone V SoC Development Kit and SoC Embedded Design Suite) and doesn’t work for the de10-nano.

Also, I’m using debian and my own custom bootloader as explained on my tutorial at https://github.com/zangman/de10-nano/wiki. Perhaps that is also preventing this method from working.

After a lot of digging around, I found the following link which shows what registers need to be modified:

https://support.criticallink.com/redmine/projects/mityarm-5cs/wiki/Important_Note_about_FPGAHPS_SDRAM_Bridge

I used the following reference to get the addresses of the various registers mentioned in the article:

https://www.intel.com/content/www/us/en/programmable/hps/cyclone-v/hps.html#sfo1411577374877.html

So basically, when starting up I cancel autoboot at the beginning and enter the following commands:

  1. Set fpgaportreset to 0x0:

     mw 0xFFC25080 0x0
    
  2. Load a design that reads the sdram with the following comamnds:

    fatload mmc 0:1 0x2000000 sdr.rbf
    fpga load 0 0x2000000 0x700000
    
  3. Set the apply bit of the staticcfg:

    mw 0xFFC2505C 0xA
    
  4. Re-enable fpgaportreset:

    mw 0xFFC25080 0xFFFF
    
  5. Continue booting:

    boot
    

After this, the SDRAM read works as expected.

Best way to automate this is to create a boot script and enable u-boot to trigger this. I will share details on this step if I can get to documenting it.

Have documented the steps to create a bootscript here:

I have written a full tutorial here to read from SDRAM:

Hi Jake,

Thanks for this. I’m studying what you’ve done. Very educational to see how someone else has gotten the SDRAM interface working.

Question: in u-boot, have you tried loading your FPGA image and then using the command ‘bridge enable’ to activate the FPGA to SDRAM interface(s)? That works for the PCIe interface to SDRAM 0 interface and the HPS2FPGA interfaces for me. I think you also need to ensure the drivers are loaded in the linux kernel and the linux dts has the features enabled.

I’m struggling with the FPGA to SDRAM 1 interface at the moment. My reads appear to return random data and nothing I do makes any difference. The other interfaces that I’m using: HPS2FPGA_lite, HPS2FPGA, FPGA2SDRAM0 all work, but I can’t get the second one to work.

I have a simulation up and running using the intel mentor simulator with intel’s Avalon Bus Functional models that are built into Platform Designer.

I’m about to embark on the ‘SignalTap’ capability of the tools to try to get some insight into the FPGA, after I make sure that I’ve got u-boot and linux configured correctly to enable the FPGA interfaces.

FYI: here’s what dmesg reports when the modules are loaded:

[    2.466387] fpga_manager fpga0: Altera SOCFPGA FPGA Manager registered
[    2.480052] altera_hps2fpga_bridge ff400000.fpga_bridge: fpga bridge [lwhps2fpga] registered
[    2.493971] altera_hps2fpga_bridge ff500000.fpga_bridge: fpga bridge [hps2fpga] registered
[    2.507234] altera_hps2fpga_bridge ff600000.fpga-bridge: fpga bridge [fpga2hps] registered
[    2.525094] altera_fpga2sdram_bridge ffc25080.fpga-bridge: fpga bridge [fpga2sdram] registered
[    2.537327] altera_fpga2sdram_bridge ffc25080.fpga-bridge: driver initialized with handoff 00000313
[    2.550506] altera_fpga2sdram_bridge ffc25080.fpga-bridge: enabling bridge

Update: I got SDRAM working. Using the Avalon Slave BFM was the proper way to debug the interface.

I’d like to point out to anyone who attempts to use the FPGA to SDRAM interface: you have to make sure you burst at least eight reads from the SDRAM because the controller will not produce any return data until it receives 8. I believe the 8 total refers to the width of the SDRAM. So you need 8 16 bit words for a 16 or 24 bit dram data path and 8 32 bit words if you have a 32 or 40 bit datapath.

Question: in u-boot, have you tried loading your FPGA image and then using the command ‘bridge enable’ to activate the FPGA to SDRAM interface(s)?

As I understand it, the bridge enable command is a custom macro that is available in Terasic’s version of u-boot. Since I’m using a full custom u-boot, the command isn’t available for me.

In any case, as I understand it, the command also only enables the various bridges between HPS and FPGA. I’ve done this by enabling the bridges in the device tree directly. I verified this later as well with the following command:

cat /sys/class/fpga_bridge/*/state

Also, FPGA-HPS and LW FPGA-HPS was working fine as well as SDRAM Write!

I think you also need to ensure the drivers are loaded in the linux kernel and the linux dts has the features enabled.

Yup, I’ve documented these in detail here.

The only thing that finally worked for me was setting the HPS registers as I explained above.

I’m struggling with the FPGA to SDRAM 1 interface at the moment. My reads appear to return random data and nothing I do makes any difference.

I’m wondering if you’ve checked the addressUnits property for your Avalon Master component and whether it is set to WORDS or SYMBOLS. You can check this in the tcl file for your custom component.

If SDRAM1 for you is 64 bits wide, then to read from 0x8, you need to specify the addresses as follows:

  • WORDS - 0x8, has to align with address width
  • SYMBOLS - 0x1, this refers to 0x8 in memory.

I’d like to point out to anyone who attempts to use the FPGA to SDRAM interface: you have to make sure you burst at least eight reads from the SDRAM because the controller will not produce any return data until it receives 8.

This sounds very odd to me. In your avalon master, are you correctly waiting for waitrequest and readdatavalid? Also make sure that burstcount is set correctly to the number of reads you want to make consecutively. This can cause errors with your reads since the SDRAM controller may be in a strange state. You can see the example I have here.

Another thing to check would also be the WORDS/SYMBOLS thing I mentioned above.

Using the Avalon Slave BFM was the proper way to debug the interface.

This is absolutely right, but I couldn’t make head or tail out of it :slight_smile:. I spent a few days struggling to understand this, but since I have no idea how to use Modelsim also, I just abandoned this. If you have any links or references for beginners on how to use BFM, I’d appreciate it!

I was curious and I tried out your scenario with two fpga to sdram interfaces and it worked fine with both of them. Here is my design:

What I did:

  • avm_m0_address is set with a parameter so that I can have a different read address for each of the read masters.
  • do_read is wired to both the read masters outside of platform designer because I couldn’t figure out how to connect conduits between 3 different components (works for 2).
  • avalon_control_0 triggers do_read to be high for exactly 1 cycle to signal to both read masters to initiate reading.
  • out_data1 is connected to LED[7:5] and out_data2 is connected to LED[3:1] to see that the values are being read correctly.
  • VERY IMPORTANT: The rbf file needs to be flashed in u-boot so that the FPGA-SDRAM config bits are enabled for f2h_sdram0 and f2h_sdram1.

Hope that helps.

Thanks Jake.

I did get it working. It was my RTL that was lacking. I had bugs as well as needing the burst.