[SOLVED] DE10-Nano Lightweight HPS to FPGA works fine but hangs with heavyweight HPS to FPGA

tl;dr - Writing to the base address of lightweight HPS to FPGA bridge works fine at 0xFF200000, but writing to the heavyweight HPS to FPGA bridge at 0xC0000000 causes linux to hang. Any ideas what I’m doing wrong?

I have a very simple design with a single address mapped to LEDs allowing for reading and writing. The verilog file is below along with a screenshot of platform designer (h2f_axi is set to 32 bits same as h2f_lw_axi).

module custom_leds
(
    input  logic        clk,                // clock.clk
    input  logic        reset,              // reset.reset
    
    // Memory mapped read/write slave interface
    input  logic        avs_s0_address,     // avs_s0.address
    input  logic        avs_s0_read,        // avs_s0.read
    input  logic        avs_s0_write,       // avs_s0.write
    output logic [31:0] avs_s0_readdata,    // avs_s0.readdata
    input  logic [31:0] avs_s0_writedata,   // avs_s0.writedata
    
    // The LED outputs
    output logic [7:0]  leds
);

// Read operations performed on the Avalon-MM Slave interface
always_comb begin
    if (avs_s0_read) begin
        case (avs_s0_address)
            1'b0    : avs_s0_readdata = {24'b0, leds};
            default : avs_s0_readdata = 'x; 
        endcase
    end else begin
        avs_s0_readdata = 'x; 
    end 
end

// Write operations performed on the Avalon-MM Slave interface
always_ff @ (posedge clk) begin
    if (reset) begin
        leds <= '0; 
    end else if (avs_s0_write) begin
        case (avs_s0_address)
            1'b0    : leds <= avs_s0_writedata;
            default : leds <= leds;
        endcase
    end 
end

endmodule // custom_leds

When I run the code below, it just hangs and is non-responsive until I restart the device. You can see the values I used for the lightweight bridge which works perfectly without changing anything in the design.

#include "hps_0.h"
/* header includes the following:
#define CUSTOM_LEDS_0_COMPONENT_TYPE custom_leds
#define CUSTOM_LEDS_0_COMPONENT_NAME custom_leds_0
#define CUSTOM_LEDS_0_BASE 0x0
#define CUSTOM_LEDS_0_SPAN 8
#define CUSTOM_LEDS_0_END 0x7
*/

#include <iomanip>
#include <iostream>

#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
  // if (argc != 2) {
  //   std::cerr << "Missing argument on how many times to blink led."
  //             << std::endl;
  //   return -1;
  // }
  // constexpr uint32_t HPS_TO_FPGA_LW_BASE = 0xFF200000;
  // constexpr uint32_t HPS_TO_FPGA_LW_SPAN = 0x0020000;

  constexpr uint32_t HPS_TO_FPGA_BASE = 0xC0000000;
  constexpr uint32_t HPS_TO_FPGA_SPAN = 0x40000000;
  constexpr uint32_t HPS_TO_FPGA_MASK = HPS_TO_FPGA_SPAN - 1;

  // int blink_times = std::stoi(argv[1]);

  int devmem_fd = open("/dev/mem", O_RDWR | O_SYNC);
  if (devmem_fd < 0) {
    std::cerr << "Couldn't open /dev/mem\n";
    return -1; 
  }

  auto bridge_map = static_cast<uint32_t *>( 
      mmap(nullptr, HPS_TO_FPGA_SPAN, PROT_READ | PROT_WRITE, MAP_SHARED,
           devmem_fd, HPS_TO_FPGA_BASE));

  if (bridge_map == MAP_FAILED) {
    std::cerr << "Couldn't create mmap.\n";
    close(devmem_fd);
    return -2; 
  }

  uint32_t *custom_led_map =
      bridge_map + (static_cast<uint32_t>(0x0 + CUSTOM_LEDS_0_BASE) &
                    static_cast<uint32_t>(HPS_TO_FPGA_MASK));
  //*custom_led_map = 0x55; // Fails
  std::cout << *custom_led_map << std::endl; // Also fails

  std::cout << "Done.\n";

  int result = munmap(bridge_map, HPS_TO_FPGA_SPAN);
  if (result < 0) {
    std::cerr << "Couldn't unmmap.\n";
    close(devmem_fd);
    return -3; 
  }

  close(devmem_fd);

  return 0;
}

All the bridges are enabled as you can see below.

root@de10-15aug21:~/fat/extlinux# cat /sys/class/fpga_bridge/*/state                                                                       
enabled
enabled
enabled
enabled

I have also restricted the kernel to use only 512MB ram by adding mem=512M to the kernel boot options.

What am I doing wrong? Any help will be greatly appreciated!

Okay, so looks like there wasn’t really a problem. Everything listed above works perfectly. The actual issue was with the conversion of the output .sof file to .rbf file. I was using a saved conversion setup data file from another project. The paths in the .cof file are absolute, so even though my output file (.sof) was correct, the .rbf which was being generated is from the original GHRD which didn’t have the heavyweight bus enabled.

I converted the programming file without using a saved config and it works brilliantly :slight_smile: