Reproduce 100G example design failed

[habrade]

Hello Gabriele,

I am trying to reproduce the 100G example design on VCU118 with a Mellanox ConnectX-5 NIC. I followed the instructions in the project's README to configure the QP, and I am able to send ROCE packets using send or write operations — the PC receives them correctly, and I can capture them with tcpdump.

However, after sending just one ROCE packet, the FPGA's network interface becomes completely unresponsive — no further UDP (both CM and ROCE) or ICMP packets are sent from the FPGA, even though the PC have already responded a normal ACKs. I am currently using ILA to debug this issue, but I haven't found a solution yet.

Do you have any suggestions on how to resolve this problem?

regards,
Sheng

[habrade]

BTW, the code commit is: ebc9d3a

[Gabriele-bot]

Hello Sheng,

I am glad that some one is using my code (considering the mess that is right now).

To get back to your problem, that commit uses RoCE_udp_tx_simple.v to pack the infiniband headers into the axi stream udp payload. That version gave me quite a lot of head ache and it the end I swapped back the the previous version (RoCE_udp_tx.v), which is more complex, but is faster and more reliable.

Can you try to swap these lines

100G-verilog-RoCEv2-lite/rtl/RoCE_minimal_stack.sv

Lines 1444 to 1517 in ebc9d3a

	RoCE_udp_tx_simple #(
	.DATA_WIDTH(DATA_WIDTH)
	) RoCE_udp_tx_instance (
	.clk (clk),
	.rst (rst),
	.s_roce_bth_valid (roce_bth_valid),
	.s_roce_bth_ready (roce_bth_ready),
	.s_roce_bth_op_code (roce_bth_op_code),
	.s_roce_bth_p_key (roce_bth_p_key),
	.s_roce_bth_psn (roce_bth_psn),
	.s_roce_bth_dest_qp (roce_bth_dest_qp),
	.s_roce_bth_ack_req (roce_bth_ack_req),
	//.s_roce_reth_valid (roce_reth_valid),
	//.s_roce_reth_ready (roce_reth_ready),
	.s_roce_reth_v_addr (roce_reth_v_addr),
	.s_roce_reth_r_key (roce_reth_r_key),
	.s_roce_reth_length (roce_reth_length),
	//.s_roce_immdh_valid (roce_immdh_valid),
	//.s_roce_immdh_ready (roce_immdh_ready),
	.s_roce_immdh_data (roce_immdh_data),
	//.s_eth_dest_mac (48'd0),
	//.s_eth_src_mac (48'd0),
	//.s_eth_type (16'd0),
	.s_ip_version (4'd4),
	.s_ip_ihl (4'd0),
	.s_ip_dscp (6'd0),
	.s_ip_ecn (2'd0),
	.s_ip_identification (16'd0),
	.s_ip_flags (3'b001),
	.s_ip_fragment_offset (13'd0),
	.s_ip_ttl (8'h40),
	.s_ip_protocol (8'h11),
	.s_ip_header_checksum (16'd0),
	.s_ip_source_ip (loc_ip_addr),
	.s_ip_dest_ip (ip_dest_ip),
	.s_udp_source_port (ROCE_UDP_TX_SOURCE_PORT),
	.s_udp_dest_port (RoCE_udp_port),
	.s_udp_length (udp_length),
	.s_udp_checksum (16'h0000),
	.s_roce_payload_axis_tdata (m_roce_payload_axis_tdata),
	.s_roce_payload_axis_tkeep (m_roce_payload_axis_tkeep),
	.s_roce_payload_axis_tvalid (m_roce_payload_axis_tvalid),
	.s_roce_payload_axis_tready (m_roce_payload_axis_tready),
	.s_roce_payload_axis_tlast (m_roce_payload_axis_tlast),
	.s_roce_payload_axis_tuser (m_roce_payload_axis_tuser),
	.m_udp_hdr_valid (roce_tx_udp_hdr_valid),
	.m_udp_hdr_ready (roce_tx_udp_hdr_ready),
	//.m_eth_dest_mac (roce_tx_eth_dest_mac),
	//.m_eth_src_mac (roce_tx_eth_src_mac),
	//.m_eth_type (roce_tx_eth_type),
	.m_ip_version (roce_tx_ip_version),
	.m_ip_ihl (roce_tx_ip_ihl),
	.m_ip_dscp (roce_tx_ip_dscp),
	.m_ip_ecn (roce_tx_ip_ecn),
	.m_ip_length (roce_tx_ip_length),
	.m_ip_identification (roce_tx_ip_identification),
	.m_ip_flags (roce_tx_ip_flags),
	.m_ip_fragment_offset (roce_tx_ip_fragment_offset),
	.m_ip_ttl (roce_tx_ip_ttl),
	.m_ip_protocol (roce_tx_ip_protocol),
	.m_ip_header_checksum (roce_tx_ip_header_checksum),
	.m_ip_source_ip (roce_tx_ip_source_ip),
	.m_ip_dest_ip (roce_tx_ip_dest_ip),
	.m_udp_source_port (roce_tx_udp_source_port),
	.m_udp_dest_port (roce_tx_udp_dest_port),
	.m_udp_length (roce_tx_udp_length),
	.m_udp_checksum (roce_tx_udp_checksum),
	.m_udp_payload_axis_tdata (roce_tx_udp_payload_axis_tdata),
	.m_udp_payload_axis_tkeep (roce_tx_udp_payload_axis_tkeep),
	.m_udp_payload_axis_tvalid (roce_tx_udp_payload_axis_tvalid),
	.m_udp_payload_axis_tready (roce_tx_udp_payload_axis_tready),
	.m_udp_payload_axis_tlast (roce_tx_udp_payload_axis_tlast),
	.m_udp_payload_axis_tuser (roce_tx_udp_payload_axis_tuser)
	);

with these ones?

100G-verilog-RoCEv2-lite/rtl/RoCE_minimal_stack.sv

Lines 1367 to 1443 in fd1d7f1

	RoCE_udp_tx #(
	.DATA_WIDTH(DATA_WIDTH)
	) RoCE_udp_tx_instance (
	.clk (clk),
	.rst (rst),
	.s_roce_bth_valid (roce_bth_valid),
	.s_roce_bth_ready (roce_bth_ready),
	.s_roce_bth_op_code (roce_bth_op_code),
	.s_roce_bth_p_key (roce_bth_p_key),
	.s_roce_bth_psn (roce_bth_psn),
	.s_roce_bth_dest_qp (roce_bth_dest_qp),
	.s_roce_bth_ack_req (roce_bth_ack_req),
	.s_roce_reth_valid (roce_reth_valid),
	.s_roce_reth_ready (roce_reth_ready),
	.s_roce_reth_v_addr (roce_reth_v_addr),
	.s_roce_reth_r_key (roce_reth_r_key),
	.s_roce_reth_length (roce_reth_length),
	.s_roce_immdh_valid (roce_immdh_valid),
	.s_roce_immdh_ready (roce_immdh_ready),
	.s_roce_immdh_data (roce_immdh_data),
	.s_eth_dest_mac (48'd0),
	.s_eth_src_mac (48'd0),
	.s_eth_type (16'd0),
	.s_ip_version (4'd4),
	.s_ip_ihl (4'd0),
	.s_ip_dscp (6'd0),
	.s_ip_ecn (2'd0),
	.s_ip_identification (16'd0),
	.s_ip_flags (3'b001),
	.s_ip_fragment_offset (13'd0),
	.s_ip_ttl (8'h40),
	.s_ip_protocol (8'h11),
	.s_ip_header_checksum (16'd0),
	.s_ip_source_ip (loc_ip_addr),
	.s_ip_dest_ip (ip_dest_ip),
	.s_udp_source_port (ROCE_UDP_TX_SOURCE_PORT),
	.s_udp_dest_port (RoCE_udp_port),
	.s_udp_length (udp_length),
	.s_udp_checksum (16'h0000),
	.s_roce_payload_axis_tdata (m_roce_payload_axis_tdata),
	.s_roce_payload_axis_tkeep (m_roce_payload_axis_tkeep),
	.s_roce_payload_axis_tvalid (m_roce_payload_axis_tvalid),
	.s_roce_payload_axis_tready (m_roce_payload_axis_tready),
	.s_roce_payload_axis_tlast (m_roce_payload_axis_tlast),
	.s_roce_payload_axis_tuser (m_roce_payload_axis_tuser),
	.m_udp_hdr_valid (roce_tx_udp_hdr_valid),
	.m_udp_hdr_ready (roce_tx_udp_hdr_ready),
	.m_eth_dest_mac (roce_tx_eth_dest_mac),
	.m_eth_src_mac (roce_tx_eth_src_mac),
	.m_eth_type (roce_tx_eth_type),
	.m_ip_version (roce_tx_ip_version),
	.m_ip_ihl (roce_tx_ip_ihl),
	.m_ip_dscp (roce_tx_ip_dscp),
	.m_ip_ecn (roce_tx_ip_ecn),
	.m_ip_length (roce_tx_ip_length),
	.m_ip_identification (roce_tx_ip_identification),
	.m_ip_flags (roce_tx_ip_flags),
	.m_ip_fragment_offset (roce_tx_ip_fragment_offset),
	.m_ip_ttl (roce_tx_ip_ttl),
	.m_ip_protocol (roce_tx_ip_protocol),
	.m_ip_header_checksum (roce_tx_ip_header_checksum),
	.m_ip_source_ip (roce_tx_ip_source_ip),
	.m_ip_dest_ip (roce_tx_ip_dest_ip),
	.m_udp_source_port (roce_tx_udp_source_port),
	.m_udp_dest_port (roce_tx_udp_dest_port),
	.m_udp_length (roce_tx_udp_length),
	.m_udp_checksum (roce_tx_udp_checksum),
	.m_udp_payload_axis_tdata (roce_tx_udp_payload_axis_tdata),
	.m_udp_payload_axis_tkeep (roce_tx_udp_payload_axis_tkeep),
	.m_udp_payload_axis_tvalid (roce_tx_udp_payload_axis_tvalid),
	.m_udp_payload_axis_tready (roce_tx_udp_payload_axis_tready),
	.m_udp_payload_axis_tlast (roce_tx_udp_payload_axis_tlast),
	.m_udp_payload_axis_tuser (roce_tx_udp_payload_axis_tuser),
	.busy (busy),
	.error_payload_early_termination(error_payload_early_termination),
	.RoCE_udp_port(RoCE_udp_port)
	);

The repository is quite messy right now and we are also working on the 400G version. The code is quite prone to deadlocks, as you discovered.

If this solution doesn't work, I'll try to reproduce it and get back to you.

Cheers,
Gabriele

[habrade]

Hi Gabriele,

Thank you so much for your prompt reply!

I tried the solution you provided, and it worked perfectly — the issue is resolved!

I also tested your latest commit (i.e., fd1d7f1), and there is no deadlock issue; communication works flawlessly.

I'm currently conducting some research on RDMA on FPGA myself, and your work has been extremely helpful and valuable as a reference. Many thanks again!

best,
Sheng