Skip to content

[feature] ASIC-focused multicast replication and dendrite API #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 28 commits into
base: main
Choose a base branch
from

Conversation

zeeshanlakhani
Copy link
Contributor

@zeeshanlakhani zeeshanlakhani commented Apr 1, 2025

This brings multicast group management and hardware-accelerated forwarding to Dendrite, providing the foundation for our multicast networking stack.

The PR includes:

API Layer:

  • Multicast group operations and lifecycle management
  • Integrated Source-Specific Multicast (SSM) support
  • Bulk reset functionality for clearing multicast state
  • Endpoints designed for integration with higher-level orchestration

Hardware Integration:

  • Updated sidecar.p4 for native Tofino ASIC multicast replication
  • ASIC table management for efficient multicast forwarding
  • Geneve option parsing for multicast tagging
  • Egress table handling for decapped packets and MAC rewriting
  • Range-based port bitmap matching for multicast egress (for decap)

Network Processing:

  • Leverages the Tofino packet replication engine for replication
  • Handles proper encapsulation/decapsulation flows
  • Bifurcated multicast handling: Separate processing paths for internal admin-scoped IPv6 groups vs external IPv4/IPv6 groups
  • Multiple replication scenarios:
    • Unencapped external multicast → encapsulated for underlay transport
    • Encapped IPv6 multicast → underlay-only replication (infrastructure traffic)
    • Encapped IPv6 multicast → external-only replication with decapsulation
    • Bifurcated replication: Both external members (decapped) and underlay members (encapped) from a group
  • Admin-scoped NAT target validation: External groups properly reference internal admin-scoped groups

Validation:

  • Extensive iperf3 testing shows no performance regression between baseline and multicast-enhanced versions
  • Comprehensive integration test suite covering:
    • All replication scenarios (external-only, underlay-only, bifurcated)
    • Encapsulation/decapsulation flows with Geneve multicast tagging
    • Source-Specific Multicast (SSM) filtering with exact and prefix matching
    • Admin-scoped NAT target validation and lifecycle management
    • Performance validation with concurrent packet replication
    • Edge cases including TTL handling, MAC derivation, and VLAN propagation

Context:

  • Implements the dendrite side of the bifurcated multicast design, handling both internal admin-scoped IPv6 groups (admin-local, site-local, and organization-local scopes) and external IPv4/IPv6 groups with NAT encapsulation
  • External multicast traffic gets encapsulated for underlay transport, then properly decapped and MAC-rewritten for local delivery
  • Internal groups stay within the admin scope for rack-local multicast communication
  • Supports complex replication patterns where single groups can serve both customer traffic (external, decapped) and infrastructure traffic (underlay, encapsulated)

Associated PRs

@zeeshanlakhani zeeshanlakhani force-pushed the zl/p4-mcast branch 3 times, most recently from 4e2fc4b to 1ce4908 Compare April 2, 2025 02:40
Includes:
    * Multicast Group API management: add, modify, delete, reset, SSM handling
    * sidecar.p4 tofino_asic updates to integrate multicast packet replication
    * Table mangagement for ASIC switch tables for multicast
    * integration tests and test utility additions
@zeeshanlakhani zeeshanlakhani changed the title wip [feature] ASIC-focused 1st draft of multicast PRE Apr 15, 2025
@zeeshanlakhani zeeshanlakhani marked this pull request as ready for review April 15, 2025 19:37
@zeeshanlakhani
Copy link
Contributor Author

I'll add some notes to the PR tomorrow morning (to explain through pieces).

Copy link
Contributor

@rcgoodfellow rcgoodfellow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Zeeshan. I've provided some comments on the API and am about half way through the P4 code. Have to run, but wanted to get these initial comments in.

( 0, false, true, true, _, USER_SPACE_SERVICE_PORT, true, _, _, _, _, _, _ ) : forward_from_userspace;
( 0, false, false, _, _, _, false, true, _, _, _, _, _ ) : forward_to_userspace;
( 0, false, false, _, true, _, _, _, _, _, _, _, _ ) : forward_to_userspace;
// Link-local multicast
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're changing quite a bit around link-local multicast, we need to test this e2e with ddm.

Copy link

@FelixMcFelix FelixMcFelix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this Zeeshan, some notes so far. I haven't yet made it through dpd/src/mcast.rs and the table management code yet.

Copy link

@FelixMcFelix FelixMcFelix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Zeeshan, I've had a look mainly focussing on the new changes and dpd/drc/mcast/*.rs. I do like the removal of v4 multicast from the routing table, and then relying on the NAT target to make use of it from external ports.

Copy link

@FelixMcFelix FelixMcFelix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for being patient. I think I'm happy with the API shape, just some points on cleanup and lock hygiene.

Includes:
* kyle's patch on scopedguard/group changes (after modifications) with slight mod on del
@zeeshanlakhani
Copy link
Contributor Author

@FelixMcFelix should now be in a good spot API-wise.

Comment on lines 1416 to 1418
DpdError::Invalid(
"external group ID should have been pre-allocated".to_string(),
)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like an invariant maintained server-side, rather than something derived from poor user input. DpdError::Invalid will convert into an HTTP 400, whereas this seems like a 500 on that note. DpdError::Other might fill that niche?

Comment on lines 1439 to 1441
DpdError::Invalid(
"underlay group ID should have been pre-allocated".to_string(),
)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto here on something which can convert to a 500?

Copy link

@FelixMcFelix FelixMcFelix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not likely to spot much more at this point, but I think things are generally in good shape. The last error code changes are minor. Thanks!

@zeeshanlakhani
Copy link
Contributor Author

To determine if we can target main instead of a multicast branch, I ran some iperf3 benchmarks between two instances (8 vCPUs/32 GiB RAM | Debian 12), comparing a recent Omicron commit (#94dda3c29fd3680561de0ae2e859d9f01bad0107) to another running commit #4e6889a442f78ad18f1d7de01f55fbb3344d75a3 on branch zl/dendrite-mcast.

Basic TCP: iperf3 -c 172.30.0.5 -t 30

client omicron ~ 94dda3c
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 772 MBytes 6.47 Gbits/sec 169 608 KBytes
[ 5] 1.00-2.00 sec 740 MBytes 6.21 Gbits/sec 136 485 KBytes
[ 5] 2.00-3.00 sec 905 MBytes 7.59 Gbits/sec 1364 331 KBytes
[ 5] 3.00-4.00 sec 898 MBytes 7.54 Gbits/sec 1858 526 KBytes
[ 5] 4.00-5.00 sec 969 MBytes 8.13 Gbits/sec 1866 583 KBytes
[ 5] 5.00-6.00 sec 978 MBytes 8.20 Gbits/sec 2203 607 KBytes
[ 5] 6.00-7.00 sec 896 MBytes 7.52 Gbits/sec 1439 576 KBytes
[ 5] 7.00-8.00 sec 955 MBytes 8.01 Gbits/sec 2075 827 KBytes
[ 5] 8.00-9.00 sec 938 MBytes 7.87 Gbits/sec 2000 387 KBytes
[ 5] 9.00-10.00 sec 978 MBytes 8.21 Gbits/sec 1471 611 KBytes
[ 5] 10.00-11.00 sec 972 MBytes 8.16 Gbits/sec 1940 438 KBytes
[ 5] 11.00-12.00 sec 978 MBytes 8.20 Gbits/sec 1416 730 KBytes
[ 5] 12.00-13.00 sec 937 MBytes 7.86 Gbits/sec 2054 619 KBytes
[ 5] 13.00-14.00 sec 986 MBytes 8.27 Gbits/sec 1822 461 KBytes
[ 5] 14.00-15.00 sec 963 MBytes 8.08 Gbits/sec 2092 578 KBytes
[ 5] 15.00-16.00 sec 958 MBytes 8.03 Gbits/sec 1791 424 KBytes
[ 5] 16.00-17.00 sec 986 MBytes 8.28 Gbits/sec 1689 455 KBytes
[ 5] 17.00-18.00 sec 948 MBytes 7.95 Gbits/sec 1344 601 KBytes
[ 5] 18.00-19.00 sec 985 MBytes 8.26 Gbits/sec 1915 626 KBytes
[ 5] 19.00-20.00 sec 968 MBytes 8.12 Gbits/sec 1148 757 KBytes
[ 5] 20.00-21.00 sec 1003 MBytes 8.41 Gbits/sec 2026 605 KBytes
[ 5] 21.00-22.00 sec 1006 MBytes 8.44 Gbits/sec 1407 457 KBytes
[ 5] 22.00-23.00 sec 1.07 GBytes 9.21 Gbits/sec 885 445 KBytes
[ 5] 23.00-24.00 sec 978 MBytes 8.20 Gbits/sec 2350 414 KBytes
[ 5] 24.00-25.00 sec 1.02 GBytes 8.80 Gbits/sec 3797 628 KBytes
[ 5] 25.00-26.00 sec 1.21 GBytes 10.4 Gbits/sec 2269 549 KBytes
[ 5] 26.00-27.00 sec 1.09 GBytes 9.35 Gbits/sec 2156 819 KBytes
[ 5] 27.00-28.00 sec 995 MBytes 8.34 Gbits/sec 1376 553 KBytes
[ 5] 28.00-29.00 sec 944 MBytes 7.92 Gbits/sec 2209 409 KBytes
[ 5] 29.00-30.00 sec 938 MBytes 7.87 Gbits/sec 2074 624 KBytes
---[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 28.4 GBytes 8.13 Gbits/sec 52341 sender
[ 5] 0.00-30.00 sec 28.4 GBytes 8.13 Gbits/sec receiver

client mcast ~ 4e6889a:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 721 MBytes 6.05 Gbits/sec 263 591 KBytes
[ 5] 1.00-2.00 sec 699 MBytes 5.86 Gbits/sec 33 455 KBytes
[ 5] 2.00-3.00 sec 692 MBytes 5.81 Gbits/sec 0 512 KBytes
[ 5] 3.00-4.00 sec 927 MBytes 7.78 Gbits/sec 1240 595 KBytes
[ 5] 4.00-5.00 sec 973 MBytes 8.16 Gbits/sec 1425 390 KBytes
[ 5] 5.00-6.00 sec 965 MBytes 8.09 Gbits/sec 1327 584 KBytes
[ 5] 6.00-7.00 sec 1.21 GBytes 10.4 Gbits/sec 2382 595 KBytes
[ 5] 7.00-8.00 sec 1.28 GBytes 11.0 Gbits/sec 1989 574 KBytes
[ 5] 8.00-9.00 sec 1.21 GBytes 10.4 Gbits/sec 1996 577 KBytes
[ 5] 9.00-10.00 sec 920 MBytes 7.72 Gbits/sec 1378 543 KBytes
[ 5] 10.00-11.00 sec 989 MBytes 8.29 Gbits/sec 1070 636 KBytes
[ 5] 11.00-12.00 sec 989 MBytes 8.29 Gbits/sec 1976 591 KBytes
[ 5] 12.00-13.00 sec 977 MBytes 8.20 Gbits/sec 1260 590 KBytes
[ 5] 13.00-14.00 sec 965 MBytes 8.10 Gbits/sec 1574 571 KBytes
[ 5] 14.00-15.00 sec 979 MBytes 8.22 Gbits/sec 1841 509 KBytes
[ 5] 15.00-16.00 sec 954 MBytes 8.00 Gbits/sec 1463 464 KBytes
[ 5] 16.00-17.00 sec 981 MBytes 8.23 Gbits/sec 1824 585 KBytes
[ 5] 17.00-18.00 sec 993 MBytes 8.33 Gbits/sec 1837 509 KBytes
[ 5] 18.00-19.00 sec 982 MBytes 8.23 Gbits/sec 1731 556 KBytes
[ 5] 19.00-20.00 sec 951 MBytes 7.98 Gbits/sec 2238 602 KBytes
[ 5] 20.00-21.00 sec 979 MBytes 8.21 Gbits/sec 1997 428 KBytes
[ 5] 21.00-22.00 sec 944 MBytes 7.91 Gbits/sec 1088 598 KBytes
[ 5] 22.00-23.00 sec 1.01 GBytes 8.65 Gbits/sec 1303 694 KBytes
[ 5] 23.00-24.00 sec 1010 MBytes 8.47 Gbits/sec 1236 571 KBytes
[ 5] 24.00-25.00 sec 1.14 GBytes 9.78 Gbits/sec 830 634 KBytes
[ 5] 25.00-26.00 sec 1023 MBytes 8.58 Gbits/sec 1588 403 KBytes
[ 5] 26.00-27.00 sec 968 MBytes 8.12 Gbits/sec 1773 561 KBytes
[ 5] 27.00-28.00 sec 975 MBytes 8.18 Gbits/sec 1957 409 KBytes
[ 5] 28.00-29.00 sec 945 MBytes 7.93 Gbits/sec 1738 563 KBytes
[ 5] 29.00-30.00 sec 938 MBytes 7.87 Gbits/sec 2066 284 KBytes
---[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 28.7 GBytes 8.23 Gbits/sec 44423 sender
[ 5] 0.00-30.00 sec 28.7 GBytes 8.23 Gbits/sec receiver

server omicron ~ 94dda3c
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 772 MBytes 6.47 Gbits/sec
[ 5] 1.00-2.00 sec 740 MBytes 6.21 Gbits/sec
[ 5] 2.00-3.00 sec 903 MBytes 7.57 Gbits/sec
[ 5] 3.00-4.00 sec 899 MBytes 7.54 Gbits/sec
[ 5] 4.00-5.00 sec 968 MBytes 8.12 Gbits/sec
[ 5] 5.00-6.00 sec 978 MBytes 8.20 Gbits/sec
[ 5] 6.00-7.00 sec 896 MBytes 7.51 Gbits/sec
[ 5] 7.00-8.00 sec 956 MBytes 8.01 Gbits/sec
[ 5] 8.00-9.00 sec 937 MBytes 7.87 Gbits/sec
[ 5] 9.00-10.00 sec 979 MBytes 8.21 Gbits/sec
[ 5] 10.00-11.00 sec 972 MBytes 8.16 Gbits/sec
[ 5] 11.00-12.00 sec 978 MBytes 8.20 Gbits/sec
[ 5] 12.00-13.00 sec 937 MBytes 7.86 Gbits/sec
[ 5] 13.00-14.00 sec 986 MBytes 8.27 Gbits/sec
[ 5] 14.00-15.00 sec 963 MBytes 8.08 Gbits/sec
[ 5] 15.00-16.00 sec 957 MBytes 8.03 Gbits/sec
[ 5] 16.00-17.00 sec 987 MBytes 8.28 Gbits/sec
[ 5] 17.00-18.00 sec 948 MBytes 7.95 Gbits/sec
[ 5] 18.00-19.00 sec 985 MBytes 8.26 Gbits/sec
[ 5] 19.00-20.00 sec 968 MBytes 8.12 Gbits/sec
[ 5] 20.00-21.00 sec 1002 MBytes 8.41 Gbits/sec
[ 5] 21.00-22.00 sec 1003 MBytes 8.42 Gbits/sec
[ 5] 22.00-23.00 sec 1.07 GBytes 9.23 Gbits/sec
[ 5] 23.00-24.00 sec 978 MBytes 8.21 Gbits/sec
[ 5] 24.00-25.00 sec 1.02 GBytes 8.80 Gbits/sec
[ 5] 25.00-26.00 sec 1.21 GBytes 10.4 Gbits/sec
[ 5] 26.00-27.00 sec 1.09 GBytes 9.33 Gbits/sec
[ 5] 27.00-28.00 sec 996 MBytes 8.36 Gbits/sec
[ 5] 28.00-29.00 sec 943 MBytes 7.91 Gbits/sec
[ 5] 29.00-30.00 sec 938 MBytes 7.87 Gbits/sec
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate
[ 5] (sender statistics not available)
[ 5] 0.00-30.00 sec 28.4 GBytes 8.13 Gbits/sec receiver

server mcast ~ 4e6889a:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 721 MBytes 6.04 Gbits/sec
[ 5] 1.00-2.00 sec 699 MBytes 5.86 Gbits/sec
[ 5] 2.00-3.00 sec 692 MBytes 5.81 Gbits/sec
[ 5] 3.00-4.00 sec 925 MBytes 7.75 Gbits/sec
[ 5] 4.00-5.00 sec 973 MBytes 8.16 Gbits/sec
[ 5] 5.00-6.00 sec 964 MBytes 8.09 Gbits/sec
[ 5] 6.00-7.00 sec 1.21 GBytes 10.4 Gbits/sec
[ 5] 7.00-8.00 sec 1.28 GBytes 11.0 Gbits/sec
[ 5] 8.00-9.00 sec 1.21 GBytes 10.4 Gbits/sec
[ 5] 9.00-10.00 sec 921 MBytes 7.72 Gbits/sec
[ 5] 10.00-11.00 sec 989 MBytes 8.29 Gbits/sec
[ 5] 11.00-12.00 sec 988 MBytes 8.29 Gbits/sec
[ 5] 12.00-13.00 sec 978 MBytes 8.20 Gbits/sec
[ 5] 13.00-14.00 sec 965 MBytes 8.10 Gbits/sec
[ 5] 14.00-15.00 sec 979 MBytes 8.21 Gbits/sec
[ 5] 15.00-16.00 sec 954 MBytes 8.00 Gbits/sec
[ 5] 16.00-17.00 sec 981 MBytes 8.22 Gbits/sec
[ 5] 17.00-18.00 sec 994 MBytes 8.34 Gbits/sec
[ 5] 18.00-19.00 sec 981 MBytes 8.23 Gbits/sec
[ 5] 19.00-20.00 sec 951 MBytes 7.98 Gbits/sec
[ 5] 20.00-21.00 sec 979 MBytes 8.21 Gbits/sec
[ 5] 21.00-22.00 sec 943 MBytes 7.91 Gbits/sec
[ 5] 22.00-23.00 sec 1.00 GBytes 8.63 Gbits/sec
[ 5] 23.00-24.00 sec 1012 MBytes 8.49 Gbits/sec
[ 5] 24.00-25.00 sec 1.14 GBytes 9.76 Gbits/sec
[ 5] 25.00-26.00 sec 1.00 GBytes 8.61 Gbits/sec
[ 5] 26.00-27.00 sec 969 MBytes 8.13 Gbits/sec
[ 5] 27.00-28.00 sec 974 MBytes 8.17 Gbits/sec
[ 5] 28.00-29.00 sec 946 MBytes 7.94 Gbits/sec
[ 5] 29.00-30.00 sec 937 MBytes 7.86 Gbits/sec
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate
[ 5] (sender statistics not available)
[ 5] 0.00-30.00 sec 28.7 GBytes 8.23 Gbits/sec receiver

Multiple TCP streams: iperf3 -c 172.30.0.5 -P 8 -t 30

client omicron ~ 94dda3c:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 92.6 MBytes 776 Mbits/sec 809 41.0 KBytes
[ 7] 0.00-1.00 sec 150 MBytes 1.26 Gbits/sec 1030 76.4 KBytes
[ 9] 0.00-1.00 sec 131 MBytes 1.10 Gbits/sec 1196 431 KBytes
[ 11] 0.00-1.00 sec 148 MBytes 1.24 Gbits/sec 989 173 KBytes
[ 13] 0.00-1.00 sec 113 MBytes 950 Mbits/sec 724 2.83 KBytes
[ 15] 0.00-1.00 sec 95.3 MBytes 800 Mbits/sec 1119 191 KBytes
[ 17] 0.00-1.00 sec 76.3 MBytes 640 Mbits/sec 740 1.41 KBytes
[ 19] 0.00-1.00 sec 120 MBytes 1.01 Gbits/sec 475 32.5 KBytes
[SUM] 0.00-1.00 sec 926 MBytes 7.77 Gbits/sec 7082
---[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 4.41 GBytes 1.26 Gbits/sec 45988 sender
[ 5] 0.00-30.00 sec 4.40 GBytes 1.26 Gbits/sec receiver
[ 7] 0.00-30.00 sec 4.76 GBytes 1.36 Gbits/sec 42299 sender
[ 7] 0.00-30.00 sec 4.76 GBytes 1.36 Gbits/sec receiver
[ 9] 0.00-30.00 sec 4.49 GBytes 1.28 Gbits/sec 48417 sender
[ 9] 0.00-30.00 sec 4.48 GBytes 1.28 Gbits/sec receiver
[ 11] 0.00-30.00 sec 4.03 GBytes 1.15 Gbits/sec 50683 sender
[ 11] 0.00-30.00 sec 4.02 GBytes 1.15 Gbits/sec receiver
[ 13] 0.00-30.00 sec 3.32 GBytes 950 Mbits/sec 45299 sender
[ 13] 0.00-30.00 sec 3.32 GBytes 950 Mbits/sec receiver
[ 15] 0.00-30.00 sec 2.50 GBytes 715 Mbits/sec 38295 sender
[ 15] 0.00-30.00 sec 2.50 GBytes 715 Mbits/sec receiver
[ 17] 0.00-30.00 sec 2.20 GBytes 629 Mbits/sec 35567 sender
[ 17] 0.00-30.00 sec 2.20 GBytes 629 Mbits/sec receiver
[ 19] 0.00-30.00 sec 1.70 GBytes 487 Mbits/sec 27335 sender
[ 19] 0.00-30.00 sec 1.70 GBytes 486 Mbits/sec receiver
[SUM] 0.00-30.00 sec 27.4 GBytes 7.85 Gbits/sec 333883 sender
[SUM] 0.00-30.00 sec 27.4 GBytes 7.84 Gbits/sec receiver

client mcast ~ 4e6889a:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 149 MBytes 1.25 Gbits/sec 1462 82.0 KBytes
[ 7] 0.00-1.00 sec 175 MBytes 1.47 Gbits/sec 2144 189 KBytes
[ 9] 0.00-1.00 sec 91.5 MBytes 767 Mbits/sec 1476 160 KBytes
[ 11] 0.00-1.00 sec 131 MBytes 1.10 Gbits/sec 2384 9.90 KBytes
[ 13] 0.00-1.00 sec 88.6 MBytes 743 Mbits/sec 1490 36.8 KBytes
[ 15] 0.00-1.00 sec 150 MBytes 1.26 Gbits/sec 1931 103 KBytes
[ 17] 0.00-1.00 sec 197 MBytes 1.65 Gbits/sec 2846 43.8 KBytes
[ 19] 0.00-1.00 sec 89.2 MBytes 748 Mbits/sec 1266 25.5 KBytes
[SUM] 0.00-1.00 sec 1.05 GBytes 8.98 Gbits/sec 14999
---[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 4.88 GBytes 1.40 Gbits/sec 42191 sender
[ 5] 0.00-30.00 sec 4.88 GBytes 1.40 Gbits/sec receiver
[ 7] 0.00-30.00 sec 5.54 GBytes 1.59 Gbits/sec 47672 sender
[ 7] 0.00-30.00 sec 5.54 GBytes 1.59 Gbits/sec receiver
[ 9] 0.00-30.00 sec 4.59 GBytes 1.31 Gbits/sec 51847 sender
[ 9] 0.00-30.00 sec 4.59 GBytes 1.31 Gbits/sec receiver
[ 11] 0.00-30.00 sec 3.74 GBytes 1.07 Gbits/sec 49638 sender
[ 11] 0.00-30.00 sec 3.74 GBytes 1.07 Gbits/sec receiver
[ 13] 0.00-30.00 sec 2.74 GBytes 785 Mbits/sec 42323 sender
[ 13] 0.00-30.00 sec 2.74 GBytes 784 Mbits/sec receiver
[ 15] 0.00-30.00 sec 2.04 GBytes 583 Mbits/sec 34583 sender
[ 15] 0.00-30.00 sec 2.04 GBytes 583 Mbits/sec receiver
[ 17] 0.00-30.00 sec 2.03 GBytes 583 Mbits/sec 36841 sender
[ 17] 0.00-30.00 sec 2.03 GBytes 582 Mbits/sec receiver
[ 19] 0.00-30.00 sec 1.77 GBytes 506 Mbits/sec 30231 sender
[ 19] 0.00-30.00 sec 1.76 GBytes 505 Mbits/sec receiver
[SUM] 0.00-30.00 sec 27.3 GBytes 7.83 Gbits/sec 335326 sender
[SUM] 0.00-30.00 sec 27.3 GBytes 7.82 Gbits/sec receiver

server omicron ~ 94dda3c:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 91.0 MBytes 763 Mbits/sec
[ 8] 0.00-1.00 sec 149 MBytes 1.25 Gbits/sec
[ 10] 0.00-1.00 sec 129 MBytes 1.08 Gbits/sec
[ 12] 0.00-1.00 sec 146 MBytes 1.23 Gbits/sec
[ 14] 0.00-1.00 sec 113 MBytes 945 Mbits/sec
[ 16] 0.00-1.00 sec 93.4 MBytes 783 Mbits/sec
[ 18] 0.00-1.00 sec 75.1 MBytes 630 Mbits/sec
[ 20] 0.00-1.00 sec 119 MBytes 997 Mbits/sec
[SUM] 0.00-1.00 sec 915 MBytes 7.67 Gbits/sec
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate
[ 5] (sender statistics not available)
[ 5] 0.00-30.00 sec 4.40 GBytes 1.26 Gbits/sec receiver
[ 8] (sender statistics not available)
[ 8] 0.00-30.00 sec 4.76 GBytes 1.36 Gbits/sec receiver
[ 10] (sender statistics not available)
[ 10] 0.00-30.00 sec 4.48 GBytes 1.28 Gbits/sec receiver
[ 12] (sender statistics not available)
[ 12] 0.00-30.00 sec 4.02 GBytes 1.15 Gbits/sec receiver
[ 14] (sender statistics not available)
[ 14] 0.00-30.00 sec 3.32 GBytes 950 Mbits/sec receiver
[ 16] (sender statistics not available)
[ 16] 0.00-30.00 sec 2.50 GBytes 715 Mbits/sec receiver
[ 18] (sender statistics not available)
[ 18] 0.00-30.00 sec 2.20 GBytes 629 Mbits/sec receiver
[ 20] (sender statistics not available)
[ 20] 0.00-30.00 sec 1.70 GBytes 486 Mbits/sec receiver
[SUM] (sender statistics not available)
[SUM] 0.00-30.00 sec 27.4 GBytes 7.84 Gbits/sec receiver

server mcast ~ 4e6889a:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 148 MBytes 1.24 Gbits/sec
[ 8] 0.00-1.00 sec 173 MBytes 1.45 Gbits/sec
[ 10] 0.00-1.00 sec 89.7 MBytes 753 Mbits/sec
[ 12] 0.00-1.00 sec 129 MBytes 1.08 Gbits/sec
[ 14] 0.00-1.00 sec 87.6 MBytes 735 Mbits/sec
[ 16] 0.00-1.00 sec 148 MBytes 1.24 Gbits/sec
[ 18] 0.00-1.00 sec 195 MBytes 1.63 Gbits/sec
[ 20] 0.00-1.00 sec 87.8 MBytes 736 Mbits/sec
[SUM] 0.00-1.00 sec 1.03 GBytes 8.86 Gbits/sec
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate
[ 5] (sender statistics not available)
[ 5] 0.00-30.00 sec 4.88 GBytes 1.40 Gbits/sec receiver
[ 8] (sender statistics not available)
[ 8] 0.00-30.00 sec 5.54 GBytes 1.59 Gbits/sec receiver
[ 10] (sender statistics not available)
[ 10] 0.00-30.00 sec 4.59 GBytes 1.31 Gbits/sec receiver
[ 12] (sender statistics not available)
[ 12] 0.00-30.00 sec 3.74 GBytes 1.07 Gbits/sec receiver
[ 14] (sender statistics not available)
[ 14] 0.00-30.00 sec 2.74 GBytes 784 Mbits/sec receiver
[ 16] (sender statistics not available)
[ 16] 0.00-30.00 sec 2.04 GBytes 583 Mbits/sec receiver
[ 18] (sender statistics not available)
[ 18] 0.00-30.00 sec 2.03 GBytes 582 Mbits/sec receiver
[ 20] (sender statistics not available)
[ 20] 0.00-30.00 sec 1.76 GBytes 505 Mbits/sec receiver
[SUM] (sender statistics not available)
[SUM] 0.00-30.00 sec 27.3 GBytes 7.82 Gbits/sec receiver

TCP Bidir: iperf3 -c 172.30.0.5 --bidir -t 30

client omicron ~ 94dda3c:
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 5][TX-C] 0.00-1.00 sec 721 MBytes 6.05 Gbits/sec 1272 436 KBytes
[ 7][RX-C] 0.00-1.00 sec 614 MBytes 5.15 Gbits/sec
[ 5][TX-C] 1.00-2.00 sec 754 MBytes 6.33 Gbits/sec 1757 525 KBytes
[ 7][RX-C] 1.00-2.00 sec 618 MBytes 5.19 Gbits/sec
---[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-30.00 sec 20.5 GBytes 5.87 Gbits/sec 20844 sender
[ 5][TX-C] 0.00-30.00 sec 20.5 GBytes 5.87 Gbits/sec receiver
[ 7][RX-C] 0.00-30.00 sec 17.8 GBytes 5.10 Gbits/sec 17699 sender
[ 7][RX-C] 0.00-30.00 sec 17.8 GBytes 5.10 Gbits/sec receiver

client mcast ~ 4e6889a:
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 5][TX-C] 0.00-1.00 sec 839 MBytes 7.03 Gbits/sec 1966 526 KBytes
[ 7][RX-C] 0.00-1.00 sec 654 MBytes 5.48 Gbits/sec
[ 5][TX-C] 1.00-2.00 sec 868 MBytes 7.28 Gbits/sec 1246 522 KBytes
[ 7][RX-C] 1.00-2.00 sec 301 MBytes 2.53 Gbits/sec
---[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-30.00 sec 20.6 GBytes 5.89 Gbits/sec 14710 sender
[ 5][TX-C] 0.00-30.00 sec 20.6 GBytes 5.89 Gbits/sec receiver
[ 7][RX-C] 0.00-30.00 sec 17.7 GBytes 5.06 Gbits/sec 15267 sender
[ 7][RX-C] 0.00-30.00 sec 17.7 GBytes 5.06 Gbits/sec receiver

server omicron ~ 94dda3c:
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 5][RX-S] 0.00-1.00 sec 719 MBytes 6.03 Gbits/sec
[ 8][TX-S] 0.00-1.00 sec 616 MBytes 5.17 Gbits/sec 602 269 KBytes
[ 5][RX-S] 1.00-2.00 sec 754 MBytes 6.33 Gbits/sec
[ 8][TX-S] 1.00-2.00 sec 619 MBytes 5.19 Gbits/sec 1605 590 KBytes
---Test Complete. Summary Results:
[ ID][Role] Interval Transfer Bitrate Retr
[ 5] (sender statistics not available)
[ 5][RX-S] 0.00-30.00 sec 20.5 GBytes 5.87 Gbits/sec receiver
rcv_tcp_congestion cubic
[ 8][TX-S] 0.00-30.00 sec 17.8 GBytes 5.10 Gbits/sec 17699 sender
[ 8] (receiver statistics not available)
CPU Utilization: local/sender 56.6% (0.9%u/55.7%s), remote/receiver 30.1% (0.5%u/29.6%s)
CPU Utilization: local/receiver 56.6% (0.9%u/55.7%s), remote/sender 30.1% (0.5%u/29.6%s)

server mcast ~ 4e6889a:
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 5][RX-S] 0.00-1.00 sec 836 MBytes 7.01 Gbits/sec
[ 8][TX-S] 0.00-1.00 sec 656 MBytes 5.50 Gbits/sec 1916 346 KBytes
[ 5][RX-S] 1.00-2.00 sec 868 MBytes 7.28 Gbits/sec
[ 8][TX-S] 1.00-2.00 sec 304 MBytes 2.55 Gbits/sec 259 513 KBytes
---Test Complete. Summary Results:
[ ID][Role] Interval Transfer Bitrate Retr
[ 5] (sender statistics not available)
[ 5][RX-S] 0.00-30.00 sec 20.6 GBytes 5.89 Gbits/sec receiver
rcv_tcp_congestion cubic
[ 8][TX-S] 0.00-30.00 sec 17.7 GBytes 5.06 Gbits/sec 15267 sender
[ 8] (receiver statistics not available)
CPU Utilization: local/sender 50.5% (0.7%u/49.8%s), remote/receiver 27.3% (0.3%u/27.0%s)
CPU Utilization: local/receiver 50.5% (0.7%u/49.8%s), remote/sender 27.3% (0.3%u/27.0%s)

UDP 1: iperf3 -c 172.30.0.5 -u -b 1M -t 10

client omicron ~ 94dda3c:
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 123 KBytes 1.01 Mbits/sec 87
[ 5] 1.00-2.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 2.00-3.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 3.00-4.00 sec 123 KBytes 1.01 Mbits/sec 87
[ 5] 4.00-5.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 5.00-6.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 6.00-7.00 sec 123 KBytes 1.01 Mbits/sec 87
[ 5] 7.00-8.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 8.00-9.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 9.00-10.00 sec 123 KBytes 1.01 Mbits/sec 87
---[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.19 MBytes 1.00 Mbits/sec 0.000 ms 0/864 (0%) sender
[ 5] 0.00-10.00 sec 1.19 MBytes 1.00 Mbits/sec 0.063 ms 0/864 (0%) receiver

client mcast ~ 4e6889a:
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 123 KBytes 1.01 Mbits/sec 87
[ 5] 1.00-2.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 2.00-3.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 3.00-4.00 sec 123 KBytes 1.01 Mbits/sec 87
[ 5] 4.00-5.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 5.00-6.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 6.00-7.00 sec 123 KBytes 1.01 Mbits/sec 87
[ 5] 7.00-8.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 8.00-9.00 sec 122 KBytes 996 Kbits/sec 86
[ 5] 9.00-10.00 sec 123 KBytes 1.01 Mbits/sec 87
---[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.19 MBytes 1.00 Mbits/sec 0.000 ms 0/864 (0%) sender
[ 5] 0.00-10.00 sec 1.19 MBytes 1.00 Mbits/sec 0.031 ms 0/864 (0%) receiver

server omicron ~ 94dda3c:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 123 KBytes 1.01 Mbits/sec 0.072 ms 0/87 (0%)
[ 5] 1.00-2.00 sec 122 KBytes 996 Kbits/sec 0.040 ms 0/86 (0%)
[ 5] 2.00-3.00 sec 122 KBytes 996 Kbits/sec 0.149 ms 0/86 (0%)
[ 5] 3.00-4.00 sec 123 KBytes 1.01 Mbits/sec 0.109 ms 0/87 (0%)
[ 5] 4.00-5.00 sec 122 KBytes 996 Kbits/sec 0.100 ms 0/86 (0%)
[ 5] 5.00-6.00 sec 122 KBytes 996 Kbits/sec 0.074 ms 0/86 (0%)
[ 5] 6.00-7.00 sec 123 KBytes 1.01 Mbits/sec 0.101 ms 0/87 (0%)
[ 5] 7.00-8.00 sec 122 KBytes 996 Kbits/sec 0.107 ms 0/86 (0%)
[ 5] 8.00-9.00 sec 122 KBytes 996 Kbits/sec 0.153 ms 0/86 (0%)
[ 5] 9.00-10.00 sec 123 KBytes 1.01 Mbits/sec 0.063 ms 0/87 (0%)
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] (sender statistics not available)
[ 5] 0.00-10.00 sec 1.19 MBytes 1.00 Mbits/sec 0.063 ms 0/864 (0%) receiver

server mcast ~ 4e6889a
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 123 KBytes 1.01 Mbits/sec 0.108 ms 0/87 (0%)
[ 5] 1.00-2.00 sec 122 KBytes 996 Kbits/sec 0.027 ms 0/86 (0%)
[ 5] 2.00-3.00 sec 122 KBytes 996 Kbits/sec 0.107 ms 0/86 (0%)
[ 5] 3.00-4.00 sec 123 KBytes 1.01 Mbits/sec 0.061 ms 0/87 (0%)
[ 5] 4.00-5.00 sec 122 KBytes 996 Kbits/sec 0.069 ms 0/86 (0%)
[ 5] 5.00-6.00 sec 122 KBytes 996 Kbits/sec 0.031 ms 0/86 (0%)
[ 5] 6.00-7.00 sec 123 KBytes 1.01 Mbits/sec 0.095 ms 0/87 (0%)
[ 5] 7.00-8.00 sec 122 KBytes 996 Kbits/sec 0.051 ms 0/86 (0%)
[ 5] 8.00-9.00 sec 122 KBytes 996 Kbits/sec 0.029 ms 0/86 (0%)
[ 5] 9.00-10.00 sec 123 KBytes 1.01 Mbits/sec 0.031 ms 0/87 (0%)
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] (sender statistics not available)
[ 5] 0.00-10.00 sec 1.19 MBytes 1.00 Mbits/sec 0.031 ms 0/864 (0%) receiver

UDP 2: iperf3 -c 172.30.0.5 -u -b 1G -t 60

client omicron ~ 94dda3c:
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 65.8 MBytes 552 Mbits/sec 47623
[ 5] 1.00-2.00 sec 69.0 MBytes 579 Mbits/sec 50002
[ 5] 2.00-3.00 sec 67.7 MBytes 568 Mbits/sec 49007
---[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-60.00 sec 3.88 GBytes 555 Mbits/sec 0.000 ms 0/2875328 (0%) sender
[ 5] 0.00-60.00 sec 3.17 GBytes 454 Mbits/sec 0.019 ms 525756/2875320 (18%) receiver

client mcast ~ 4e6889a:
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 64.6 MBytes 542 Mbits/sec 46780
[ 5] 1.00-2.00 sec 67.1 MBytes 563 Mbits/sec 48610
[ 5] 2.00-3.00 sec 67.4 MBytes 565 Mbits/sec 48812
---[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-60.00 sec 3.85 GBytes 551 Mbits/sec 0.000 ms 0/2854715 (0%) sender
[ 5] 0.00-60.00 sec 3.40 GBytes 486 Mbits/sec 0.042 ms 336668/2854714 (12%) receiver

server omicron ~ 94dda3c:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 54.3 MBytes 455 Mbits/sec 0.019 ms 8020/47314 (17%)
[ 5] 1.00-2.00 sec 62.2 MBytes 522 Mbits/sec 0.055 ms 5219/50289 (10%)
[ 5] 2.00-3.00 sec 49.8 MBytes 417 Mbits/sec 0.034 ms 12943/48980 (26%)
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] (sender statistics not available)
[ 5] 0.00-60.00 sec 3.17 GBytes 454 Mbits/sec 0.019 ms 525756/2875320 (18%) receiver

server mcast ~ 4e6889a:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 59.2 MBytes 496 Mbits/sec 0.025 ms 3925/46766 (8.4%)
[ 5] 1.00-2.00 sec 60.3 MBytes 505 Mbits/sec 0.051 ms 4964/48601 (10%)
[ 5] 2.00-3.00 sec 63.2 MBytes 530 Mbits/sec 0.074 ms 2967/48721 (6.1%)
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] (sender statistics not available)
[ 5] 0.00-60.00 sec 3.40 GBytes 486 Mbits/sec 0.042 ms 336668/2854714 (12%) receiver

UDP 3: iperf3 -c 172.30.0.5 -u -b 200M -l 1024 -t 30

client omicron ~ 94dda3c:
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 23.8 MBytes 200 Mbits/sec 24402
[ 5] 1.00-2.00 sec 23.8 MBytes 200 Mbits/sec 24421
[ 5] 2.00-3.00 sec 23.8 MBytes 200 Mbits/sec 24411
---[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-30.00 sec 715 MBytes 200 Mbits/sec 0.000 ms 0/732399 (0%) sender
[ 5] 0.00-30.00 sec 715 MBytes 200 Mbits/sec 0.028 ms 2/732399 (0.00027%) receiver

client mcast ~ 4e6889a:
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 23.8 MBytes 200 Mbits/sec 24399
[ 5] 1.00-2.00 sec 23.8 MBytes 200 Mbits/sec 24408
[ 5] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 24434
---[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-30.00 sec 715 MBytes 200 Mbits/sec 0.000 ms 0/732421 (0%) sender
[ 5] 0.00-30.00 sec 715 MBytes 200 Mbits/sec 0.022 ms 0/732421 (0%) receiver

server omicron ~ 94dda3c:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 23.8 MBytes 200 Mbits/sec 0.025 ms 0/24402 (0%)
[ 5] 1.00-2.00 sec 23.8 MBytes 200 Mbits/sec 0.030 ms 0/24408 (0%)
[ 5] 2.00-3.00 sec 23.9 MBytes 200 Mbits/sec 0.022 ms 0/24424 (0%)
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] (sender statistics not available)
[ 5] 0.00-30.00 sec 715 MBytes 200 Mbits/sec 0.028 ms 2/732399 (0.00027%) receiver

server mcast ~ 4e6889a:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 23.8 MBytes 200 Mbits/sec 0.028 ms 0/24399 (0%)
[ 5] 1.00-2.00 sec 23.8 MBytes 200 Mbits/sec 0.028 ms 0/24408 (0%)
[ 5] 2.00-3.00 sec 23.8 MBytes 200 Mbits/sec 0.039 ms 0/24421 (0%)
---Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] (sender statistics not available)
[ 5] 0.00-30.00 sec 715 MBytes 200 Mbits/sec 0.022 ms 0/732421 (0%) receiver

@zeeshanlakhani zeeshanlakhani changed the base branch from multicast to main July 10, 2025 15:19
@zeeshanlakhani
Copy link
Contributor Author

Re-targeted to main after perf tests, as per @askfongjojo:

your mcast branch shows no perf profile change afaict

zeeshanlakhani added a commit to oxidecomputer/omicron that referenced this pull request Jul 11, 2025
zeeshanlakhani added a commit to oxidecomputer/omicron that referenced this pull request Jul 11, 2025
@zeeshanlakhani
Copy link
Contributor Author

Pinging @Nieuwejaar and @rcgoodfellow for any other looks here beyond @FelixMcFelix's extensive review!

@rcgoodfellow
Copy link
Contributor

Unless @Nieuwejaar has any more feedback I'd say let's get this landed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
multicast Multicast-related feature/impl
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants