-
Time-Triggered transmission with flexiable period and payload based on Linux Earliest TxTime First (ETF) Qdisc.
-
High-precision gPTP synchronization based on LinuxPTP-project.
-
Hardware timestamping for accurate delay and jitter measurement based on Linux SO_TIMESTAMPING.
This project is based on Intel I210 NIC, RaspberryPi CM4, RaspberryPi IO board, and Ethernet RJ45 cable. The Linux distribution version is Ubuntu 22.04.1 LTS and kernel version is 5.15.0-1021-raspi. It is recommended to install Ubuntu using the Raspberry Pi Imager tool.
There are several reasons that we choose i210 NIC:
- What is LaunchTime function?
- Why PCIe NIC is Faster than on board DMA NIC?
- What is Receive Side Scaling (RSS)?
- Why does synchronization need NIC PHC clock?
One additional advantage of the I210 NIC is that it is relatively inexpensive and easy to obtain from online retailers with around 40$ :)
rtgen-kernel achieves deterministic communication based on Linux* Traffic Control (TC) System implemented by TC Queuing Disciplines. There are several qdisc defined in linux kernel as SFQ, CBS, TAPRIO, and ETF, please see document. The ETF (Earliest Transmission Time First) queuing discipline in particular allows applications to control the precise moment when a packet is dequeued from the traffic control layer and sent to the network device, enabling packets to be scheduled with nanosecond-level accuracy.
To obtain the igb driver for the CM4, you can download the folder from the Raspberry Pi kernel GitHub repository here. Alternatively, you can use the same igb driver provided in this repository.
Note that there are several versions of the igb driver available from Intel and other communities, such as Avnu-igb, but many of them do not support ARM architecture. Some common compilation errors that you may encounter when building the driver for ARM include
implicit declaration of function read_barrier_dependsandincompatible pointer type of function igb_get_coalesce.
Once you have downloaded the driver code, navigate to the igb folder and run the following command to build the igo.ko module:
make -C /lib/modules/`uname -r`/build M=$PWD
After building the igo.ko module, you can copy it to the kernel modules folder using the following command:
sudo mkdir /lib/modules/`uname -r`/kernel/drivers/net/igb/
sudo cp ./igb.ko /lib/modules/`uname -r`/kernel/drivers/net/igb/igb.ko
Finally, you can use the following command to load it into the kernel and make it available for use.
sudo depmod -a
sudo rmmod igb
sudo modprobe i2c_algo_bit
sudo modprobe igb
If you don't want to install the module permanently, you can install the .ko file locally and load by following commands:
sudo depmod -a
sudo rmmod igb
sudo insmod igb.ko
sudo modprobe i2c_algo_bit
sudo modprobe igb
After installing the igb driver, you can use the ifconfig command to check that the corresponding interface for your i210 NIC has been created.
To help you diagnose and configure your network interfaces, I have provided some additional commands below:
## Extract detailed information on the hardware configuration of network cards
lspci
lshw -class network
## Add and delete ip address of network interface
ip addr show
ip addr add 0.0.0.0/24 dev [interface]
ip addr del 0.0.0.0/24 dev [interface]
On my 64-bit AMD machine, I have found that using the mainline kernel version 5.14.0 and 5.15.0 with the Intel official driver resulted in a GPF error in the kernel logs and caused the system to freeze. If you encounter similar issues, you may need to use an older or newer kernel version, or try a different igb driver.
The Intel Ethernet Controller I210 support a feature called LaunchTime, which allows frames to be transmitted at specific times. This feature is enabled through the SO_TXTIME sockopt and the ETF queuing discipline (qdisc). The detailed tutorial of the ETF qdisc follows the guidelines are also provided by the LinuxTSN project.
The ETF qdisc operates on a per-queue basis, using the MQPRIO configuration. Therefore, you will need to configure MQPRIO in order to use ETF:
sudo tc qdisc add dev [interface] parent root handle 6666 mqprio \
num_tc 3 \
map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
queues 1@0 1@1 2@2 \
hw 0
Once you have configured the MQPRIO settings, you can proceed with configuring the ETF qdisc:
sudo tc qdisc add dev [interface] parent 6666:1 etf \
clockid CLOCK_TAI \
delta 350000 \
offload
An important parameter that significantly affects the performance of deterministic communication needs to be optimized. A simple way to do this is to use the cyclictest tool to estimate, which has similar behavior to the ETF qdisc:
sudo cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0
Use an upper bound for the delay measurement as the value for the delta parameter:
policy: fifo: loadavg: 0.43 0.21 0.14 1/495 27522
T: 0 (27519) P:80 I:200 C: 8302 Min: 15 Act: 28 Avg: 28 Max: 83
T: 1 (27520) P:80 I:200 C: 8215 Min: 15 Act: 26 Avg: 27 Max: 68
T: 2 (27521) P:80 I:200 C: 8123 Min: 16 Act: 29 Avg: 28 Max: 63
T: 3 (27522) P:80 I:200 C: 8025 Min: 15 Act: 27 Avg: 29 Max: 338
Since the maximum delay measured here is 338 us, the delta in the configuration should be set as 350 us.
To make it easier to configure the ETF qdisc, a config/config_qdisc.sh script is provided.
sudo bash ./config/config_qdisc.sh -d [interface] -o 350000
I will leave out this part for future integration with the TSN testbed. In the meantime, a config/config_vlan.sh script is provided to make it easier to configure VLAN and priority mapping:
sudo bash config_vlan.sh -d [interface] -v 10 -i "192.168.10.33/24" -p 3
The Precision Time Protocol (PTP), specified by the IEEE 1588 profile, is used for synchronization in this project. PTP allows network devices to maintain synchronized clocks with sub-microsecond precision, which is necessary for applications such as real-time control and high-speed communication. gPTP, which uses a master-slave architecture, is used in this project. In this architecture, a designated master device distributes a precise reference time to slave devices on the network, and the slave devices adjust their local clocks to match the reference time provided by the master.
LinuxPTP is used as the PTP stack in this project. To install it, run the following command:
sudo apt install linuxptp
Synchronizing the PHC to CLOCK_REALTIME (CLOCK_TAI) is necessary for the ETF qdisc to directly schedule offload time on the NIC. To ensure that your network interface supports PTP, run the following command. If you are not using an Intel i210, the TX/RT_HARDWARE and Hardware Transmit/Receive Modes values must be present in the output of the command."
Time stamping parameters for i210_1:
Capabilities:
hardware-transmit
software-transmit
hardware-receive
software-receive
software-system-clock
hardware-raw-clock
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
off
on
Hardware Receive Filter Modes:
none
all
To set up synchronization, the first step is to disable the Ubuntu synchronization service, which may prevent stable synchronization:
sudo systemctl stop systemd-timesyncd
sudo systemctl stop ntp
Then run the following command to set up synchronization between PHC and CLOCK_REALTIME
sudo phc2sys -s [interface] -c CLOCK_REALTIME -m -O 0
From the output of the command, the synchronization between PHC and CLOCK_REALTIME is successfully set up once the offset is stable within 1 us.
phc2sys[171810.140]: CLOCK_REALTIME phc offset -14 s0 freq +9342 delay 1667
phc2sys[171811.141]: CLOCK_REALTIME phc offset 0 s2 freq +9356 delay 1666
phc2sys[171812.141]: CLOCK_REALTIME phc offset -12 s2 freq +9344 delay 1704
phc2sys[171813.142]: CLOCK_REALTIME phc offset -7 s2 freq +9345 delay 1667
phc2sys[171814.142]: CLOCK_REALTIME phc offset 0 s2 freq +9350 delay 1704
phc2sys[171815.143]: CLOCK_REALTIME phc offset 13 s2 freq +9363 delay 1667
phc2sys[171816.143]: CLOCK_REALTIME phc offset -3 s2 freq +9351 delay 1704
phc2sys[171817.144]: CLOCK_REALTIME phc offset 8 s2 freq +9361 delay 1722
I will leave out this part for future integration with the TSN testbed. In the meantime, a simple gPTP profile is provided in config/gptp.cfg:
# 802.1AS example configuration containing those attributes which
# differ from the defaults. See the file, default.cfg, for the
# complete list of available options.
#
[global]
gmCapable 1
priority1 248
priority2 248
logAnnounceInterval 0
logSyncInterval -3
syncReceiptTimeout 3
neighborPropDelayThresh 800
min_neighbor_prop_delay -20000000
assume_two_step 1
path_trace_enabled 1
follow_up_info 1
transportSpecific 0x1
ptp_dst_mac 01:80:C2:00:00:0E
network_transport L2
delay_mechanism P2P
Run the network wide synchronization with command:
sudo ptp4l -i [interface] -f gptp.cfg -m
Check the synchronization status
sudo pmc -u -b 0 -t 1 'GET CURRENT_DATA_SET'
In this project, I developed a C application that uses the Linux SO_TXTIME and SO_TIMESTAMPING APIs to achieve deterministic communication.
Client Usage:
Stream a UDP traffic to 192.168.0.22 on port 54321 via interface i210_1
sudo client -d 192.168.0.22 -p 54321 -i i210_1
Server Usage:
Listen a UDP traffic via port 54321 via interface i210_1
sudo server -p 54321 -i i210_1
cd src
make clean
make client
make server
| Argument | Description | Default value |
|---|---|---|
| -d | Destination ip address [str] | 127.0.0.1 |
| -p | Remote port number [int] | 1998 |
| -i | Network interface name [str] | eth0 |
| -o | Delta from wake up to txtime in nanoseconds [int] | 1000000 |
| -t | Traffic period in nanoseconds [int] | 1000000000 |
| -l | Traffic paylaod size in bytes [int] | 256 (+46) |
| -k | Traffic SO_PRIORITY assign [int] | 3 |
| -a | Disable debug mode | |
| -b | Disable loopback | |
| -s | Disable ETF scheduler | |
| -w | Disable Hardware Timestamping | |
| -h | Help page |
| Parameter | Value |
|---|---|
| disc-delta | 100_000 |
| time-delta | 1_000_000 |
| payload | 100 |
There are two known issues:
- Significant packet loss when inter-packet distance < 1 $\mu$s, this is limited by kernel's network stack (to solve this try rtgen-dpdk with equal precision but higher throughput and low latency)
- < 1% packet loss even for low frequency traffic
Sending jitter on different End-Station
| Device | Packet Loss | Jitter (L7) | Jitter (L2) | |
|---|---|---|---|---|
| 1 s | PC | 0 | 1.54e-5 | 0.00e-9 |
| CM4 | 0 | 1.58e-6 | 1.10e-8 | |
| 100 ms | PC | 0.00% | 1.71e-5 | 7.18e-9 |
| CM4 | 1.02% | 3.50e-5 | 1.27e-8 | |
| 10 ms | PC | 0.03% | 2.19e-5 | 3.28e-7 |
| CM4 | 0.00% | 4.14e-5 | 1.10e-8 | |
| 1 ms | PC | 0.00% | 2.13e-5 | 7.99e-9 |
| CM4 | 0.56% | 1.22e-5 | 1.17e-8 | |
| 100 us | PC | 0.76% | 2.22e-5 | 7.03e-9 |
| CM4 | 0.79% | 7.14e-5 | 3.00e-9 |
Receiving jitter
| Device | Packet Loss | Jitter (L7) | Jitter (L2) | |
|---|---|---|---|---|
| 1 s | CM4->PC | 0.00% | 6.21e-5 | 6.41e-9 |
| 100 ms | CM4->PC | 0.00% | 5.46e-5 | 1.00e-8 |
| 10 ms | CM4->PC | 0.00% | 2.19e-5 | 3.28e-7 |
| 1 ms | CM4->PC | 0.04% | 1.70e-5 | 1.47e-8 |
| 200 us | CM4->PC | 0.76% | 1.73e-4 | 1.85e-8 |
Synchronization error
MAC layer RTT delay on different NIC
Please cite the following paper if you use this tool in your research
@inproceedings{10.1145/3649329.3658260,
author = {Xue, Chuanyu and Zhang, Tianyu and Han, Song},
title = {Towards Cost-Effective Real-Time High-Throughput End Station Design for Time-Sensitive Networking (TSN)},
year = {2024},
url = {https://doi.org/10.1145/3649329.3658260},
doi = {10.1145/3649329.3658260},
booktitle = {Proceedings of the 61st ACM/IEEE Design Automation Conference},
articleno = {110},
numpages = {6},
location = {San Francisco, CA, USA},
series = {DAC '24}
}