ICMP Packet Too Big

[neilalexander]

Describe the bug

The TUN adapter sets up with an MTU 1500 and this seems to be the network-wide MTU.

However, if you want to route beyond a given lokinet router (e.g. masquerading NAT outbound to allow some other network to reach lokinet, or destination NAT inbound) across a link with an MTU of lower than 1500, it appears lokinet ignores the ICMP Packet Too Big messages generated as a result of upstream links and continues to send packets that are too large.

To Reproduce

Set up lokinet on machine A with masquerading NAT and forwarding enabled. Set up some kind of tunnel between machine A and B with an MTU of less than 1500. Try to reach a lokinet service from machine B whilst watching tcpdump on machine A. Lots of ICMP Packet Too Big messages are generated because the return traffic from lokinet on machine A for machine B is too large.

e.g. continuing to send 1448 bytes when the next hop interface is MTU 1410:

15:33:37.021991 IP 172.19.0.1 > 172.19.0.2: ICMP 172.19.0.1 unreachable - need to frag (mtu 1410), length 556
15:33:37.034400 IP 172.19.0.2.80 > 172.19.0.1.58478: Flags [.], seq 2217:3665, ack 694, win 505, options [nop,nop,TS val 2824688482 ecr 1978533898], length 1448: HTTP
15:33:37.034567 IP 172.19.0.1 > 172.19.0.2: ICMP 172.19.0.1 unreachable - need to frag (mtu 1410), length 556
15:33:37.054451 IP 172.19.0.2.80 > 172.19.0.1.58478: Flags [.], seq 3665:5113, ack 694, win 505, options [nop,nop,TS val 2824688482 ecr 1978533898], length 1448: HTTP
15:33:37.054647 IP 172.19.0.1 > 172.19.0.2: ICMP 172.19.0.1 unreachable - need to frag (mtu 1410), length 556
15:33:37.080589 IP 172.19.0.2.80 > 172.19.0.1.58478: Flags [.], seq 5113:6561, ack 694, win 505, options [nop,nop,TS val 2824688493 ecr 1978533898], length 1448: HTTP
15:33:37.081042 IP 172.19.0.1 > 172.19.0.2: ICMP 172.19.0.1 unreachable - need to frag (mtu 1410), length 556
15:33:37.101681 IP 172.19.0.2.80 > 172.19.0.1.58478: Flags [.], seq 6561:8009, ack 694, win 505, options [nop,nop,TS val 2824688516 ecr 1978533898], length 1448: HTTP
15:33:37.102010 IP 172.19.0.1 > 172.19.0.2: ICMP 172.19.0.1 unreachable - need to frag (mtu 1410), length 556
15:33:37.123696 IP 172.19.0.2.80 > 172.19.0.1.58478: Flags [.], seq 8009:9457, ack 694, win 505, options [nop,nop,TS val 2824688539 ecr 1978533898], length 1448: HTTP
15:33:37.123939 IP 172.19.0.1 > 172.19.0.2: ICMP 172.19.0.1 unreachable - need to frag (mtu 1410), length 556
15:33:37.148182 IP 172.19.0.2.80 > 172.19.0.1.58478: Flags [.], seq 9457:10905, ack 694, win 505, options [nop,nop,TS val 2824688562 ecr 1978533898], length 1448: HTTP
15:33:37.148425 IP 172.19.0.1 > 172.19.0.2: ICMP 172.19.0.1 unreachable - need to frag (mtu 1410), length 556
15:33:37.171572 IP 172.19.0.2.80 > 172.19.0.1.58478: Flags [.], seq 10905:12353, ack 694, win 505, options [nop,nop,TS val 2824688585 ecr 1978533898], length 1448: HTTP
15:33:37.171920 IP 172.19.0.1 > 172.19.0.2: ICMP 172.19.0.1 unreachable - need to frag (mtu 1410), length 556
15:33:37.194388 IP 172.19.0.2.80 > 172.19.0.1.58478: Flags [.], seq 12353:13801, ack 694, win 505, options [nop,nop,TS val 2824688607 ecr 1978533898], length 1448: HTTP
15:33:37.194605 IP 172.19.0.1 > 172.19.0.2: ICMP 172.19.0.1 unreachable - need to frag (mtu 1410), length 556

Incidentally, we had a similar problem in Yggdrasil and we largely avoided this by allowing each side to configure their own MTU and then exchanging MTUs with anyone we opened a session to. Both sides agree to use the lower MTU and ICMP Packet Too Big messages are synthesised by the TUN reader when packets that exceed that size are read, which applications use to pick smaller packet sizes.

Device and Operating system (please complete the following information):

• OS: Linux
• Lokinet Version number or Git commit hash: 0.9.5

[neilalexander]

As an additional note, clamping the MSS to (upstream MTU - 40) alleviates the issue for TCP, although it won't do much for any other protocol:

-A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1370

[majestrate]

would ip fragmentation make sense here or am i misunderstanding the problem?

[neilalexander]

I don't think you want to get into the realms of fragmenting IP manually, that's fraught with issues :D

Let's say you have something like this:

lokitun0 (1500) <--> wg0 (1410) <--- wireguard tunnel ---> wg0 (1410) <--> application
^_________ machine A ________ ^                            ^_______ machine B ______ ^

So:

1. Machine B sends packet for Lokinet, which gets routed over the lower-MTU Wireguard link using existing routing
2. Machine A receives the packet from Wireguard and forwards it into lokitun0 (the masquerading NAT is happening at that point)
3. The remote lokinet host responds, not knowing about our smaller MTU on the wireguard link to machine B, so it responds with packets up to 1500 bytes
4. Machine A tries to forward the response from lokinet back to machine B, but it's too big for the MTU 1410 wireguard link
5. Machine A generates an ICMP Packet Too Big and sends it into lokitun0

I suspect what happens is that lokinet either drops the ICMP Packet Too Big, or it's forwarding it to the remote side but the remote side can't match it to a traffic flow because the lokinet IP/port tuples are different on each side (thanks to the dynamic mappings or differing IP ranges being configured on each side). In any case, it doesn't get handled by the kernel.

What is probably better to do is:

1. Allow each side to configure their own MTU, either in the config file or on lokitun0 directly
2. When you establish a circuit to another host, send your MTU to them, and wait for their MTU back
3. Both sides should pick the lower of the local and remote MTUs
4. If the TUN reads a packet for that circuit that exceeds the agreed MTU, generate our own ICMP Packet Too Big packet and write it back to the TUN instead of sending the traffic over lokinet, so that the kernel learns and adjusts the segment size for the next packet

Alternatively, rewrite the fields in the ICMP Packet Too Big but fix the fields for the addresses so they are right on both sides.

[majestrate]

what if we rewrote the ip header inside the ICMP packet such that it would match?

i think we'll rewrite the ip header inside the icmp packet

[neilalexander]

That might work, but I think it's common practice to include some of the first couple hundred bytes of the original failed packet in the ICMP Packet Too Big payload too, so you'd also need to rewrite those IP (and possibly TCP?) headers in the payload also or else the operating system will not be able to match the ICMP error to a traffic flow.

[majestrate]

ah.... that is tricky, rewriting the tcp checksums.....

[majestrate]

@neilalexander try out #1701