"dumb_http_server" getting stuck in "net_tcp : Cannot allocate a new TCP connection"

[nikhilbhume-eaton]

Controller STM32
Zephyr main branch dated : 26-Dec-2023

I tried the intruder tool from burp suite sommunity edition [https://portswigger.net/burp/communitydownload] on "dump_http_server".
I wanted to test DOS and brute force attack on this server, just to check any crash happens on server or network layer. But once I start the attack immediately within some seconds, device stops responding, only ping works and gets below messages on serial console.
net_tcp: Cannot allocate a new TCP connection
net_tcp: net_context_get(): -2

I checked in wireshark, server only respond "RST, ACK" to any http request. Webpage is not loading at all.

Then I kept device as it is for overnight to check recovery [burp suite stopped and no request going to server].
But it is still not getting recovered. Means after many idle hours as well, device is not able to load webpage or accept any http request.

Issue is easily reproduceable.

[jukkar]

You ran out of resources so there is a allocate message printed.
There is a default ack timeout (1 second) that the tcp stack waits for peer to send an ack. There seems to be an issue somewhere if the system does not recover from this.
Are you using latest upstream zephyr or some point release?
Never mind my version question, you had that info already in the question.

[jukkar]

Tried the TCP connection exhaust scenario with echo-server and I could not replicate the issue. I am seeing that TCP runs out of resources but the system recovers every time. I can try with dumb-http-server next.

[jukkar]

Tried this with nucleo_f767zi and latest upstream and I am not able to see the issue. I used siege tool instead of burp.

west build -p -b nucleo_f767zi -d build/dumb-http-server samples/net/sockets/dumb_http_server -- \
  -DCONFIG_LOG_BUFFER_SIZE=65535  -DCONFIG_NET_SHELL=y \
  -DCONFIG_NET_LOG=y -DCONFIG_NET_DHCPV4=y \
  -DCONFIG_NET_CONFIG_MY_IPV4_GW="\"192.168.1.1\""
west flash -d  build/dumb-http-server/

And the stress tester

siege -c 4 -b http://192.168.1.14:8080 

The system can run out of TCP contexts but in my test it always recovers from it, and I can re-run the tester without an issue.

[nikhilbhume-eaton]

Hello @jukkar thanks for immediate response. I tried your steps, with siege tool, where issue is not reproduceable.
I also tried earlier with https://github.com/v0re/dirb as well. As per my observation burp is much sronger than siege and dirb. I could see very high frequency requests from burp. So it immediately reproduces the issue.

If you need some more information to configure burp then I can provide.

[jukkar]

If you need some more information to configure burp then I can provide.

@nikhilbhume-eaton that would be great.

Meanwhile I tried apache bench that I have also used in the past and which can stress things quite well, but did not see any issues with it either.

[jukkar]

I perhaps managed to trigger the case with apachebench
ab -i -t 20 -n 1000 -c 10 http://192.168.1.14:8080/

In net-shell one socket is stuck in the LAST_ACK state. If this happens many times, we can run out of sockets and then we see the issue you have in your device.
@nikhilbhume-eaton if you enable net-shell (set CONFIG_NET_SHELL=y), do you see something like this when issuing net conn command.

uart:~$ net conn
     Context    Iface  Flags            Local             Remote
[ 1] 0x200361b8 1      6ST          [::]:8080             [::]:0
[ 2] 0x20036294 1      4ST       0.0.0.0:8080          0.0.0.0:0
[ 3] 0x20036370 1      4DU      0.0.0.0:41506          0.0.0.0:0
[ 4] 0x200367bc 1      4ST   192.168.1.14:8080 192.168.1.67:52192

TCP        Context   Src port Dst port   Send-Seq   Send-Ack  MSS    State
0x200563d4 0x200361b8    8080        0          0          0  1440   LISTEN
0x20056154 0x20036294    8080        0          0          0  1460   LISTEN
0x200554d4 0x200367bc    8080    52192 1800522932 1380395991  1460   LAST_ACK

Edit: for the above, I used the dumb_http_server_mt sample with this configuration:

west build -p -b nucleo_f767zi -d build/dumb-http-server-mt samples/net/sockets/dumb_http_server_mt/ -- \     -DCONFIG_LOG_BUFFER_SIZE=65535  -DCONFIG_NET_SHELL=y \
-DCONFIG_NET_LOG=y -DCONFIG_NET_DHCPV4=y -DCONFIG_NET_UDP=y \
-DCONFIG_NET_CONFIG_MY_IPV4_GW="\"192.168.1.1\"" \
-DCONFIG_NET_SAMPLE_NUM_HANDLERS=20 \
-DCONFIG_NET_SAMPLE_SERVE_LARGE_FILE=y \
-DCONFIG_OBJ_CORE=y -DCONFIG_NET_SOCKETS_OBJ_CORE=y

[nikhilbhume-eaton]

Give me some time, I will immediately start on it and update.

This is the status I am getting when device stops responding. I used Burp tool.

[jukkar]

I prepared a patch for this #67067
please try and report if it solves your issue.

[nikhilbhume-eaton]

I tested your latest patch on full load for long run. Its much more stable now. But after some hours of run, sockets gets on hold in state of "FIN_WAIT_1" and "ESTABLISHED". Then server goes dead.

Scenario - 1

Scenario - 2

[carlescufi]

@nikhilbhume-eaton thanks very much for all the investigation so far. Would you mind opening a GitHub issue so we can continue there instead of using a discussion? Issues are formally triaged and reviewed, unlike discussions. Thanks!

[nikhilbhume-eaton]

Sure, I am moving this discussion to issue.

[jukkar]

Just fyi if you did not notice, there is a link on top that can be used for the conversion called "Create issue from discussion"

[nikhilbhume-eaton]

Yes, I created the issue using the option "Create issue from discussion".