Connection (socket) not closed when connection is lost

[steteb]

I am using Mongoose v7.6 on an embedded device running FreeRTOS and LwIP. The device is a web server and is using a wifi connection and it is connected to an access point. If I am connected to the device with a client and close the browser on the client, I see that some connections are closed, but if I switch off for example the access point, and then I lost the wifi connection, I cannot see that any connection has been closed on Mongoose. I try to detect the lost of wifi connection and call mg_mgr_free(), but it seems that after some sockets that are closed, lwip is blocked on the lwip_close().

any idea ? Socket connections should not be closed when the connection has been lost ?

[scaprile]

When you close your client, it clearly sends a closure indication (FIN).

If you unplug your client's network cable, the server won't notice until a TCP keep-alive timer expires.

If you power off your AP on a WiFi server... well, TCP won't notice until keep-alive timer expires; but you should have some way of detecting link status if that is important to you. Check your vendor implementation, lwIP's netif reference is here.

You should not call mg_mgr_free() right away but close your connections first.

[steteb]

"You should not call mg_mgr_free() right away but close your connections first."

what do you mean exactly ? The only connections I have are the ones created and managed by Mongoose. Do you mean that they should be closed due to a trigger from lwip for a TCP keep-alive expiration ? This seems to be the problem, there is no a keep alive expiration that close the connections due to a link disconnection and I cannot understand why.

As I said, in someway I can detect the link status, I will try to call netif_set_link_down() or some API of lwip instead of mg_mgr_free() hoping it triggers the closing of connections on Mongoose.

Thank you

[scaprile]

It's been a long time since I played with lwIP at low level.
lwIP will handle TCP, and Mongoose will know when a connection is closed. Sniff your network with Wireshark to see if there are TCP keep-alives.
On a WiFi loss issue, as the scenario you are pointing out, you can try to detect that by asking lwIP netif_is_up(netif) or registering a callback for state changes netif_set_link_callback (struct netif *netif, netif_status_callback_fn link_callback). In this case, if you free the event manager and stop calling it, your event handlers won't be called and your applications won't know the connection has closed. You should keep calling the manager until all connections have closed.

TCP connection termination is a 4-way handshake, lwIP will stay waiting for the other end to close, unless it knows the netif is down or enough time has elapsed.

[steteb]

In my application the Mongoose as web server manages almost 8 connections. On detection of cable removal, I call netif_set_link_down() and what I can see is that only one connection is closed (the web socket one). This is because mg_sock_send() returns error in write_conn(). Other connections and related sockets remain open and if I plug again the cable and reconnect to the web server further connections are opened and after two or three times I do this operation the connection is not possible anymore because there are no resources for other sockets. So the only way I found to free connection resources is to call mg_mgr_free().

[cpq]

It looks like the old connections got stalled - maybe TCP keep-alive does not work?
@steteb would you be able to make and share a wireshark traffic capture of all traffic before, during, and after cable unplug/plug and resource exhaustion please?

[steteb]

Hello,

sorry, for the moment I cannot share a wireshark capture because it could contain data that I cannot share. I have to see if I can create a wireshark log without sensitive data.

Thank you for the moment

[steteb]

Hello,

I am wondering how the Mongoose knows that a socket has been closed by lwIP. When I disconnect, after netif_set_link_down, I can see that the web socket connection is closed by Mongoose because in write_conn() the mg_sock_send() return error. What about receiving only connection ? They remains open in Mongoose because is_readable is never true and read_conn is never called. I cannot understand this part. Why web socket connection is cleaned and all other connections instead remains active in Mongoose ?

And in general, how can I close the listening socket ? It seems the only way is mg_mgr_free().

Thank you

[scaprile]

One thing that comes to mind is that sockets can't be closed and reopened immediately, there is a possibility of data arriving after closure so the RFC mandates a minimum time before reuse. It will be closed eventually, but perhaps not as fast enough as you wish.
AFAIK Mongoose binds to a socket when you call mg_listen() and there is no function to "unlisten".
Please post a capture as requested so we can trace this.

[steteb]

Hello,

Ok, I will try to have the capture as soon as possible.

But I still don't understand why a connection in Mongoose is freed (the web socket one) and instead all the others remain there.

[cpq]

@steteb Because websocket connection is frequently used, and next read/write triggers an error.
Whilst those idle stale connections are not used. If link is down, they "do not notice". The TCP keep-alive mechanism is supposed to detect them - but the usual keep-alive interval is 45 seconds.