Incorrect Assumption There Is A TCP Connection When There May Not Be

[kentaylor]

Basic Infos

Where there is an unreliable network connection the ESP8266 using the Arduino Core can get stuck in an endless loop waiting for a message that will never arrive.

Hardware

Hardware: ESP-12E
Core Version: All

Description

The effect of the problem is described in another thread that links to several instance reports. It happens often for me when the ESP is connected to a WiFi AP that uses a mobile phone network for its internet connectivity.

The failure is a consequence of an incorrect assumption that the ESP8266 knows whether it has a TCP connection. The assumption is correct when the TCP connection is created and destroyed in accordance with this state diagram
but can be invalid if there is packet loss once the TCP connection gets to the ESTABLISHED state shown in green in the diagram.

When lost packets are recovered in the TCP layer the state diagram is followed and when packets sent from the ESP are not acknowledged the ESP should know the TCP connection is broken but when the remote socket is closed incorrectly, perhaps through a power outage or packets have been sent from a remote socket but not received by the ESP it can not know. If packets are sent from the remote socket but not received, the remote machine will retry and eventually close its socket. If the ESP is waiting for incoming packets it will still think the TCP connection exists when it doesn't.

The Arduino core relies on the ESP knowing whether it has a TCP connection. An example is WiFiClient::connected() in WiFiClient.cpp where it concludes that there is a TCP connection when the TCP state is ESTABLISHED but ESTABLISHED means there has been a TCP connection, not necesarily that it still exists. In the WiFiClient.ino example client.available() is used to decide if a TCP connection exists and this function can return true when the TCP connection has been lost. Similarly in BasicHttpClient.ino it calls http.GET() which calls sendRequest which calls handleHeaderResponse() in which a loop is controlled with the erroneous connected() function that may return true when there is no TCP connection and hence loop forever.

[mtnbrit]

Does it still respond to ping when its stuck in this loop forever state for you? if so, I have seen this issue where only a power cycle will get it back. Kicking it from the AP and having it reconnect does not help wake it back up. it does respond to ping.

[kentaylor]

@mtnbrit that is interesting. Yes it will still respond to ping. The packets can be lost anywhere between the remote machine and the ESP device, in my case most often in the Telco mobile network. If the packets are lost beyond the router then it will work fine when pinged from the local network and the packet loss can be temporary so it could be pinged from elsewhere as well once the network is working again. However, a failed TCP connection will not be healed so it will remain in the endless loop referred to regardless of the network healing.

What is interesting is where @mtnbrit says:-

Kicking it from the AP and having it reconnect does not help

because without a WiFi network it should know the TCP connection is broken. How do you kick it from the AP, is it by a command on the router? Also how long is it disconnected from the router because it should be disconnected for a while before it gives up. The TCP layer can deal with intermittent packet loss and needs a few seconds to try and recover from a short term failure before it classifies packet loss as a failed TCP connection. You could try something like blocking the ESP MAC address from connecting to your router for 10 seconds or so and see if it is still stuck in an endless loop or the TCP connection has failed gracefully.

[objarni]

Any progress on this one? WiFi down should definitely trigger a connection closed signal on a device where WiFi is the only connectivity.

[igrr]

Disconnection from an AP does indeed close all active WiFiClient connections.

The point from the original issue is a valid one. My assumption was that lwip will kill inactive PCBs after a while. I will check whether this indeed happens.

[kentaylor]

I'm thrilled to get a response from @igrr. In a real system that is expected to run for long periods this is a major flaw so I was a little disappointed there was so little interest when I first raised the issue. It seems to me to be so major that commercial products would be insufficiently reliable to be viable with this library. I have had devices run for weeks in the stuck condition so I'm confident lwip does not pick up the absence of a remote socket. In the TCP layer there is a concept of keep alive packets but these are often not implemented at all and the relevant RFS says they shouldn't be sent at less than 2 hour intervals see http://ltxfaq.custhelp.com/app/answers/detail/a_id/1512/~/tcp-keepalives-explained .

You will see all over the internet people stressing the importance of large capacitors with ESP8266's to keep them reliable. I think that claim is not in general true and mostly large capacitors are used to overcome this software error. Radio transmit increases current draw a lot which can cause a voltage drop sufficient for the processor to continue to work but the radio to fail temporarily. Then you can get into the state described above where the device is stuck waiting for a packet that will never come. I can reproduce this reliably by powering an ESP8266 that sends data at 1 minute intervals off alkaline batteries until they become flat. Alkaline batteries have a high internal resistance making the voltage sensitive to current draw and while the exact battery voltage where the ESP will get stuck is unpredictable it almost always does, long before the batteries are flat and usually from about 2/3 flat with 3 cells in series. Rebooting makes the ESP functional again on the same batteries. I'm not a hundred percent sure of this because I am assuming the processor can recover from a brownout satisfactorily. What I have checked though is that the processor is still running when in this state as the current draw fluctuates and it appears to be fluctuating enough to be still using the radio and it will still flash the led on a NodeMCU occasionally.

Most applications will work OK with an occasional reboot but stuckness is fatal and that is what, as far as I can tell, most large capacitors are added to avoid.

Looking some more, I came across the statement "LWIP_TCP_KEEPALIVE controls compiling in support for TCP keepalives and by default each connection has keepalives off." So it seems keep alives can be enabled in lwip but I hope the interval is much less than 2 hours.

[objarni]

Thanks for this information, @kentaylor. I have been experiencing TCP connections from node MCU units getting into a bad state (unit thinks it is still connected via WiFiClient.connected() API, but it cannot get any more data from server) where a power cycle of the unit, or dropping the connection from the server, has been the only way to make communication work again. On server side the connections are eventually detected to be broken (exact exception in python is TimeoutException).

My workaround so far is dropping TCP connections from node MCU units after awhile, for example never keeping it for more than five minutes, by closing it on the server. It does feel really bad though when TCP sockets are supposed to be super reliable.

The read on keep alive packets was good, it seems like a good balance between reliability and low transmission overhead.

[igrr]

We can check for (in)activity on any given PCB and close it after certain timeout. This however would break applications which use persistent TCP connections without application-level keepalives (like MQTT does). For instance, I have an application with a TCP connection which is expected to stay up for hours, without any data being passed in either direction.

Would it be best to add some reasonable timeouts at application layer? For instance HTTPClient library can have some timeout associated with the client.available() loop, so that if the connection is for some reason lost, we will eventually bail out from reading. Looking at the source code, this is what curl library does. In this case WiFiClient.ino sample also needs to be updated to include some timeout logic.

[objarni]

Updating the WiFiclient example to include connection reliability would be good.

What do you mean by "at the application layer"?

[vlast3k]

@kentaylor "Most applications will work OK with an occasional reboot but stuckness is fatal and that is what, as far as I can tell, most large capacitors are added to avoid." - as far as i remember from some posts, is that one possible issue is that the Flash operates on 3V on most of the devices. So using 2 alkaline batteries is already on it's low level.

[objarni]

@igrr My application does not use battery for power (status lamps in software development offices). So I prefer reliability in favour of low power consumption or low bytes-per-minute.

1. Is there any way to enable keep-alive packets as described by articles linked by @kentaylor?
2. If not, I guess one workaround would be a behaviour where a byte is sent every minute from the NodeMCU, and then wait for an answer byte from the server within a specific time period, say 15 seconds. If the answer byte does not arrive within 15 seconds, I could assume that the connection is broken, in which case a ESP.restart(); could be performed. Does this make sense? Is there any flaw in this reasoning?

[pwolfman]

Does anyone know how I can enable keepalive for TCP communications with the ESP8266/Arduino ? I see some mention by @igrr and @kentaylor in this chain. My issue is I have multiple clients connecting and sometimes a client can drop off and the TCP socket remains open. If I could implement a keepalive function, I could free the socket. I have found "espconn_set_keepalive" function in the espconn.h, but I have no idea how to implement it in the Arduino environment.

[drmpf]

Basically your clients need to keep sending something say every 5 sec if nothing else is happening and your server needs to reset a timer on data received. When the timer times out, the server should close the socket on its end.

See the code in this project
http://www.forward.com.au/pfod/CheapWifiShield/ESP2866_01_WiFi_Shield/index.html
for an example.

Of course your clients must send the keepalive messages :-)

[pwolfman]

Unfortunately I have no control over the attached client and I have no way of knowing if it drops from the network. The ESP8266 as the server to the client needs to send the keepalive message every minute or so to see if the client has dropped from the network so I can stop the client connection. The client may be attached for 1 minute or 1 or mores hours without sending any data, so setting a timer for lack of communications would be unpredictable. Without releasing the sockets when a client drops off the network unexpectedly I run out of sockets and the clients cannot attach to me.

[vitotai]

I seems to be able to reproduce this issue easily.
In my application, there are two ESP8266s, server and client, obviously.
The server runs ESPAsyncWebServer while the client connects by HTTPClient or WiFiClient.
The client reports status in a period, and between the reports, it deep sleeps.
If the deep sleep call follows stop call immediately, the server will become inaccessible quickly - but ping-able.
Adding a delay of 50 to 100ms seems to alleviate this situation. However, kentaylor makes a very good point. This is more serious issue than crash. If the device crashes and restarts, users may not be aware of it, but users will definite know when the device stops response.