Ethernet Cable Disconnect - No more Connection or Reconnection possible (STM32F407)

[JonasReich92]

Post edited by ST moderator to be inline with the community rules especially with the code sharing. In next time please use </> button to paste your code. Please read this post: How to insert source code

Hello there,

i have some connection/re-connection problems with my opcua server, when i unplug the LAN cable.
Because of prevent memory problems i disable the most Options for the opcua Server. Perhaps in the Future i will try more options out.
But at first the Connection should be stable and when I unplug the LAN cable a re-connection should be possible again.
Perhaps you can help me here.

Hardware:
- STM32F407
- DP83826 (TI)
- etc.

Firmware:
- open62541 v1.3.6 (amalgamation)
- LwIP v2.2.0
- FreeRTOS v10.5.1
- etc.

Perhaps interesting Settings:
- Memory Management: Heap5

- lwipopts.h:

#define MEMP_NUM_NETBUF 4
#define MEMP_NUM_NETCONN 8

#define LWIP_COMPAT_MUTEX 1
#define LWIP_COMPAT_MUTEX_ALLOWED 1

#define LWIP_TCP_KEEPALIVE 1

/* STM32CubeMX Specific Parameters (not defined in opt.h) ---------------------*/
/* Parameters set in STM32CubeMX LwIP Configuration GUI -*/
/*----- WITH_RTOS enabled (Since FREERTOS is set) -----*/
#define WITH_RTOS 1

/*----- CHECKSUM_BY_HARDWARE enabled -----*/
#define CHECKSUM_BY_HARDWARE 1
/*-----------------------------------------------------------------------------*/

#define LWIP_TIMEVAL_PRIVATE 0 // Wenn 0, verwendet LWIP die standardmäßige time.h-Zeitstruktur
/* LwIP Stack Parameters (modified compared to initialization value in opt.h) -*/
/* Parameters set in STM32CubeMX LwIP Configuration GUI -*/
/*----- Value in opt.h for MEM_ALIGNMENT: 1 -----*/
#define MEM_ALIGNMENT 4
/*----- Value in opt.h for LWIP_ETHERNET: LWIP_ARP || PPPOE_SUPPORT -*/
#define LWIP_ETHERNET 1
/*----- Value in opt.h for LWIP_DNS_SECURE: (LWIP_DNS_SECURE_RAND_XID | LWIP_DNS_SECURE_NO_MULTIPLE_OUTSTANDING | LWIP_DNS_SECURE_RAND_SRC_PORT) -*/
#define LWIP_DNS_SECURE 7
#if 1 //jonas.reich define needed for open62541
#define LWIP_DNS 1
#endif
/*----- Value in opt.h for TCP_SND_QUEUELEN: (4*TCP_SND_BUF + (TCP_MSS - 1))/TCP_MSS -----*/
#define TCP_SND_QUEUELEN 9
/*----- Value in opt.h for TCP_SNDLOWAT: LWIP_MIN(LWIP_MAX(((TCP_SND_BUF)/2), (2 * TCP_MSS) + 1), (TCP_SND_BUF) - 1) -*/
#define TCP_SNDLOWAT 1071
/*----- Value in opt.h for TCP_SNDQUEUELOWAT: LWIP_MAX(TCP_SND_QUEUELEN)/2, 5) -*/
#define TCP_SNDQUEUELOWAT 5
/*----- Value in opt.h for TCP_WND_UPDATE_THRESHOLD: LWIP_MIN(TCP_WND/4, TCP_MSS*4) -----*/
#define TCP_WND_UPDATE_THRESHOLD 536
/*----- Value in opt.h for LWIP_NETIF_LINK_CALLBACK: 0 -----*/
#define LWIP_NETIF_LINK_CALLBACK 1
/*----- Value in opt.h for TCPIP_THREAD_STACKSIZE: 0 -----*/
#define TCPIP_THREAD_STACKSIZE 4048
/*----- Value in opt.h for TCPIP_THREAD_PRIO: 1 -----*/
#define TCPIP_THREAD_PRIO osPriorityNormal
/*----- Value in opt.h for TCPIP_MBOX_SIZE: 0 -----*/
#define TCPIP_MBOX_SIZE 6
/*----- Value in opt.h for SLIPIF_THREAD_STACKSIZE: 0 -----*/
#define SLIPIF_THREAD_STACKSIZE 1024
/*----- Value in opt.h for SLIPIF_THREAD_PRIO: 1 -----*/
#define SLIPIF_THREAD_PRIO 3
/*----- Value in opt.h for DEFAULT_THREAD_STACKSIZE: 0 -----*/
#define DEFAULT_THREAD_STACKSIZE 1024
/*----- Value in opt.h for DEFAULT_THREAD_PRIO: 1 -----*/
#define DEFAULT_THREAD_PRIO 3
/*----- Value in opt.h for DEFAULT_UDP_RECVMBOX_SIZE: 0 -----*/
#define DEFAULT_UDP_RECVMBOX_SIZE 6
/*----- Value in opt.h for DEFAULT_TCP_RECVMBOX_SIZE: 0 -----*/
#define DEFAULT_TCP_RECVMBOX_SIZE 6
/*----- Value in opt.h for DEFAULT_ACCEPTMBOX_SIZE: 0 -----*/
#define DEFAULT_ACCEPTMBOX_SIZE 6
/*----- Value in opt.h for LWIP_COMPAT_SOCKETS: 1 -----*/
#define LWIP_COMPAT_SOCKETS 0
/*----- Value in opt.h for LWIP_SOCKET_SET_ERRNO: 1 -----*/
#define LWIP_SOCKET_SET_ERRNO 0
/*----- Value in opt.h for RECV_BUFSIZE_DEFAULT: INT_MAX -----*/
#define RECV_BUFSIZE_DEFAULT 2000000000
/*----- Default Value for SO_REUSE: 0 ---*/
#define SO_REUSE 1 //Enables SO_REUSEADDR too
/*----- Value in opt.h for LWIP_STATS: 1 -----*/
#define LWIP_STATS 1
/*----- Value in opt.h for CHECKSUM_GEN_IP: 1 -----*/
#define CHECKSUM_GEN_IP 0
/*----- Value in opt.h for CHECKSUM_GEN_UDP: 1 -----*/
#define CHECKSUM_GEN_UDP 0
/*----- Value in opt.h for CHECKSUM_GEN_TCP: 1 -----*/
#define CHECKSUM_GEN_TCP 0
/*----- Value in opt.h for CHECKSUM_GEN_ICMP: 1 -----*/
#define CHECKSUM_GEN_ICMP 0
/*----- Value in opt.h for CHECKSUM_GEN_ICMP6: 1 -----*/
#define CHECKSUM_GEN_ICMP6 0
/*----- Value in opt.h for CHECKSUM_CHECK_IP: 1 -----*/
#define CHECKSUM_CHECK_IP 0
/*----- Value in opt.h for CHECKSUM_CHECK_UDP: 1 -----*/
#define CHECKSUM_CHECK_UDP 0
/*----- Value in opt.h for CHECKSUM_CHECK_TCP: 1 -----*/
#define CHECKSUM_CHECK_TCP 0
/*----- Value in opt.h for CHECKSUM_CHECK_ICMP: 1 -----*/
#define CHECKSUM_CHECK_ICMP 0
/*----- Value in opt.h for CHECKSUM_CHECK_ICMP6: 1 -----*/
#define CHECKSUM_CHECK_ICMP6 0

- open62541.h (options):
- the most options are disabled

#define UA_OPEN62541_VER_MAJOR 1
#define UA_OPEN62541_VER_MINOR 3
#define UA_OPEN62541_VER_PATCH 6
#define UA_OPEN62541_VER_LABEL "" /* Release candidate label, etc. */
#define UA_OPEN62541_VER_COMMIT "v1.3.6"
#define UA_OPEN62541_VERSION "v1.3.6"

/**
* Feature Options
* ---------------
* Changing the feature options has no effect on a pre-compiled library. */

#define UA_LOGLEVEL 700
#ifndef UA_ENABLE_AMALGAMATION
#define UA_ENABLE_AMALGAMATION
#endif
//#define UA_ENABLE_METHODCALLS
//#define UA_ENABLE_NODEMANAGEMENT
//#define UA_ENABLE_SUBSCRIPTIONS
//#define UA_ENABLE_PUBSUB
/* #undef UA_ENABLE_PUBSUB_ENCRYPTION */
/* #undef UA_ENABLE_PUBSUB_FILE_CONFIG */
/* #undef UA_ENABLE_PUBSUB_ETH_UADP */
/* #undef UA_ENABLE_PUBSUB_DELTAFRAMES */
//#define UA_ENABLE_PUBSUB_INFORMATIONMODEL
/* #undef UA_ENABLE_PUBSUB_INFORMATIONMODEL_METHODS */
//#define UA_ENABLE_DA
/* #undef UA_ENABLE_DIAGNOSTICS */
/* #undef UA_ENABLE_HISTORIZING */
//#define UA_ENABLE_PARSING
/* #undef UA_ENABLE_EXPERIMENTAL_HISTORIZING */
//#define UA_ENABLE_SUBSCRIPTIONS_EVENTS
/* #undef UA_ENABLE_JSON_ENCODING */
/* #undef UA_ENABLE_PUBSUB_MQTT */
/* #undef UA_ENABLE_MQTT_TLS */
/* #undef UA_ENABLE_MQTT_TLS_OPENSSL */
/* #undef UA_ENABLE_MQTT_TLS_MBEDTLS */
/* #undef UA_ENABLE_ENCRYPTION_MBEDTLS */
/* #undef UA_ENABLE_TPM2_SECURITY */
/* #undef UA_ENABLE_ENCRYPTION_OPENSSL */
/* #undef UA_ENABLE_ENCRYPTION_LIBRESSL */
#if defined(UA_ENABLE_ENCRYPTION_MBEDTLS) || defined(UA_ENABLE_ENCRYPTION_OPENSSL) || defined(UA_ENABLE_ENCRYPTION_LIBRESSL)
#define UA_ENABLE_ENCRYPTION
#endif
/* #undef UA_ENABLE_SUBSCRIPTIONS_ALARMS_CONDITIONS */

/* Multithreading */
/* #undef UA_ENABLE_IMMUTABLE_NODES */
#define UA_MULTITHREADING 0

/* Advanced Options */
//#define UA_ENABLE_STATUSCODE_DESCRIPTIONS
//#define UA_ENABLE_TYPEDESCRIPTION
//#define UA_ENABLE_NODESET_COMPILER_DESCRIPTIONS
/* #undef UA_ENABLE_DETERMINISTIC_RNG */
/* #undef UA_ENABLE_DISCOVERY */
/* #undef UA_ENABLE_DISCOVERY_MULTICAST */
/* #undef UA_ENABLE_WEBSOCKET_SERVER */
/* #undef UA_ENABLE_QUERY */
/* #undef UA_ENABLE_MALLOC_SINGLETON */
//#define UA_ENABLE_DISCOVERY_SEMAPHORE
/* #undef UA_ENABLE_UNIT_TEST_FAILURE_HOOKS */
/* #undef UA_ENABLE_VALGRIND_INTERACTIVE */
//#define UA_VALGRIND_INTERACTIVE_INTERVAL 1000
//#define UA_GENERATED_NAMESPACE_ZERO
/* #undef UA_GENERATED_NAMESPACE_ZERO_FULL */
/* #undef UA_ENABLE_PUBSUB_MONITORING */
/* #undef UA_ENABLE_PUBSUB_BUFMALLOC */

/* #undef UA_PACK_DEBIAN */

/* Options for Debugging */
/* #undef UA_DEBUG */
/* #undef UA_DEBUG_DUMP_PKGS */
/* #undef UA_DEBUG_FILE_LINE_INFO */

- OPCUA_TASK.c - Task to setup the server and start it:

/**
* Initialize the OPC UA Task
*/
void OpcUaTaskRun(void)
{
osThreadDef(opcuaTask, opcuaMain, osPriorityNormal, 0, 4*1024);
opcuaTaskHandle = osThreadCreate(osThread(opcuaTask), NULL);

}

/**
* The OPC UA task which initialize the OPC UA Server
*
* @PAram argument Task arguments
*/
void opcuaMain(void const * argument)
{
UA_StatusCode status;
char* address;
UA_UInt32 sendBufferSize = 8192; //64 KB was too much for my platform
UA_UInt32 recvBufferSize = 8192; //64 KB was too much for my platform
UA_UInt16 portNumber = 4840;
IPADDRESS_T IpAddress;
static BaseType_t notify_status;
uint32_t notificationValue;
UA_UInt16 timeout = 0;

notify_status = xTaskNotifyWait(0x00, 0xFFFFFFFF, &notificationValue, portMAX_DELAY);

IpAddress = GetIpSettings();

address = GetAddressChar(IpAddress.address);

UA_Server* mUaServer = UA_Server_new();

UA_ServerConfig *uaServerConfig = UA_Server_getConfig(mUaServer);

UA_ServerConfig_setMinimalCustomBuffer(uaServerConfig, portNumber, 0, sendBufferSize, recvBufferSize);

UA_ServerConfig_setCustomHostname(uaServerConfig, UA_STRING(address));

status = Init_Server_Nodes(mUaServer);

status = UA_Server_run(mUaServer, &running);

UNUSED(status);

UA_Server_delete(mUaServer);
}

[mbarg.1]

This bug is well known and was reported years ago.

As LWIP/FreeRtos was stopped in favour of AZRTOS/NetxDuo, no fix have been implemented.

LWIP/FreeRtos has many more bugs and I will suggest to have alook at AZRTOS/NetxDuo.

AZRTOS so far has proven to be reliable and stable.

NetXDuo is not bug free but many step better than LWIP.

Or read all  LWIP/FreeRtos code and fix yourself all bugs.

[Andrew Neil]

---

@mbarg.1 wrote:

This bug is well known and was reported years ago.

---

Have you got a reference for that?

Is it a general LWIP/FreeRtos issue, or something specific to STM32?

[mbarg.1]

I have to roll back to 2021/22 ...

I found several bugs, many more were in contributions from other developers -

Than I had a F2F meeting with STM developer and he refused to discuss FreeRtos bugs as AZRTOS was the only system with ongoing maintenance - my bug list was of no interest.

I remember that plugging/unplugging ethernet cable was one open issue.

I did not have checked recently in new releases but I did not see any announcement that FreeRtos has been fixed - same for LWIP.

Do you have some insider that FreeRTOS/LWIP is coming again?

[Andrew Neil]

---

@mbarg.1 wrote:

Do you have some insider that FreeRTOS/LWIP is coming again?

---

No - just wondering.

@JonasReich92 a forum search for "plug unplug ethernet" showed some possibly interesting hits...

https://community.st.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=false&q=plug%20unplug%20ethernet

[JonasReich92]

Hi, thank you for your replies.

I need to use LwIP to get open62541 working and it is necessary.
The connection problems come up when I have a open port. For OPCUA for example it is the 4840 - here i cannot reconnect again after cable disconnect, but with modbusTCP i have the problem after the 3 cable disconnect - here i have setup a workaround in my application to disconnect the socket when this happend and with this workaround i dont have the problem anymore.

But with the opcua its very difficult to find a workaround in the same way and i dont want to change to much in the open62541 stack.

When the bug is so well known, perhaps you have some helping links to good description of this bugs and the discussions  @mbarg.1 

Sry but the "forum search" don't help me very much. @Andrew Neil 

[JonasReich92]

Hello there, this Re-connection Problem engaged me till now.

I only want to share my fixings for this Problem.

At First I am not complete sure of that all these Fixings are needed.

There are some change like before:

Hardware:
- STM32F427
- DP83826 (TI)
- etc.

Firmware:
- open62541 v1.3.15 (amalgamation)
- LwIP v2.2.1
- FreeRTOS v10.5.1

- Modbus TCP and RTU
- Memory Management: Heap5

- lwipopts.h

- etc.

I try to update all Libs to be at the actual state.

1. Problem - I get RAM Memory Problems with STM32F407, because a lot of Tasks and Code, especially on Startup because of the OPC UA. -> Looks like the OPC UA requiresa lot of Memory at start.

2.Problem - Because of RAM Memory Problems i reduced all FreeRTOS Task in Stack size.

3.Problem - Because of RAM Memory Problems i reduced the LWIP NETBUF and NETCONN

4.Problem - there was no good Timeout Handling in ModbusTCP lib and OPC UA for my Re-Connection Problem.

1. Solution - I use now STM32F427 for more RAM Memory - this solve a lot of Problems for me, but not the Re-Connection

2.Solution - Now I can increase the Stack size for the most of my task and this help me a lot, too. But not the Re-Connection Problem

3.Solution - Now I can increase the LWIP NETBUF and NETCONN - This looks like at first to solve the problem, but it only delayed the Re-Connection problem. 

#define MEMP_NUM_NETCONN 16         // max number of sumultaneous netconn structures
#define PBUF_POOL_SIZE 8            // number of network-puffer - influence: more packets at the same time
#define MEMP_NUM_TCP_PCB 10         // max TCP-Connections - influence: more tcp-connection at the same time
#define MEMP_NUM_TCP_PCB_LISTEN 8   // number of socket that can listen on the network
#define MEMP_NUM_TCP_SEG 15         // max TCP-segments - influence: prevent data accumulation
//#define MEM_SIZE 1024 * 2 * 15

#define LWIP_HOOK_IP4_INPUT my_ip4_input_hook

#define LWIP_COMPAT_MUTEX         1
#define LWIP_COMPAT_MUTEX_ALLOWED 1

/* important for Open62541 und Modbus */
#define LWIP_SOCKET               1
#define LWIP_TCP_KEEPALIVE        1
/*----- Default Value for SO_REUSE: 0 ---*/
#define SO_REUSE                  1 //Enables SO_REUSEADDR too
#define LWIP_DNS                  1

4.Solution - I add some Timeout Handling to the Modbus TCP and OPC UA to free my lwip rescource to start a new connection when it is necessary - This looks very good.... At First.

But this only make a different Problem occur. 

I think this a unique one - Because I have very lot of Data and Task which block my Controller Resources.

Now the Connection dont work any more after a sporadic time and ONLY when i pull the Cable - wait 5-10sec - attach it again. 

For Example the Test Sequence:

1. Start the DUT

2. Connect Modbus TCP (do some Communication, i poll some Input register very fast for 8ms)

3. wait 1-2 minutes 

4. pull the Cable - wait 5-10sec - attach it again - do this 2-3 times

5. wait now 10 minutes

6. When the Connection works fine pull the Cable - wait 5-10sec - attach it again - do this 2-3 times

7. wait now 10minutes

8. The Error can occurs sporadic.

 This can i retry a lot of times, but the Error can occurs on a sporadic time.

It looks like in the ethernetif.c, which was generated with STM32CubeIDE trigger this Problem.

/**
 * @brief Should allocate a pbuf and transfer the bytes of the incoming
 * packet from the interface into the pbuf.
 *
 * @PAram netif the lwip network interface structure for this ethernetif
 * @return a pbuf filled with the received packet (including MAC header)
 *         NULL on memory error
   */
static struct pbuf * low_level_input(struct netif *netif)
{
  struct pbuf *p = NULL;

  if(RxAllocStatus == RX_ALLOC_OK)
  {
    HAL_ETH_ReadData(&heth, (void **)&p);
  }

  return p;
}

What exactly TRIGGERS the situation when pulling cables? Flow example:

Link Up → Burst an ARP / DHCP / mDNS if applicable.

Very fast input several frames → Rx DMA, for example, 8–10 buffers → pool (12) are almost empty.

One or two more frames → alloc fails → RX_ALLOC_ERROR.

At the same time, your input task may still be in initial processing or is currently waiting for RxPktSemaphore.

Because of if(RxAllocStatus == RX_ALLOC_OK) it will NOW NOT be picked up any further → the time when the older buffers could come to the stack and be released later will be postponed.

Only when lwIP (TCP/IP Thread) releases the buffers that have already been passed on will your pbuf_free_custom() reach the code path again that sets RxAllocStatus to OK and posts Semaphore. T

This happens “sporadically” – depending on stack load, offload, timer run.

Cable pulling itself does not directly create the error, but rather increases the time window between feeding (frame burst) and processing.

Why doesn't it happen:

Because at some point you will ultimately get pbufs free (TCP/IP stack releases them → custom free → reset flag). The mechanism recovers – but with latency and unclean pattern.

5. Solution:

/**
 * @brief Should allocate a pbuf and transfer the bytes of the incoming
 * packet from the interface into the pbuf.
 *
 * @PAram netif the lwip network interface structure for this ethernetif
 * @return a pbuf filled with the received packet (including MAC header)
 *         NULL on memory error
   */
static struct pbuf * low_level_input(struct netif *netif)
{
  struct pbuf *p = NULL;
 
  HAL_ETH_ReadData(&heth, (void **)&p);

  return p;
}

In Conclusion:

Perhaps this Problem occurs only because on my Sytem, because i use a lot of Resources of the Controller and poll with some data with 8ms on ModbusTCP. 
When you have some Re-Connection issues perhaps try my Solution 5 without:

if(RxAllocStatus == RX_ALLOC_OK)

[JonasReich92]

Perhaps this Whireshark section helps to understand: