STM32F429 LWIP project stops all network comms after running for a few hours.

[paulrbryson]

My STM32F429 LWIP project stops all network comms after running for a few hours.  
I am hoping that this is a known bug with a known fix. But I would appreciate any insight that anyone can provide.
I have a CubeMX generated project using LWIP and FreeRTOS running on an STM32F429. I have an ethernet interface using a LAN8742A.
After running for a few hours my project stops communicating and does not recover. I think I have ruled out any kind of memory or buffer exhaustion.  I suspect a race condition of some sort.
Every time the comms stop, all of the RX descriptors are owned by the CPU and the ethernet RX descriptors are stuck in this same pattern.
The numbers change but RxDescIdx is always (RxBuildDescIdx + 1).  And the counts are always = 1. 
heth->RxDescList->
ItMode =          1    
RxDescIdx    =          27    
RxDescCnt =          1    
RxDataLength =      118    
RxBuildDescIdx =    26    
RxBuildDescCnt =    1    
pRxLastRxDesc    =      7734049    

FreeRTOS is still running the idle task and my one non-network task.  All of my networking tasks are blocked.  They all have time outs on their network calls but they do not seem to time out.

I would like to find a way to eliminate this comms bug.  But I would settle for a graceful way to detect and recover from it.  

[paulrbryson]

I was mistaken about the memory exhaustion.  My RX_POOL was momentarily maxing out periodically.  This did not immediately cause a deadlock; but I have increased the number of RX buffers and I am re-testing.  This should not create a permanent deadlock. 
I have also discovered that I sometimes get a hard fault in pbuf_free() called from HAL_ETH_TxFreeCallback().  The pointer p is invalid and not NULL.  But this happens much less often and is not a direct cause of my deadlock.

[Pavel A.]

>  I sometimes get a hard fault in pbuf_free() called from HAL_ETH_TxFreeCallback().  The pointer p is invalid and not NULL. 

This is a known issue with the example code. We could not find the reason quickly, but  seen it “fixed” or worked-around in some 3rd party ETH driver variants.

[Andrew Neil]

Some LwIP debug/diagnostic tips/resources:

https://community.st.com/t5/stm32-mcus-products/lwip-ethernet-in-stm32h7-working-in-stm32cubeide-but-not-in-iar/m-p/748501/highlight/true#M267524

[paulrbryson]

I finally found the problem.  It turns out that in the STM32CubeMx generated code, the function, low_level_output(), was being called simultaneously by two freertos tasks (EthLink and tcpip_thread) - and they were over-writing each other.  I have patched it by putting a mutex on low_level_output() which seems so far to have resolved the issue.

It took me a while to realize that many of the errors flagged by the ethernet hardware peripheral were not real.  They actually occurred after I hit my breakpoints or otherwise stopped the CPU.  I was also hindered by the fact that it often took many hours for the lock up to occur.

[Andrew Neil]

I finally found the problem. 

Excellent! So please mark that post as the solution  - not this one!

[paulrbryson]

But TX_Config is not the end of it.
low_level_output() through HAL_ETH_Transmit_IT() and ETH_Prepare_Tx_Descriptors() also write to global “heth” and actually writes the ethernet hardware registers and the DMA TX descriptors.  

[Guillaume K]

Normally this situation of two threads calling low_level_output() at same time should not happen with the example configuration.

In lwipopts.h, is NO_SYS defined to 0 ? (RTOS is used)

is LWIP_TCPIP_CORE_LOCKING set to 1 ? (if not set in lwipopts.h, the default value 1 is defined in LwIP/src/include/lwip/opt.h )

Are you calling lwip “raw” APIs (tcp_write(), tcp_output(), udp_sendto() ...) while NO_SYS = 0  ?

 

 

 

[paulrbryson]

NO_SYS is 0.
I am not using the raw api.
However, I do have LWIP_TCPIP_CORE_LOCKING set to 0.  I don’t remember exactly why I originally set it to 0 (probably suggested by an AI).  I tried many many things trying to address my original problem.  It is likely that I created new problems while trying to fix the original problem. 
I just now tried setting LWIP_TCPIP_CORE_LOCKING to 1 and my application stopped working.