LWIP + freeRTOS + OPENER, Ethernet DMA Bus Error

[ALAMI_Othmane]

I have followed the instructions in GIT  to make ethernet works correctly on STM32H750 (setup TX/RX descriptors and their MPU config, also the LWIP heap and its MPU config, and the RX buffers).

Everything seems working correctly after hours of setup and debugging, but the thing that break Ethernet and make it not working is when handling the unicast UDP or the UDP IO stream of OpEner it trigger a BUS Error in MAC DMA and stop everything from working, the explicit communication based on TCP works correctly and doesn’t break the Ethernet.

I debugged for days of what could generate a bus fault error, then i find the udp when uses netconn it doesn’t copy the prepared buffer on lwip heap for sendig and in other way it give it directly the address of udp payload from the task stack, and this trigger a Bus Error when calling HAL_ETH_Transmit_IT because the task stack is not DMA accessible.

So I made a workarround in low_level_output to detect the payload from outside lwip heap and copy that payload in lwip heap, i used a linker varriables to detect the begin and the end of lwip heap:

static err_t low_level_output(struct netif *netif, struct pbuf *p)

{

uint32_t i = 0U;

struct pbuf *q = NULL;

err_t errval = ERR_OK;

ETH_BufferTypeDef Txbuffer[ETH_TX_DESC_CNT];

struct pbuf *dma_copy = NULL; /* tracks if we allocated a DMA-safe copy */

 

extern uint8_t __ETH_TX_START[];

extern uint8_t __ETH_TX_END[];

 

memset(Txbuffer, 0, ETH_TX_DESC_CNT * sizeof(ETH_BufferTypeDef));

 

/* Check if the entire pbuf chain is DMA-safe */

int needs_copy = 0;

for (q = p; q != NULL; q = q->next)

{

uint8_t *start = (uint8_t *)q->payload;

uint8_t *end = start + q->len;

if (start < __ETH_TX_START || end > __ETH_TX_END)

{

needs_copy = 1;

break;

}

}

 

/* If any buffer is outside the DMA region, allocate a flat DMA-safe pbuf and copy */

if (needs_copy)

{

dma_copy = pbuf_alloc(PBUF_RAW, p->tot_len, PBUF_RAM);

if (dma_copy == NULL)

{

return ERR_MEM;

}

 

/* Verify the new pbuf landed in DMA-safe memory */

if ((uint8_t *)dma_copy->payload < __ETH_TX_START ||

((uint8_t *)dma_copy->payload + dma_copy->len) > __ETH_TX_END)

{

pbuf_free(dma_copy);

return ERR_MEM;

}

 

pbuf_copy(dma_copy, p);

p = dma_copy; /* use the DMA-safe copy from here on */

}

 

for (q = p; q != NULL; q = q->next)

{

if (i >= ETH_TX_DESC_CNT)

{

if (dma_copy != NULL) pbuf_free(dma_copy);

return ERR_IF;

}

 

Txbuffer[i].buffer = q->payload;

Txbuffer[i].len = q->len;

 

if (i > 0)

{

Txbuffer[i - 1].next = &Txbuffer[i];

}

 

if (q->next == NULL)

{

Txbuffer[i].next = NULL;

}

 

i++;

}

 

TxConfig.Length = p->tot_len;

TxConfig.TxBuffer = Txbuffer;

TxConfig.pData = p;

 

pbuf_ref(p);

if (HAL_ETH_Transmit_IT(&heth, &TxConfig) == HAL_OK)

{

errval = ERR_OK;

}

 

while (osSemaphoreAcquire(TxPktSemaphore, TIME_WAITING_FOR_INPUT) != osOK)

{

}

 

HAL_ETH_ReleaseTxPacket(&heth);

 

/* Free our DMA copy if we made one (decrements ref added by pbuf_ref too) */

if (dma_copy != NULL)

{

pbuf_free(dma_copy);

}

 

return errval;

}

 

I share this to help if anyone face the same issue.

[mƎALLEm]

Hello ​@ALAMI_Othmane ,

While starting this thread did you notice this pop-up and read its content?

 

[ALAMI_Othmane]

Hi @mƎALLEm, sorry I forgot to include the code inside the code section.

Can you confirm that the low_level_output function generated by CubeMX is triggering issues with UDP communications?

[LCE]

First of all: great that you found a solution and posted it here!

 

2nd… have you defined and structured all the internal SRAM areas?

I found that the combination of
a) the compiler putting data somewhere in DTCM or AXI SRAM, and
b) that ETH DMA cannot access all SRAM areas 
is very dangerous!

So I made sure via variable placement with the linker file and section attributes that things like that cannot happen.

It’s tedious, but it works.

[ALAMI_Othmane]

The problem here is : 

LwIP allocates specific UDP message buffers on the task stack instead of the DMA-configured heap, causing DMA bus errors during UDP communications. So when LwIP constructs small UDP packets (headers, short payloads), it may avoid heap allocation and instead build them on the stack for efficiency. The DMA controller can't access this memory, resulting in a bus error.

And low_level_outputThe function generated by CubeMX passes the pbuf payload pointer directly to the Ethernet DMA engine without checking if the memory is in a DMA-accessible region.

[LCE]

Interesting, happy that I didn’t even start with an OS! :D

So, you could either move the task stack to some accessible RAM, or modify lwip so it doesn’t use the stack for UDP.

Maybe that would be faster than your copy workaround. If speed is an issue.

[Guillaume K]

Hi

In what circumstance does the ETH DMA bus error occur ?

What kind of memory area is it trying to use ?

Ethernet DMA bus errors could happen if the data is in a memory area not accessible by the Ethernet peripheral .

Example: on STM32H750 the internal CPU DTCM isn’t accessible by the ethernet IP.

The memory architecture is visible in the reference manual for STM32H750 (search for “STM32H750 reference manual” on st.com) in chapter 2.1 “memory architecture” figure 1 . It shows what memory areas are reachable for all peripherals.

Normally on STM32H750 the Ethernet IP can access all the other SRAMs (AHB SRAM and AXI SRAM through AHB to AXI interface).

Also the address must be aligned to the Ethernet IP bus width. On STM32H7 with AHB Ethernet it is 4 bytes. if you were on STM32N6 it would be 8 bytes (AXI bus).

There may be errors if MPU configuration forbids access.

After that, if the data cache is enabled we want to access data from memory areas that are defined with non cached strongly ordered MPU configuration.

Using other areas will not trigger a bus error. It’s just that there may be data synchronisation problems with the CPU if you use memory area that is cached.

One thing that could explain the ethernet DMA bus error is a pbuf payload pointer that is corrupted by another thread overflow so the address is invalid.

 

 

[LCE]

> Example: on STM32H750 the internal CPU DTCM isn’t accessible by the ethernet IP.

> Normally on STM32H750 the Ethernet IP can access all the other SRAMs (AHB and AXI though AHB to AXI interface).

 

The problem might be that by default (CubeMx / Github examples) the linker file does NOT distinguish between DTCM and the other RAM.

And this makes the compiler put stuff somewhere…

In the OP’s project, my guess is that the OS’s task stack was placed in DTCM (which makes sense), but somehow lwIP put some data there / or tried to get it from there. Which then made the ETH DMA fail.

[Guillaume K]

On STM32H750, DTCM is 0x20000000 - 0x2001FFFF.

What is strange is lwip using the C stack for pbuf.

I thought lwip would always use pbufs allocated in its own heap for transmit buffers ?

In STM32Cube ethernet lwip examples the LwIP heap is fixed to addresses in non cached memory areas with parameter LWIP_RAM_HEAP_POINTER. 

Example: in lwipopts.h:

#define LWIP_RAM_HEAP_POINTER    (0x30004000)      // SRAM 1

[LCE]

> What is strange is lwip using the C stack for pbuf.

Agreed.

But I haven’t used lwip with OS...

[ALAMI_Othmane]

Yeah it’s strange also for me to see lwip use the c heap( which is placed inside DTCMRAM which is forbidden from Ethernet IP)  instead of LWIP_RAM_HEAP_POINTER in some specific udp transmissions, like this:

Maybe if I move the heap location to AXI RAM will solve the issue, but I still don’t understand why the lwip uses the c heap instead of its own heap.

[Guillaume K]

After some research with our AI friends:

For UDP socket send() and sendto(), if LWIP_NETIF_TX_SINGLE_PBUF is disabled, lwip uses netbuf_ref() at sockets.c:1701, which creates a PBUF_REF in netbuf.c:151. Payload points to your application buffer, so no payload copy to lwip heap.

if option LWIP_NETIF_TX_SINGLE_PBUF is enabled, netbuf_alloc() is used and the data is copied in the allocated buffer.

Another possible solution if you want to keep LWIP_NETIF_TX_SINGLE_PBUF disabled and use socket sendto() for UDP packets would be to allocate buffers in AXI or AHB SRAM that is accessible by Ethernet and pass them to lwip sendto(). Preferably buffers in memory area declared as non cacheable strongly ordered with the MPU like it is done for the custom RX_POOL pbuf pool.

For TCP sockets I guess there is always a copy so no problem.