2022-11-30 12:48 AM
Hello,
ST's and all Ethernet experts, please.
STM32F767,
Nucleo-144, and custom board
no OS
lwIP 2.1.3, IPv4 only
STM32CubeIDE
no ETH interrupts used
Application:
Industrial frontend,
streaming "audio" data from SAIs via ethernet,
at high data rates, for long periods of time (weeks)
TCP is a must, losing packets is not an option.
Audio streaming mostly uses UDP, and they use interpolation
to mask lost packets - we are not allowed do that.
Problem:
ETH transmission: TCP header checksum is = ZERO = 0.
Depending on settings below (SRAM usage, CPU clock),
this happens after a few MB, or many GB of data,
at 25.6 Mbps it's running sometimes for hours,
sometimes it stops after a few minutes.
IP4 header checksum is okay (at least not 0).
All checked with Wireshark.
Then the PC side stops ACKnowledging,
then lwIP shuts down TCP.
Checked:
Findings:
Settings so far with an impact on the "zero-checksum-failure":
Best "setup" until now:
-> Much better, but still not perfect, failed on one board after 1 hour or so.
Having used FPGAs for years, I had the hope of leaving these
"assumed race conditions" behind (naive me...). At least in the
FPGA you can get control of these problems.
For the final product, right now it seems the STM32F7 is not an option.
Which is sad, after having spent a lot of time on that one, having the
firmware at about 99% finished.
So, what am I doing wrong?
Or is there a known issue?
Source code of ethernetif.c etc. attached.
P.S.: I spammed the code with lots of __DSB()... I think I can remove many of these. But as it seems to be memory related, I had some hope.
2022-12-06 05:02 AM
@Community member Thanks again for having another look at it.
Here are the 2 last good packets (Wireshark's "header analysis" for better readability), then the 1st bad packet (TCP cs = 0), and the 2nd bad packet (IPv4 cs = 0).
I have checked the payload, that's okay, starts with a 20 byte header incl. timestamp, packet number, then comes audio data.
Summary:
All is well and as it should be. Except for that "§%!* checksums. 😉
2nd last good packet:
Frame 40177: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits)
Ethernet II, Src: ce:22:29:43:04:20 (ce:22:29:43:04:20), Dst: Dell_Laptop
Internet Protocol Version 4, Src: 192.168.178.70, Dst: 192.168.178.39
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0xb8 (DSCP: EF PHB, ECN: Not-ECT)
Total Length: 1500
Identification: 0x68d7 (26839)
000. .... = Flags: 0x0
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 255
Protocol: TCP (6)
Header Checksum: 0x66cd [correct]
[Header checksum status: Good]
[Calculated Checksum: 0x66cd]
Source Address: 192.168.178.70
Destination Address: 192.168.178.39
Transmission Control Protocol, Src Port: 9603, Dst Port: 52127, Seq: 38951341, Ack: 1, Len: 1460
Source Port: 9603
Destination Port: 52127
[Stream index: 4]
[Conversation completeness: Incomplete (12)]
[TCP Segment Len: 1460]
Sequence Number: 38951341 (relative sequence number)
Sequence Number (raw): 38957876
[Next Sequence Number: 38952801 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 285078846
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window: 5840
[Calculated window size: 5840]
[Window size scaling factor: -1 (unknown)]
Checksum: 0x4506 [correct]
[Checksum Status: Good]
[Calculated Checksum: 0x4506]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (1460 bytes)
Data (1460 bytes) looks good (easily identified by own header before audio data)
_______________________________________________________________________________
last good packet:
Frame 40178: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits)
Ethernet II, Src: ce:22:29:43:04:20 (ce:22:29:43:04:20), Dst: Dell_Laptop
Internet Protocol Version 4, Src: 192.168.178.70, Dst: 192.168.178.39
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0xb8 (DSCP: EF PHB, ECN: Not-ECT)
Total Length: 1500
Identification: 0x68d8 (26840)
000. .... = Flags: 0x0
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 255
Protocol: TCP (6)
Header Checksum: 0x66cc [correct]
[Header checksum status: Good]
[Calculated Checksum: 0x66cc]
Source Address: 192.168.178.70
Destination Address: 192.168.178.39
Transmission Control Protocol, Src Port: 9603, Dst Port: 52127, Seq: 38952801, Ack: 1, Len: 1460
Source Port: 9603
Destination Port: 52127
[Stream index: 4]
[Conversation completeness: Incomplete (12)]
[TCP Segment Len: 1460]
Sequence Number: 38952801 (relative sequence number)
Sequence Number (raw): 38959336
[Next Sequence Number: 38954261 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 285078846
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window: 5840
[Calculated window size: 5840]
[Window size scaling factor: -1 (unknown)]
Checksum: 0x838e [correct]
[Checksum Status: Good]
[Calculated Checksum: 0x838e]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (1460 bytes)
Data (1460 bytes) looks good (easily identified by own header before audio data)
_______________________________________________________________________________
1st bad packet: TCP checksum = 0
Frame 40179: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits)
Ethernet II, Src: ce:22:29:43:04:20 (ce:22:29:43:04:20), Dst: Dell_Laptop
Internet Protocol Version 4, Src: 192.168.178.70, Dst: 192.168.178.39
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0xb8 (DSCP: EF PHB, ECN: Not-ECT)
Total Length: 1500
Identification: 0x68d9 (26841)
000. .... = Flags: 0x0
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 255
Protocol: TCP (6)
Header Checksum: 0x66cb [correct]
[Header checksum status: Good]
[Calculated Checksum: 0x66cb]
Source Address: 192.168.178.70
Destination Address: 192.168.178.39
Transmission Control Protocol, Src Port: 9603, Dst Port: 52127, Seq: 38954261, Ack: 1, Len: 1460
Source Port: 9603
Destination Port: 52127
[Stream index: 4]
[Conversation completeness: Incomplete (12)]
[TCP Segment Len: 1460]
Sequence Number: 38954261 (relative sequence number)
Sequence Number (raw): 38960796
[Next Sequence Number: 38955721 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 285078846
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window: 5840
[Calculated window size: 5840]
[Window size scaling factor: -1 (unknown)]
Checksum: 0x0000 incorrect, should be 0x9619(maybe caused by "TCP checksum offload"?)
[Checksum Status: Bad]
[Calculated Checksum: 0x9619]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (1460 bytes)
Data (1460 bytes) looks good (easily identified by own header before audio data)
_______________________________________________________________________________
2nd bad packet: TCP and IPv4 checksum = 0
Frame 40180: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits)
Ethernet II, Src: ce:22:29:43:04:20 (ce:22:29:43:04:20), Dst: Dell_Laptop
Internet Protocol Version 4, Src: 192.168.178.70, Dst: 192.168.178.39
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0xb8 (DSCP: EF PHB, ECN: Not-ECT)
Total Length: 1500
Identification: 0x68da (26842)
000. .... = Flags: 0x0
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 255
Protocol: TCP (6)
Header Checksum: 0x0000 incorrect, should be 0x66ca(may be caused by "IP checksum offload"?)
[Header checksum status: Bad]
[Calculated Checksum: 0x66ca]
Source Address: 192.168.178.70
Destination Address: 192.168.178.39
Transmission Control Protocol, Src Port: 9603, Dst Port: 52127, Seq: 38955721, Ack: 1, Len: 1460
Source Port: 9603
Destination Port: 52127
[Stream index: 4]
[Conversation completeness: Incomplete (12)]
[TCP Segment Len: 1460]
Sequence Number: 38955721 (relative sequence number)
Sequence Number (raw): 38962256
[Next Sequence Number: 38957181 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 285078846
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window: 5840
[Calculated window size: 5840]
[Window size scaling factor: -1 (unknown)]
Checksum: 0x0000 incorrect, should be 0x1801(maybe caused by "TCP checksum offload"?)
[Checksum Status: Bad]
[Calculated Checksum: 0x1801]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (1460 bytes)
Data (1460 bytes) looks good (easily identified by own header before audio data)
2022-12-06 05:04 AM
What I don't get:
It's been perfectly stable now with reduced TCP_MSS (and reduced SAI buffer size).
2022-12-06 05:41 AM
I don't see anything suspicious in those packets, and I have no more ideas.
> It's been perfectly stable now with reduced TCP_MSS (and reduced SAI buffer size).
That of course may or may not be coincidental...
JW
2022-12-06 05:44 AM
@Community member Thanks again!
> That of course may or may not be coincidental...
That's the problem!
I'm going through the lwIP output functions again, where TCP_MSS might have an impact.
2022-12-06 07:36 AM
Same error, also with reduced MSS, but it ran much longer considering the circumstances (lots of parallel http and network traffic).
I checked all interrupts again, and I actually found the IGMP timer which might have called the output function, changed to a flag to call igmp_tmr() in main, didn't help.
But now I'm quite sure there are:
So far it happens less when - but I'm not so sure...:
There must be something stupid somewhere...
Again, current ethernetif.c & co attached.
2022-12-08 07:05 AM
I think I found it....
After turning off all the stuff not needed for TCP transfers, including the source SAIs and sending only static buffers, nothing changed, still zero checksum.
Then I went a few weeks back when I did not yet have good software for testing on the PC side, at that time I built a packet queue between SAI DMA buffers and tcp_write().
And there's some "hole" which somehow might access the buffers given to ETH DMA.
Anyway, without the queue, transfers are whacky, but never fail due to checksum.
Although this needs some more testing, didn't have enough time this morning and had to get on the road - company's christmas party...
If that's it I'll come back, report, and mark topic as solved.
@Piranha & @Community member big thanks again for your help!
2022-12-14 06:47 AM
Again, conicidental...
I still have the checksum problem.
And I changed a lot over the last few days.
Let me start with a confession, and I really feel stupid about how I approached the STM32 in the first place:
I have worked with FPGAs and "slow" 8 -bit controllers for years, together with some specialized interface ICs (e.g., high-speed USB).
So any time-critical stuff happened in the FPGA and between that and the "interface IC".
When working with FPGAs in VHDL, I basically had control over each bit, at each clock edge.
Being fed up with too many ICs (FPGA, SRAM, uC, IF-IC, ...) and too expensive FPGAs, we decided to try an all-in-one solution, and because I had some previous (good) experience with the STM32, we chose that one.
So far, so good.
Problem: I somehow approached the STM32 as if it was kinda magic device, throw in some register values, then let it run...
Which means I did not worry about things like SRAM and DMA usage - where's what, how is it connected.
AND even worse, being used to the slow non-OS 8-bitters, I just threw everything in the main loop as I was used to. With the result that some things were constantly checked at full clock speed - absolutely not necessary, and surely blocking the internal busses.
An example:
For the SAI data buffers, which should go zero-copy to ethernet, I had some control variables "in front" of the data, which I checked very often. These being in the same RAM area, I definitely took some time from the DMAs working on the data buffers.
So again I changed a lot:
So, did that help? No. :(
I still get zero checksums, which is at least "fixable" by flushing the TX FIFO (but that takes looong! Why?).
What I found so far, these things are making things worse -> higher chances for the checksum failures:
Could there be any kind of race condition between DTCM / SRAM1 / CPU / DMA2 / ETH-DMA, on the interface busses ?
(Again: I have lots of memory barriers in place, esp. before/after the using the descriptors' OWN bit)
When the SAI DMA complete interrupt occurrs, is it really done writing the buffers from the SAI FIFOs into internal SRAM?
What could "confuse" the ETH TX FIFO when filling it with many small buffers, like it's done with http / SSI ?
2022-12-15 03:46 AM
I understand your frustration, but it's very unlikely you will be able to get help here, as we are mere users with inevitably limited experience and zero access to inside information.
The Synopsys modules (both ETH and OTG_USB) used in STM32 are not only very complex, but also quirky and laden with historical layers. The documentation is lacking and it's only partially ST's fault, as they mostly copypaste what they purchased. To make things worse, the modules tend to change in time, i.e. they are around in various versions, and those have various quirks.
Look for example at the Successive write operations to the same register might not be fully
taken into account erratum in the STM32F407 errata...
JW
2022-12-15 04:14 AM
Ah, that's where stuff like that in the HAL driver comes from:
/* Wait until the write operation will be taken into account :
at least four TX_CLK/RX_CLK clock cycles */
tmpreg = (heth->Instance)->MACCR;
HAL_Delay(ETH_REG_WRITE_DELAY);
(heth->Instance)->MACCR = tmpreg;
So maybe F4 & F7 ETH MAC are very much the same?
Will check...
Thanks again!
2022-12-15 07:07 AM
I changed register writing to this method, didn't change anything.
BUT... as I use the 128 kB DTCM as "main" RAM, I changed some variables' alignment to "8", and so I did with the lwIP stuff.
It's running very stable now for 3 hours, only when I access the webserver at the same time with a page using SSI tags, the checksum error comes quickly.
Spamming the device with GET requests to non-SSI pages (either html from flash/RAM or JSON replies) are no problem at all.
And in general, even on my work PC I have less buffer overflows.
Okay, let's dig into lwIP's SSI stuff...