Question
Ethernet TX fails to transmit
I have a custom board using STM32MP131D integrated with ethernet PHY TI DP83867CR. I am using RGMII interface with a Magjack RJ45.
I have been benchmarking the ethernet performance at 100Mbps using iperf3 with the custom board as the iperf client and my Linux test PC as the iperf server.
When I run iperf in client mode on the custom board (DUT) in TCP, I see really bad TX performance:
root@stm32mp131-sga:~# iperf3 -c 10.10.14.200
Connecting to host 10.10.14.200, port 5201
[ 5] local 10.10.14.102 port 35028 connected to 10.10.14.200 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 0.00 Bytes 0.00 bits/sec 3 1.41 KBytes
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 2 1.41 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec 9 sender
[ 5] 0.00-10.06 sec 1.41 KBytes 1.15 Kbits/sec receiverI then redid the test with different MSS size which still indicated bad TX transmissions:
root@stm32mp131-sga:~# iperf3 -c 10.10.14.200 -M 536
Connecting to host 10.10.14.200, port 5201
[ 5] local 10.10.14.102 port 35494 connected to 10.10.14.200 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 48 524 Bytes
[ 5] 1.00-2.00 sec 768 KBytes 6.29 Mbits/sec 50 1.02 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 11 524 Bytes
[ 5] 3.00-4.00 sec 128 KBytes 1.05 Mbits/sec 23 1.02 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 10 524 Bytes
[ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 23 1.02 KBytes
[ 5] 6.00-7.00 sec 256 KBytes 2.10 Mbits/sec 42 1.02 KBytes
[ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 27 524 Bytes
[ 5] 8.00-9.00 sec 1.00 MBytes 8.38 Mbits/sec 37 5.63 KBytes
[ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 110 1.02 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.75 MBytes 3.14 Mbits/sec 381 sender
[ 5] 0.00-10.06 sec 3.76 MBytes 3.14 Mbits/sec receiver
iperf Done.
root@stm32mp131-sga:~# iperf3 -c 10.10.14.200 -M 800
Connecting to host 10.10.14.200, port 5201
[ 5] local 10.10.14.102 port 41968 connected to 10.10.14.200 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 23 788 Bytes
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 6 788 Bytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 12 788 Bytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 2 788 Bytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 5 788 Bytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 1 788 Bytes
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 5 788 Bytes
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 4 788 Bytes
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 1 788 Bytes
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 788 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 128 KBytes 105 Kbits/sec 59 sender
[ 5] 0.00-10.05 sec 105 KBytes 85.3 Kbits/sec receiverTo check if it was only in the TX direction, I ran iperf in reverse mode so that the test PC is sending packets instead:
root@stm32mp131-sga:~# iperf3 -c 10.10.14.200 -R
Connecting to host 10.10.14.200, port 5201
Reverse mode, remote host 10.10.14.200 is sending
[ 5] local 10.10.14.102 port 38968 connected to 10.10.14.200 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 11.2 MBytes 93.6 Mbits/sec
[ 5] 1.01-2.00 sec 11.1 MBytes 93.9 Mbits/sec
[ 5] 2.00-3.00 sec 11.1 MBytes 93.3 Mbits/sec
[ 5] 3.00-4.00 sec 11.2 MBytes 94.4 Mbits/sec
[ 5] 4.00-5.00 sec 11.1 MBytes 93.3 Mbits/sec
[ 5] 5.00-6.00 sec 11.2 MBytes 94.4 Mbits/sec
[ 5] 6.00-7.01 sec 11.2 MBytes 93.8 Mbits/sec
[ 5] 7.01-8.00 sec 11.0 MBytes 92.9 Mbits/sec
[ 5] 8.00-9.00 sec 11.2 MBytes 94.4 Mbits/sec
[ 5] 9.00-10.00 sec 11.1 MBytes 93.3 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.06 sec 114 MBytes 94.9 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 112 MBytes 93.7 Mbits/sec receiverThis test was running at the expected data rate (my link is configured to 100Mbps)
Running the reverse test at different MSS also gave similar results:
root@stm32mp131-sga:~# iperf3 -c 10.10.14.200 -R -M 536
Connecting to host 10.10.14.200, port 5201
Reverse mode, remote host 10.10.14.200 is sending
[ 5] local 10.10.14.102 port 50058 connected to 10.10.14.200 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 10.1 MBytes 84.3 Mbits/sec
[ 5] 1.01-2.00 sec 10.0 MBytes 84.4 Mbits/sec
[ 5] 2.00-3.01 sec 10.1 MBytes 84.4 Mbits/sec
[ 5] 3.01-4.00 sec 10.0 MBytes 84.4 Mbits/sec
[ 5] 4.00-5.00 sec 10.1 MBytes 84.9 Mbits/sec
[ 5] 5.00-6.00 sec 10.0 MBytes 83.8 Mbits/sec
[ 5] 6.00-7.00 sec 10.1 MBytes 85.0 Mbits/sec
[ 5] 7.00-8.01 sec 10.1 MBytes 84.5 Mbits/sec
[ 5] 8.01-9.00 sec 10.0 MBytes 84.4 Mbits/sec
[ 5] 9.00-10.00 sec 10.1 MBytes 84.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.06 sec 103 MBytes 85.4 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 101 MBytes 84.5 Mbits/sec receiver
iperf Done.
root@stm32mp131-sga:~# iperf3 -c 10.10.14.200 -R -M 800
Connecting to host 10.10.14.200, port 5201
Reverse mode, remote host 10.10.14.200 is sending
[ 5] local 10.10.14.102 port 55590 connected to 10.10.14.200 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 10.6 MBytes 88.4 Mbits/sec
[ 5] 1.01-2.00 sec 10.6 MBytes 89.8 Mbits/sec
[ 5] 2.00-3.00 sec 10.6 MBytes 89.1 Mbits/sec
[ 5] 3.00-4.00 sec 10.6 MBytes 89.2 Mbits/sec
[ 5] 4.00-5.00 sec 10.6 MBytes 89.1 Mbits/sec
[ 5] 5.00-6.00 sec 10.6 MBytes 89.1 Mbits/sec
[ 5] 6.00-7.00 sec 10.6 MBytes 89.1 Mbits/sec
[ 5] 7.00-8.00 sec 10.6 MBytes 89.2 Mbits/sec
[ 5] 8.00-9.00 sec 10.6 MBytes 89.0 Mbits/sec
[ 5] 9.00-10.00 sec 10.6 MBytes 89.1 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.06 sec 107 MBytes 89.4 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 106 MBytes 89.1 Mbits/sec receiverTo check if the custom board hardware lines are valid, I changed iperf to run in UDP instead.
root@stm32mp131-sga:~# iperf3 -c 10.10.14.200 -u -b 100M
Connecting to host 10.10.14.200, port 5201
[ 5] local 10.10.14.102 port 39787 connected to 10.10.14.200 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 11.4 MBytes 95.7 Mbits/sec 8281
[ 5] 1.00-2.00 sec 11.4 MBytes 95.5 Mbits/sec 8224
[ 5] 2.00-3.00 sec 11.4 MBytes 95.8 Mbits/sec 8274
[ 5] 3.00-4.00 sec 11.4 MBytes 95.4 Mbits/sec 8234
[ 5] 4.00-5.00 sec 11.4 MBytes 95.7 Mbits/sec 8264
[ 5] 5.00-6.00 sec 11.4 MBytes 95.8 Mbits/sec 8288
[ 5] 6.00-7.00 sec 11.3 MBytes 95.3 Mbits/sec 8210
[ 5] 7.00-8.00 sec 11.4 MBytes 95.7 Mbits/sec 8264
[ 5] 8.00-9.00 sec 11.3 MBytes 94.8 Mbits/sec 8184
[ 5] 9.00-10.00 sec 11.4 MBytes 95.5 Mbits/sec 8261
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 114 MBytes 95.5 Mbits/sec 0.000 ms 0/82484 (0%) sender
[ 5] 0.00-10.06 sec 642 KBytes 523 Kbits/sec 1.497 ms 0/82419 (0%) receiverWhich showed that the DUT was able to transmit at the specific link speed!
And then the UDP test again, but in reverse:
root@stm32mp131-sga:~# iperf3 -c 10.10.14.200 -u -b 100M -R
Connecting to host 10.10.14.200, port 5201
Reverse mode, remote host 10.10.14.200 is sending
[ 5] local 10.10.14.102 port 54679 connected to 10.10.14.200 port 5201
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 11.5 MBytes 96.0 Mbits/sec 0.139 ms 352/8653 (4.1%)
[ 5] 1.00-2.00 sec 11.4 MBytes 95.2 Mbits/sec 0.141 ms 0/8220 (0%)
[ 5] 2.00-3.00 sec 11.4 MBytes 95.3 Mbits/sec 0.153 ms 0/8224 (0%)
[ 5] 3.00-4.00 sec 11.4 MBytes 95.3 Mbits/sec 0.113 ms 0/8232 (0%)
[ 5] 4.00-5.00 sec 11.3 MBytes 95.2 Mbits/sec 0.148 ms 0/8210 (0%)
[ 5] 5.00-6.00 sec 11.4 MBytes 95.2 Mbits/sec 0.120 ms 0/8223 (0%)
[ 5] 6.00-7.00 sec 11.4 MBytes 95.3 Mbits/sec 0.098 ms 0/8227 (0%)
[ 5] 7.00-8.00 sec 11.4 MBytes 95.2 Mbits/sec 0.141 ms 0/8220 (0%)
[ 5] 8.00-9.00 sec 11.4 MBytes 95.2 Mbits/sec 0.132 ms 0/8222 (0%)
[ 5] 9.00-10.00 sec 11.4 MBytes 95.2 Mbits/sec 0.107 ms 0/8224 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.06 sec 114 MBytes 95.2 Mbits/sec 0.000 ms 0/0 (0%) sender
[ 5] 0.00-10.00 sec 114 MBytes 95.3 Mbits/sec 0.107 ms 352/82655 (0.43%) receiver
Which also showed full link speeds.
So running iperf in UDP mode results in full TX/RX speeds which running it in TCP results in full RX speeds but massive loss on TX speeds. The UDP tests indicate that the RGMII, PHY, MDI, and Magnetics are all functional, as if there was any sort of error present on the hardware, UDP would also fail.
I would like some understanding on why TCP TX is non-functional while UDP is fully functional. Is there some setting in the STM32MP131 that I am missing?
Thank you
