STM32L4 1.8.1: virtual COM port loses data during transmission to PC

[Sven Köhler]

Posted on June 12, 2018 at 04:05

I have a simple test application for an STM32L4 microcontroller. It uses the code for the virtual COM port provided by STM32CubeMX. It does the following in an endless loop:

• wait until TxState in (USBD_CDC_HandleTypeDef*)hUsbDeviceFS.pClassData is zero
• generate random string
• call USBD_CDC_SetTxBuffer
• call USBD_CDC_TransmitPacket

The data sent is a randomly generated string, which includes a terminating newline character.

On the PC (Windows), a python script reads the data, line by line, and checks the links for consistency.

Turns out, that if the python script reads fast enough, data is consistent. However, if the python script sleeps randomly (with 5% probability) for 100ms before reading a line, then data is inconsistent. In that case, the microcontroller sends faster than the PC is reading the data.

The obvious conclusion seems to be, that the USB driver code doesn't re-transmit data. However, when the PC does not send an ACK in response to a bulk data packet, then the USB peripheral must re-transmit the data when the host asks for the next data packet.

Anyhow, USB has built-in flow control. There should not be any data loss here.

Can somebody point me to the location in the CubeMX's USB code that handles the re-transmission of bulk data? I can't seem to find it.

My project, unfortunately, is still based on STM32L4 package version 1.8.1. I switched a test project to 1.12.0, but then the virtual COM port won't work on Windows anymore. I cannot open the COM port. The same issue was reported here recently by another person.

Is there ChaneLog for the STM32L4 packages, so I can study what was changed between 1.8.1 and 1.12.0?

[Imen.D]

Posted on June 12, 2018 at 16:15

Hello

sven.koehler

,

Try to increase the heap and stack sizes in the STM32CubeMx setting.

The last version of STM32CubeL4 version 1.0 containsthe release note file inside the package which listed the history of each release with main changes andKnown limitation.

With Regards,

Imen

[Sven Köhler]

Posted on June 12, 2018 at 16:25

The stack is 32kByte and the heap is 4kByte. The heap (malloc/free) is never used.

[gbm]

Posted on June 12, 2018 at 20:02

First and most important: Transmit routine must be called from an interrupt of the same priority as USB interrupt. It cannot be called from any other priority code. You may try to disable interrupts for Transmit call if you want to call it from main().

[Bob S]

Posted on June 12, 2018 at 21:05

I think you are looking in the wrong place.  When you randomly delay the python code, what is 'inconsistent' about the data that the python script sees?  Is it seeing dropped or missing data?  Corrupted data?

I would guess that what you are seeing is missing data.  If so, the most likely issue is the buffering on the PC side of things.  Unless of course you specifically seen your embedded code dropping data.

When you run a USB device as a CDC peripheral, there is NO data rate throttling across the USB bus.  If you configure the virtual serial port for, say, 9600 baud, that has no affect on how fast data is transferred across the USB.  If your code on the ST CPU is sending data as fast as the USBD_CDC functions will transmit it, you are effectively sending data at close to 10MBits/sec (12MBits/sec minus a wild guess for USB protocol overhead), or however fast the firmware can stuff data into the USB.  Depending on how large a buffer the ST VCom driver has (or the python COM port library), you could easily overflow that if the python script stops reading from the serial port for 100ms. 

[Sven Köhler]

Posted on June 12, 2018 at 22:14

Dear Bob,

for Bulk transfers, which is what virtual COM ports use for the actual data, the USB standard will give you something called flow control. The buffers in the ST VCom (which is really just using Microsoft's usbserial.sys) will never overflow because flow control means that the microcontroller can only send data if the PC can accept them. If you study 'USB in a Nutshell' you will see that every bulk transfer data packet has to be acknowledged by the PC. Also, USB has checksums. So data corruption is extremely unlikely to go unnoticed.

To summarize: It is well known that virtual COM ports are not only faster but also they don't come with those old-school problems like data corruption and buffer overflows.

About inconsistencies:

The microcontroller is sending strings of the following format:  'x y x*y\n'

where x and y are random integers and x*y stands for x copies of character y, and x is in the range [20,200] while y is in the range ['0','z']. 

My python test script verifies the integrity of those strings (is the string of the format above? does it contains the specified amount of characters?).

[Bob S]

Posted on June 13, 2018 at 18:09

Sven Köhler wrote:

 If you study 'USB in a Nutshell' you will see that every bulk transfer data packet has to be acknowledged by the PC.

Yes, but that ACK is *not* a high-level ACK from the usbser.sys.  At the USB bulk transfer level, all the ACK means is that the USB host device (the USB chip in your PC) received the data and the checksum was OK.

As long as the VCOM/usbser.sys driver on the PC requests data from your device (by issuing a 'read' command from the serial port), the USB protocol will allow your device to send data (via IN requests on the bulk transfer endpoint).  What follows is pure speculation on my part as I have no actual knowledge of the VCOM/usbser.sys driver internals, but the usual design for that type of driver is to read data from the device whenever it is available, because embedded devices usually have extremely limited buffer space.  If the VCOM driver's internal buffers fill up, it has to throw something away - either the new data that it just received, or the oldest data in its buffer.  Usually the new data gets discarded.  There may be some IOCTL function call to see if the driver has had a buffer overflow, I don't know.  End of speculation.

As for the inconsistencies - you mention what you look for, but you have not said exactly what was wrong with the data that the python script received.  Were there missing characters?  Were there extra characters?  Where there the correct number of characters but some of them were not the expected data?  If might be helpful if you added a field to the beginning of each string that is an integer field that increments with each string.  For example, the first string you send would be '0 x y x*y\n' (because we embedded programmers always start at zero ), and the next one would be '1 x y x*y\n', etc.  This will show if there are large blocks of data missing, which is what I would expect to see if it is a buffer overflow issue in the PC's USB driver.

[Sven Köhler]

Posted on June 14, 2018 at 02:38

The USB protocol only allows the device to send data if the PC asks for it. If usbser.sys asks for data even though it cannot stomach it, that is called a bug, clearly. Right now, I can tell you that Windows 7 works without problems. Linux works without problems. Windows 10 doesn't. Turns out, Microsoft claims to have rewritten usbser.sys for Windows 10.

This guy here, with very broken English, reports problems very similar to mine: 

https://social.msdn.microsoft.com/Forums/sqlserver/en-US/ade57269-1dbe-4a15-a1a1-54f60a1b99b4/bug-report-comport-over-cdc-usbser?forum=wdk

 

I will post my code here so you can test yourself. Right now, my test code is based on CubeMX 4.25.1 and STM32L4 1.11.0. I suspect that you don't need an L4. You can take any STM32 controller and simply send data as fast as you can. You will experience data loss, if the PC randomly pauses while reading the data on Windows 10 - not so on Windows 7 or Linux.

I have let my test program run for several hours. I have seen data loss (64 bytes at the beginning of a line are missing) and even data corruption (instead of a line resembling the format I describe above, I received completely random bytes that has nothing to do with my data).

I conclude a long evening of debugging with the realization that I ran into a bug of usbser.sys that Microsoft has known about for 3 years now. Virtual COM ports cannot be used reliable with Windows 10 (unless an alternative driver is used). I will look for alternatives, such as using WinUSB on Windows.

P.S.: I have verified that the data sent by the micro controller is correct using a logic analyzer. I apologize for suspecting ST's driver code.