CubeMX Feature Request: Add USART RX IDLE Handling

[Steve H]

Posted on May 23, 2017 at 16:36

I'm preparing to implement an STM32 project that does high-speed async serial communications, and needs to send and receive variable-length messages. My receive messages are characterized in that they are *always* continuous byte streams on the serial line, with no inter-byte gaps; when the RX serial line goes idle, the message is complete. Because the bit rate is high, I prefer to use DMA for TX/RX message handling. I've written code to do this task before, and I know exactly how to set up the USART, DMA, and interrupt handlers to do the job.

It would be great to use the CubeMX HAL's UART routines for this project, BUT they have an essential lack here: They don't provide handling for the UART_FLAG_IDLE interrupt flag in HAL_UART_IRQHandler(). This means that DMA receive operations using the HAL routines are limited to fixed-length messages, because there is no way to handle the RX idle interrupt that is generated when the last byte of a short message comes in.

I have three equally unappealing options in this case: 1) Continue rolling my own UART support code outside the HAL framework, replacing the hal_usart module for each new target MCU I write for, and creating an integration headache; 2) Hack a local copy of the hal_usart module to add the feature (again, for each target MCU I write for), creating maintenance headaches; or 3) give up on USART DMA receive operations, creating a potentially unacceptable performance bottleneck with all that USART polling or IRQ overhead.

A quick browse of these forums and others like StackOverflow tells me that a number of other developers have faced this same issue, so there's a decent chance this feature will be generally useful.

Here's the specific feature I'd like to see added to the hal_usart drivers:

• Add code to HAL_UART_IRQHandler() to handle the UART_FLAG_IDLE condition when receiving under DMA, with a callback hook that is invoked when the RX idle condition occurs. Based on the current organization of the hal_usart module, it probably makes best sense to add a separate callback hook, e.g., 'HAL_UART_RxIdleCallback()' or similar.
• Provide a way for the RX idle callback to determine how much data the RX DMA has received, probably by adding or reusing a member of UART_HandleTypeDef.

With this addition, it should be possible to use the stock HAL routines to manage variable-length RX messages while preserving the efficiency advantages of DMA.

I'm going to try hacking a copy of the hal_usart driver to add this feature, and if it's wildly successful, I'll post it as an update for the curious.

#feature-request

[Imen.D]

Posted on May 24, 2017 at 13:21

Hi

Hersey.Stephen

,

I want to thank you for your

contribution and inform you that y

our suggestion is highlighted internally for further investigation. All your feedback are welcome in order to improve our solutions.

Thanks

Imen

[Steve H]

Posted on May 24, 2017 at 18:21

Thanks, Imen.

I have built and tested such a modification of the hal_usart driver for an STM32F100 MCU, on a custom PCB that uses USART3 for serial I/O and has assorted buttons and LEDs. The project was generated under Linux in STM32CubeMX and imported into SW4STM32, where it was compiled and debugged with an STLink V2.

I've attached a tarball of my example project files. The relevant code changes are listed here with descriptions of where they can be found.

stm32f1xx_hal_usart.h gets an added function declaration at line 700:

void HAL_UART_RxIdleCallback(UART_HandleTypeDef *huart);

stm32f1xx_hal_usart.c gets the following additions:

HAL_UART_IRQHandler(), line 1247, gets an additional interrupt flag handler:

tmp_flag = __HAL_UART_GET_FLAG(huart, UART_FLAG_IDLE);

tmp_it_source = __HAL_UART_GET_IT_SOURCE(huart, UART_IT_IDLE);

/* UART RX Idle interrupt --------------------------------------------*/

if((tmp_flag != RESET) && (tmp_it_source != RESET))

{

__HAL_UART_CLEAR_IDLEFLAG(huart);

HAL_UART_RxIdleCallback(huart);

}

And a default idle callback handler is added as well:

/**

* @brief Rx idle callback.

* @param huart: Pointer to a UART_HandleTypeDef structure that contains

* the configuration information for the specified UART module.

* @retval None

*/

__weak void HAL_UART_RxIdleCallback(UART_HandleTypeDef *huart)

{

/* Prevent unused argument(s) compilation warning */

UNUSED(huart);

/* NOTE: This function should not be modified, when the callback is needed,

the HAL_UART_RxIdleCallback can be implemented in the user file

*/

}

RX DMA is initialized to noncircular mode. WARNING: The CubeMX project was configured for circular RX DMA and the generated source was changed by hand afterward, so they don't match in that regard. Regenerating the project code will overwrite all of the HAL changes!

In my user file usart.c, I add the actual implementation of the RX idle callback:

/**

* @brief UART receive process idle callback for short RX DMA transfers.

* @param huart: Pointer to a UART_HandleTypeDef structure that contains

* the configuration information for the specified UART module.

* @retval None

*/

void HAL_UART_RxIdleCallback(UART_HandleTypeDef* huart)

{

uint16_t rxXferCount = 0;

if(huart->hdmarx != NULL)

{

DMA_HandleTypeDef *hdma = huart->hdmarx;

/* Determine how many items of data have been received */

rxXferCount = huart->RxXferSize - __HAL_DMA_GET_COUNTER(hdma);

HAL_DMA_Abort(huart->hdmarx);

huart->RxXferCount = 0;

/* Check if a transmit process is ongoing or not */

if(huart->State == HAL_UART_STATE_BUSY_TX_RX)

{

huart->State = HAL_UART_STATE_BUSY_TX;

}

else

{

huart->State = HAL_UART_STATE_READY;

}

}

processMessage(huart->pRxBuffPtr, rxXferCount);

}

In my main code, my processMessage() function consumes the received message data, then calls HAL_UART_Receive_DMA() to restart the DMA RX process for the anticipated next message. I have also added a HAL_UART_RxCpltCallback() function that calls processMessage() in the event that a full-count DMA transaction occurs.

Testing:

To test this code, I connected the MCU board to a serial port and used Minicom to send it serial data. The program correctly responds to both short and full-length serial data bursts, calling the appropriate callback routine.

Design considerations:

This implementation implicitly assumes that all RX traffic will be in bursts shorter than a full RX DMA buffer, separated by sufficient time for the callbacks to start a new RX DMA operation with a fresh incoming buffer. I would advise against using the RX Idle mechanism in situations where this condition won't be met. For the intended use case, it seems to work well.

One pattern I find useful here is a ring of RX buffers and a set of volatile head/tail buffer indices, where the interrupt callbacks advance through the ring for each RX message, updating the head index to the newest nonempty buffer, and the foreground code consumes buffers from the tail index. This allows message reception and processing to overlap within limits.

The case where the RX data burst is exactly one DMA buffer long may be troublesome, as both the HAL_UART_RxCpltCallback() and HAL_UART_RxIdleCallback() functions will be called; either the HAL_UART_RxCpltCallback() should disable the idle interrupt, or HAL_UART_RxIdleCallback() should recognize that the RX data has already been consumed. There may be other design flaws I haven't yet been rudely surprised by...

Steve Hersey

________________ Attachments : var_length_DMA_RX.tgz : https://st--c.eu10.content.force.com/sfc/dist/version/download/?oid=00Db0000000YtG6&ids=0680X000006HyVP&d=%2Fa%2F0X0000000bAV%2FLybt06owuptVSs06HffmD9Gh5aqHhuu4z8PAukPYUgM&asPdf=false

[Zt Liu]

Posted on June 02, 2017 at 03:29

I just realized I am not alone facing this similar situation!

The first time I did the fast uart protocol, I made some modification of hal libraries. 

Falling into the following category:

Hack a local copy of the hal_usart module to add the feature (again, for each target MCU I write for), creating maintenance headaches;

However, i gave up soon! As new hal libraires are updated very often, I have to modify them each, this is quite terrible that I didn't realized in the beginning. Later we modify the uart protocol, and add a length bytes in somewhere. As length bytes are received in Interrupt, i start rx dma. But I realized this is not the best way to do it. I do hope there is an idle callback function. Thanks for making this suggestion, Stephen!

[Steve H]

Posted on June 06, 2017 at 15:22

Following up on my experiments with the STM32F1 CubeMX USART driver, I've built and tested essentially the same application as in my original post, now using an F2 MCU and a modified F2 HAL UART driver set. Once I stopped trying to talk to USART3 while my MCU was actually connected to USART6, I was easily able to get the RX idle mechanism to work with two different serial ports at the same time.

The modified driver files are attached for your amusement; I suggest that you diff them against the stock files to see what I've added.

________________ Attachments : stm32f2xx_hal_uart.h.zip : https://st--c.eu10.content.force.com/sfc/dist/version/download/?oid=00Db0000000YtG6&ids=0680X000006Hyct&d=%2Fa%2F0X0000000bAE%2FxfT4iHnb5ey2ICepiwA9jcu.9UVv4XhbRMZ81HAiPzc&asPdf=falsestm32f2xx_hal_uart.c.zip : https://st--c.eu10.content.force.com/sfc/dist/version/download/?oid=00Db0000000YtG6&ids=0680X000006HyYA&d=%2Fa%2F0X0000000bAF%2FlIglYslrehWKFButHqHW9nLcwfCGHYLT9ixENHqtZOw&asPdf=false

[Steve H]

Posted on June 06, 2017 at 15:27

Zhitai,

I share your frustration with having to hack the libraries; I do hope that ST picks this feature up, or else I'll be hacking my libraries for the foreseeable future. As driver changes go, this one seems pretty simple and light-weight, with an obvious benefit.

[andrewlucas9]

Posted on June 14, 2017 at 14:27

I would like to add my support for this feature.  Without it efficient DMA operation with variable length messages is hindered.  We are currently evaluating moving from the standard peripheral library to the HAL and this is one major roadblock.

Thanks,

Andrew Lucas

NCR Canada

[Adrian Adamson]

Posted on August 07, 2017 at 07:30

I also support adding better support for the idle detection.  Variable length reception may be the most common use-case for UARTs there is, so it's surprising to me that this case is not handled more elegantly by microcontroller STK developers in general (including, but not exclusively ST!) 

I'm working on a work-around to do this in a way that's compatible with the CubeMX auto-generated code, using the idle character detection interrupt.

Each USART has its own dedicated ISR vector name (e.g. USART2_IRQHandler) and the stmxxxxxxx_it.c file has a user-defined code area where new user-generated interrupt handlers can be placed.    I'm going to write my own interrupt handler specific to the UART that I care about and put it in that file.  This ISR will stop the DMA transfer, copy the complete UART transmission into an application code buffer, and restart the DMA to watch for the next transmission.   Is there a fundamental flaw in this approach?   (I'll let you guys know how this goes if I don't hear in the meantime)

Two other ways to deal with common UART use-cases that should be supported by CubeMX are:  (call this 2 new feature requests)

1. Packet end detection based on character recognition:  Many applications (maybe most?) have a termination character (such as a newline and/or carriage return) that denotes the end of a transmission.  It would be great if CubeMX and HAL would support this extremely common use-case by providing a call-back when a user-defined character comes through.   I think there may be a way to do this already with the character match interrupt, but the documentation on it is horribly buried in the modbus addressing portion of the user's manual.   If this works, then there are probably 1000 users who would like to use this for general purpose UART, if they only knew it existed, for every 1 modbus user.
2. Continuous circular buffer filling:    One way to deal with identifying the ends of UART packets is to not do it at all in hardware, but instead continuously fill a circular buffer and then have the software figure it out.   I thought that using the CubeMX with the circular buffer enabled and a very long DMA transfer size would effectively do this, but at least in my code, I'm running into hard faults when I set the DMA transfer size larger than the circular buffer size. (negating much of the benefit of using a circular buffer).   It would be great if there were a truly continuous circular buffer option (possibly by setting the transfer size to zero?) or at least a long-transmission option that worked without hard faults.

[Eric Roesinger]

Posted on August 12, 2017 at 01:48

Yet another customer with the same need, here: adding a HAL_UART_RxIdleCallback() implementation should suffice, provided it does not presume to stop and restart DMA:  this strategy should be left to the application.

To avoid a race (exacerbated by scheduling latency) with unsolicited transmission from other devices, my application will likely read SxNDTR in tournament with a count updated by HAL_UART_RxCpltCallback(), and from this determine total bytes received in comparison to bytes already processed, rather than stopping DMA.

Based on recent HAL updates to my own project, support for a receive idle callback might resemble:

void HAL_UART_IRQHandler(UART_HandleTypeDef *huart)

{   uint32_t isrflags   = READ_REG(huart->Instance->SR);   uint32_t cr1its     = READ_REG(huart->Instance->CR1);   uint32_t cr3its     = READ_REG(huart->Instance->CR3);   uint32_t errorflags = 0x00U;   uint32_t dmarequest = 0x00U;   /* If no error occurs */   errorflags = (isrflags & (uint32_t)(USART_SR_PE | USART_SR_FE | USART_SR_ORE | USART_SR_NE));   if(errorflags == RESET)   {     /* UART in mode Receiver -------------------------------------------------*/-    if(((isrflags & USART_SR_RXNE) != RESET) && ((cr1its & USART_CR1_RXNEIE) != RESET))+    if ((isrflags & USART_SR_RXNE) != RESET)     {+      if ((cr1its & USART_CR1_RXNEIE) != RESET)+      {         UART_Receive_IT(huart);         return;+      }++      if ((isrflags & USART_SR_IDLE) != RESET)+      {+        /* May also need to read DR to dismiss the IRQ */+        HAL_UART_RxIdleCallback(huart);+        return;+      }     }

[Roman Priesol]

Posted on August 13, 2017 at 22:18

Thank you guys. This is exactly what I need for GPS reception. Just wondering why it is not a standard feature...