Skip to main content
Steve H
Associate III
May 23, 2017
Question

CubeMX Feature Request: Add USART RX IDLE Handling

  • May 23, 2017
  • 24 replies
  • 11441 views
Posted on May 23, 2017 at 16:36

I'm preparing to implement an STM32 project that does high-speed async serial communications, and needs to send and receive variable-length messages. My receive messages are characterized in that they are *always* continuous byte streams on the serial line, with no inter-byte gaps; when the RX serial line goes idle, the message is complete. Because the bit rate is high, I prefer to use DMA for TX/RX message handling. I've written code to do this task before, and I know exactly how to set up the USART, DMA, and interrupt handlers to do the job.

It would be great to use the CubeMX HAL's UART routines for this project, BUT they have an essential lack here: They don't provide handling for the UART_FLAG_IDLE interrupt flag in HAL_UART_IRQHandler(). This means that DMA receive operations using the HAL routines are limited to fixed-length messages, because there is no way to handle the RX idle interrupt that is generated when the last byte of a short message comes in.

I have three equally unappealing options in this case: 1) Continue rolling my own UART support code outside the HAL framework, replacing the hal_usart module for each new target MCU I write for, and creating an integration headache; 2) Hack a local copy of the hal_usart module to add the feature (again, for each target MCU I write for), creating maintenance headaches; or 3) give up on USART DMA receive operations, creating a potentially unacceptable performance bottleneck with all that USART polling or IRQ overhead.

A quick browse of these forums and others like StackOverflow tells me that a number of other developers have faced this same issue, so there's a decent chance this feature will be generally useful.

Here's the specific feature I'd like to see added to the hal_usart drivers:

  • Add code to HAL_UART_IRQHandler() to handle the UART_FLAG_IDLE condition when receiving under DMA, with a callback hook that is invoked when the RX idle condition occurs. Based on the current organization of the hal_usart module, it probably makes best sense to add a separate callback hook, e.g., 'HAL_UART_RxIdleCallback()' or similar.
  • Provide a way for the RX idle callback to determine how much data the RX DMA has received, probably by adding or reusing a member of UART_HandleTypeDef.

With this addition, it should be possible to use the stock HAL routines to manage variable-length RX messages while preserving the efficiency advantages of DMA.

I'm going to try hacking a copy of the hal_usart driver to add this feature, and if it's wildly successful, I'll post it as an update for the curious.

#feature-request
    This topic has been closed for replies.

    24 replies

    ST Technical Moderator
    May 24, 2017
    Posted on May 24, 2017 at 13:21

    Hi

    Hersey.Stephen

    ,

    I want to thank you for your

    contribution and inform you that y

    our suggestion is highlighted internally for further investigation.

    All your feedback are welcome in order to improve our solutions.

    Thanks

    Imen

    In order to give better visibility on the answered topics, please click on 'Best answer' on the reply which solved your issue or answered your question. Thanks
    andrewlucas9
    Associate II
    June 14, 2017
    Posted on June 14, 2017 at 14:27

    I would like to add my support for this feature.  Without it efficient DMA operation with variable length messages is hindered.  We are currently evaluating moving from the standard peripheral library to the HAL and this is one major roadblock.

    Thanks,

    Andrew Lucas

    NCR Canada

    Steve H
    Steve HAuthor
    Associate III
    May 24, 2017
    Posted on May 24, 2017 at 18:21

    Thanks, Imen.

    I have built and tested such a modification of the hal_usart driver for an STM32F100 MCU, on a custom PCB that uses USART3 for serial I/O and has assorted buttons and LEDs. The project was generated under Linux in STM32CubeMX and imported into SW4STM32, where it was compiled and debugged with an STLink V2.

    I've attached a tarball of my example project files. The relevant code changes are listed here with descriptions of where they can be found.

    stm32f1xx_hal_usart.h gets an added function declaration at line 700:

    void HAL_UART_RxIdleCallback(UART_HandleTypeDef *huart);

    stm32f1xx_hal_usart.c gets the following additions:

    HAL_UART_IRQHandler(), line 1247, gets an additional interrupt flag handler:

    tmp_flag = __HAL_UART_GET_FLAG(huart, UART_FLAG_IDLE);

    tmp_it_source = __HAL_UART_GET_IT_SOURCE(huart, UART_IT_IDLE);

    /* UART RX Idle interrupt --------------------------------------------*/

    if((tmp_flag != RESET) && (tmp_it_source != RESET))

    {

    __HAL_UART_CLEAR_IDLEFLAG(huart);

    HAL_UART_RxIdleCallback(huart);

    }

    And a default idle callback handler is added as well:

    /**

    * @brief Rx idle callback.

    * @param huart: Pointer to a UART_HandleTypeDef structure that contains

    * the configuration information for the specified UART module.

    * @retval None

    */

    __weak void HAL_UART_RxIdleCallback(UART_HandleTypeDef *huart)

    {

    /* Prevent unused argument(s) compilation warning */

    UNUSED(huart);

    /* NOTE: This function should not be modified, when the callback is needed,

    the HAL_UART_RxIdleCallback can be implemented in the user file

    */

    }

    RX DMA is initialized to noncircular mode. WARNING: The CubeMX project was configured for circular RX DMA and the generated source was changed by hand afterward, so they don't match in that regard. Regenerating the project code will overwrite all of the HAL changes!

    In my user file usart.c, I add the actual implementation of the RX idle callback:

    /**

    * @brief UART receive process idle callback for short RX DMA transfers.

    * @param huart: Pointer to a UART_HandleTypeDef structure that contains

    * the configuration information for the specified UART module.

    * @retval None

    */

    void HAL_UART_RxIdleCallback(UART_HandleTypeDef* huart)

    {

    uint16_t rxXferCount = 0;

    if(huart->hdmarx != NULL)

    {

    DMA_HandleTypeDef *hdma = huart->hdmarx;

    /* Determine how many items of data have been received */

    rxXferCount = huart->RxXferSize - __HAL_DMA_GET_COUNTER(hdma);

    HAL_DMA_Abort(huart->hdmarx);

    huart->RxXferCount = 0;

    /* Check if a transmit process is ongoing or not */

    if(huart->State == HAL_UART_STATE_BUSY_TX_RX)

    {

    huart->State = HAL_UART_STATE_BUSY_TX;

    }

    else

    {

    huart->State = HAL_UART_STATE_READY;

    }

    }

    processMessage(huart->pRxBuffPtr, rxXferCount);

    }

    In my main code, my processMessage() function consumes the received message data, then calls HAL_UART_Receive_DMA() to restart the DMA RX process for the anticipated next message. I have also added a HAL_UART_RxCpltCallback() function that calls processMessage() in the event that a full-count DMA transaction occurs.

    Testing:

    To test this code, I connected the MCU board to a serial port and used Minicom to send it serial data. The program correctly responds to both short and full-length serial data bursts, calling the appropriate callback routine.

    Design considerations:

    This implementation implicitly assumes that all RX traffic will be in bursts shorter than a full RX DMA buffer, separated by sufficient time for the callbacks to start a new RX DMA operation with a fresh incoming buffer. I would advise against using the RX Idle mechanism in situations where this condition won't be met. For the intended use case, it seems to work well.

    One pattern I find useful here is a ring of RX buffers and a set of volatile head/tail buffer indices, where the interrupt callbacks advance through the ring for each RX message, updating the head index to the newest nonempty buffer, and the foreground code consumes buffers from the tail index. This allows message reception and processing to overlap within limits.

    The case where the RX data burst is exactly one DMA buffer long may be troublesome, as both the HAL_UART_RxCpltCallback() and HAL_UART_RxIdleCallback() functions will be called; either the HAL_UART_RxCpltCallback() should disable the idle interrupt, or HAL_UART_RxIdleCallback() should recognize that the RX data has already been consumed. There may be other design flaws I haven't yet been rudely surprised by...

    Steve Hersey

    ________________

    Attachments :

    var_length_DMA_RX.tgz : https://st--c.eu10.content.force.com/sfc/dist/version/download/?oid=00Db0000000YtG6&ids=0680X000006HyVP&d=%2Fa%2F0X0000000bAV%2FLybt06owuptVSs06HffmD9Gh5aqHhuu4z8PAukPYUgM&asPdf=false
    Zt Liu
    Senior III
    June 2, 2017
    Posted on June 02, 2017 at 03:29

    I just realized I am not alone facing this similar situation!

    The first time I did the fast uart protocol, I made some modification of hal libraries. 

    Falling into the following category:

    Hack a local copy of the hal_usart module to add the feature (again, for each target MCU I write for), creating maintenance headaches;

    However, i gave up soon! As new hal libraires are updated very often, I have to modify them each, this is quite terrible that I didn't realized in the beginning. Later we modify the uart protocol, and add a length bytes in somewhere. As length bytes are received in Interrupt, i start rx dma. But I realized this is not the best way to do it. I do hope there is an idle callback function. Thanks for making this suggestion, Stephen!

    Steve H
    Steve HAuthor
    Associate III
    June 6, 2017
    Posted on June 06, 2017 at 15:27

    Zhitai,

    I share your frustration with having to hack the libraries; I do hope that ST picks this feature up, or else I'll be hacking my libraries for the foreseeable future. As driver changes go, this one seems pretty simple and light-weight, with an obvious benefit.

    Steve H
    Steve HAuthor
    Associate III
    June 6, 2017
    Posted on June 06, 2017 at 15:22

    Following up on my experiments with the STM32F1 CubeMX USART driver, I've built and tested essentially the same application as in my original post, now using an F2 MCU and a modified F2 HAL UART driver set. Once I stopped trying to talk to USART3 while my MCU was actually connected to USART6, I was easily able to get the RX idle mechanism to work with two different serial ports at the same time.

    The modified driver files are attached for your amusement; I suggest that you diff them against the stock files to see what I've added.

    ________________

    Attachments :

    stm32f2xx_hal_uart.h.zip : https://st--c.eu10.content.force.com/sfc/dist/version/download/?oid=00Db0000000YtG6&ids=0680X000006Hyct&d=%2Fa%2F0X0000000bAE%2FxfT4iHnb5ey2ICepiwA9jcu.9UVv4XhbRMZ81HAiPzc&asPdf=false

    stm32f2xx_hal_uart.c.zip : https://st--c.eu10.content.force.com/sfc/dist/version/download/?oid=00Db0000000YtG6&ids=0680X000006HyYA&d=%2Fa%2F0X0000000bAF%2FlIglYslrehWKFButHqHW9nLcwfCGHYLT9ixENHqtZOw&asPdf=false
    Adrian Adamson
    Senior
    August 7, 2017
    Posted on August 07, 2017 at 07:30

    I also support adding better support for the idle detection.  Variable length reception may be the most common use-case for UARTs there is, so it's surprising to me that this case is not handled more elegantly by microcontroller STK developers in general (including, but not exclusively ST!) 

    I'm working on a work-around to do this in a way that's compatible with the CubeMX auto-generated code, using the idle character detection interrupt.

    Each USART has its own dedicated ISR vector name (e.g. USART2_IRQHandler) and the stmxxxxxxx_it.c file has a user-defined code area where new user-generated interrupt handlers can be placed.    I'm going to write my own interrupt handler specific to the UART that I care about and put it in that file.  This ISR will stop the DMA transfer, copy the complete UART transmission into an application code buffer, and restart the DMA to watch for the next transmission.   Is there a fundamental flaw in this approach?   (I'll let you guys know how this goes if I don't hear in the meantime)

    Two other ways to deal with common UART use-cases that should be supported by CubeMX are:  (call this 2 new feature requests)

    1. Packet end detection based on character recognition:  Many applications (maybe most?) have a termination character (such as a newline and/or carriage return) that denotes the end of a transmission.  It would be great if CubeMX and HAL would support this extremely common use-case by providing a call-back when a user-defined character comes through.   I think there may be a way to do this already with the character match interrupt, but the documentation on it is horribly buried in the modbus addressing portion of the user's manual.   If this works, then there are probably 1000 users who would like to use this for general purpose UART, if they only knew it existed, for every 1 modbus user.
    2. Continuous circular buffer filling:    One way to deal with identifying the ends of UART packets is to not do it at all in hardware, but instead continuously fill a circular buffer and then have the software figure it out.   I thought that using the CubeMX with the circular buffer enabled and a very long DMA transfer size would effectively do this, but at least in my code, I'm running into hard faults when I set the DMA transfer size larger than the circular buffer size. (negating much of the benefit of using a circular buffer).   It would be great if there were a truly continuous circular buffer option (possibly by setting the transfer size to zero?) or at least a long-transmission option that worked without hard faults.
    Adrian Adamson
    Senior
    October 21, 2017
    Posted on October 21, 2017 at 21:18

    I came back to this part of the project after working on some other parts, and Ifound out that #2, continuous circular buffer reception, already works,but with one requirement on the user code that's not obvious until you read the reference manual: You need to make the size of your circular buffer equal to the transfer size. If you do, then the chip will automatically wrap new data back to the start of the buffer, and go on continuously.

    I find that using the circular buffer with idle detection interrupts works great for my application, and I don't need to modify any HAL library code outside of the user-modifiable sections. The overview of the strategy is as follows:

    1.Use HAL CubeMX to set up a DMA transfer into a circular buffer on your UART of choice

    2. Define a circular buffer global variable and some variables for the head, tail, and length of the transmission

    3. Enable the interrupt for idle mode detection and enable reception

    4. In the stm32l0xx_it.c file there is an ISR that is specific to each UART, with a section for user code. Put code to ID and clear the idle mode detection interrupt here, and parse the UART transmission or set a flag to parse it somewhere else.

    In more detail, with example code:

    2:

    uint8_t cyp_cmd_buff[CYP_BUFF_LENGTH];
    uint16_t cyp_cmd_head;
    uint16_t cyp_cmd_tail;
    uint8_t fresh_cyp_cmd = 0;�?�?�?�?�?�?�?�?�?�?�?�?
    uint8_t cyp_cmd_buff[CYP_BUFF_LENGTH];
    uint16_t cyp_cmd_head;
    uint16_t cyp_cmd_tail;
    uint8_t fresh_cyp_cmd = 0;�?�?�?�?�?�?�?�?�?�?�?�?

    3: In main:

    /* USER CODE BEGIN 2 */
     __HAL_UART_ENABLE_IT(&huart1,UART_IT_IDLE);
     status = HAL_UART_Receive_DMA(&huart1, &cyp_cmd_buff, CYP_BUFF_LENGTH);�?�?�?

    4:

    /**
    * @brief This function handles USART1 global interrupt / USART1 wake-up interrupt through EXTI line 
    */
    void USART1_IRQHandler(void)
    {
     /* USER CODE BEGIN USART1_IRQn 0 */
     int16_t length = 0;
    // Receive time-out interrupt
     if (__HAL_UART_GET_FLAG(&huart1, UART_FLAG_IDLE) == SET) {
     __HAL_UART_CLEAR_IDLEFLAG(&huart1);
    cyp_cmd_tail = cyp_cmd_head;
     cyp_cmd_head = huart1.RxXferSize -(uint16_t)huart1.hdmarx->Instance->CNDTR;
    if (cyp_cmd_tail || cyp_cmd_head) { //Ignore case where the interrupt fires before any data is received, when both are zero
     HAL_GPIO_WritePin(GPIOB, DEV_GRN_Pin, 1);
     timers[LED_OFF].timeval = 50 + HAL_GetTick();
     if(cyp_cmd_tail >= cyp_cmd_head) { //Wrapped around circular buffer
     length = (cyp_cmd_head + CYP_BUFF_LENGTH) - cyp_cmd_tail;
     } else {
     length = cyp_cmd_head - cyp_cmd_tail;
     }
     fresh_cyp_cmd = 1; //Flag to let command parser know that data is available.
     }
     }
    /* USER CODE END USART1_IRQn 0 */
     HAL_UART_IRQHandler(&huart1);
     /* USER CODE BEGIN USART1_IRQn 1 */
    /* USER CODE END USART1_IRQn 1 */�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?

    Eric Roesinger
    Visitor II
    August 11, 2017
    Posted on August 12, 2017 at 01:48

    Yet another customer with the same need, here: adding a HAL_UART_RxIdleCallback() implementation should suffice, provided it does not presume to stop and restart DMA:  this strategy should be left to the application.

    To avoid a race (exacerbated by scheduling latency) with unsolicited transmission from other devices, my application will likely read SxNDTR in tournament with a count updated by HAL_UART_RxCpltCallback(), and from this determine total bytes received in comparison to bytes already processed, rather than stopping DMA.

    Based on recent HAL updates to my own project, support for a receive idle callback might resemble:

    void HAL_UART_IRQHandler(UART_HandleTypeDef *huart)

    {

       uint32_t isrflags   = READ_REG(huart->Instance->SR);

       uint32_t cr1its     = READ_REG(huart->Instance->CR1);

       uint32_t cr3its     = READ_REG(huart->Instance->CR3);

       uint32_t errorflags = 0x00U;

       uint32_t dmarequest = 0x00U;

       /* If no error occurs */

       errorflags = (isrflags & (uint32_t)(USART_SR_PE | USART_SR_FE | USART_SR_ORE | USART_SR_NE));

       if(errorflags == RESET)

       {

         /* UART in mode Receiver -------------------------------------------------*/

    -    if(((isrflags & USART_SR_RXNE) != RESET) && ((cr1its & USART_CR1_RXNEIE) != RESET))

    +    if ((isrflags & USART_SR_RXNE) != RESET)

         {

    +      if ((cr1its & USART_CR1_RXNEIE) != RESET)

    +      {

             UART_Receive_IT(huart);

             return;

    +      }

    +

    +      if ((isrflags & USART_SR_IDLE) != RESET)

    +      {

    +        /* May also need to read DR to dismiss the IRQ */

    +        HAL_UART_RxIdleCallback(huart);

    +        return;

    +      }

         }
    Roman Priesol
    Associate II
    August 13, 2017
    Posted on August 13, 2017 at 22:18

    Thank you guys. This is exactly what I need for GPS reception. Just wondering why it is not a standard feature...

    MDS
    Associate III
    August 17, 2017
    Posted on August 17, 2017 at 18:29

    I would like to see the RTO feature also supported. My design uses the UART Receive Timeout (RTO) interrupt for variable length receive messages. I like RTO because you can specify the idle time in characters. I would eventually like to get RTO working with Double Buffer (DB) DMA mode to provide continuous receive, so I can do slow receive like from a keyboard or fast for receiving a byte stream. I have DB working for SPIs, so UART should be very similar.

    John Craven
    Senior
    September 1, 2017
    Posted on September 01, 2017 at 15:47

    I like RTO before idle. It's far more flexible than a fixed idle timing.

    Andrei Chichak
    Lead
    August 17, 2017
    Posted on August 17, 2017 at 20:06

    I support this request. I've written this same damn code as well. 

    Andrei Chichak

    CBF Systems Inc

    The Embedded.fm podcast and blog.

    Bob jenkins
    Associate II
    September 13, 2017
    Posted on September 13, 2017 at 16:51

    in your HAL_UART_RxIdleCallback() function, when you 'zero' huart->RxXferCount, you [probably] should also reset the uart read state to HAL_UART_STATE_READY:

    huart->RxState = HAL_UART_STATE_READY;

    Otherwise I was having a problem with the next call to HAL_UART_Receive_DMA, returning HAL_BUSY  and being unable to receive next chunks of data from USART.

    Ricky Black
    Visitor II
    April 18, 2018
    Posted on April 19, 2018 at 00:14

    Hi! Thanks for your solution. I'm starting with programming STM32. I have some problems, maybe you can solve it.

    I've used your solution and implemented it on STM32F103RBT6. It works, but I can't read rxBuffer. I'm using module GPS SIM808. I need to find a special sequence in rxBuffer received from module, like three signs '@' 'C' '1' because I want to control my equipment by SMS message. When I send sms to module, it transmit on UART sequence with date, phone number etc. and message. Message is a key, which help me to control my equipment (STM32+SIM808).