Parsing Raw Data

[AE104]

Hello,

I need to parse a raw data that I receive UART3 interface. Then, I will send to a PC to show results in serial monitor. Before implementing in the microcontroller, I tested my approach in a C compiler. A sample raw data in the buffer array and here is C code to do this job 

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main() {
    char buffer[] = "e\nM0000\nPda7F85F3Fu;ba48D503Dp,10,288\nPda7F9234Bu;ba4E2C324p,10,288\nPda806EC24u;baAE16C6Dp,10,288\nPda807B031u;baB360495p,10,288\n*\n\n";
    char *token;
    token = strtok(buffer, "\n");
    int package_nr = 0;

    while (token != NULL) {
        if (strcmp(token, "e") == 0) {
            printf("Response begin\n");
        } else if (token[0] == 'M') {
            printf("Measuring...\n");
            package_nr = 0;
        } else if (token[0] == 'P') {
            if (package_nr++ == 0) {
                printf("Receiving measurement responses:");
            }
            printf("\n%d. ", package_nr);
            char *P = strchr(token, 'P');
            char *packageLine = P + 1;
            char *running = packageLine;
            char *param;
            while ((param = strtok_r(running, ";,", &running)) != NULL) {
                if (strlen(param) == 0) {
                    continue;
                }
                char paramIdentifier[3];
                char paramValue[10];
                strncpy(paramIdentifier, param, 2);
                paramIdentifier[2] = '\0';
                strncpy(paramValue, param + 2, 9);
                paramValue[9] = '\0';
                printf("%s%s ", paramIdentifier, paramValue);
            }
        } else if (strcmp(token, "*") == 0) {
            printf("\nMeasurement completed.\n");
            printf("%d data point(s) received.\n", package_nr);
        } else {
            printf("%s", token);
        }
        token = strtok(NULL, "\n");
    }

    return 0;
}

Then I got this result:

Response begin
Measuring...
Receiving measurement responses:
1. da7F85F3Fu ba48D503Dp 10 288 
2. da7F9234Bu ba4E2C324p 10 288 
3. da806EC24u baAE16C6Dp 10 288 
4. da807B031u baB360495p 10 288 
Measurement completed.
4 data point(s) received.

To implement this code in the microcontroller, I simply removed the buffer array and replace it with Rx_Buffer which collects all raw data with HAL_UART_RxCpltCallback function. Then, here is the modified code:

 while (1)
  {
    /* USER CODE END WHILE */

    /* USER CODE BEGIN 3 */
 
	  char *token;
          char msg[100];
	      token = strtok(RX_Buffer, "\n");

	      while (token != NULL) {
	          if (strcmp(token, "e") == 0) {
	              sprintf(msg, "Response begin\n");
	              HAL_UART_Transmit(&huart2, (uint8_t*)msg, strlen(msg), HAL_MAX_DELAY);
	          } else if (token[0] == 'M') {
	              sprintf(msg, "Measuring...\n");
	              HAL_UART_Transmit(&huart2, (uint8_t*)msg, strlen(msg), HAL_MAX_DELAY);
	              package_nr = 0;
	          } else if (token[0] == 'P') {
	              if (package_nr++ == 0) {
	                  sprintf(msg, "Receiving measurement responses:");
	                  HAL_UART_Transmit(&huart2, (uint8_t*)msg, strlen(msg), HAL_MAX_DELAY);
	              }
	              sprintf(msg, "\n%d. ", package_nr);
	              HAL_UART_Transmit(&huart2, (uint8_t*)msg, strlen(msg), HAL_MAX_DELAY);
	              char *P = strchr(token, 'P');
	              char *packageLine = P + 1;
	              char *running = packageLine;
	              char *param;
	              while ((param = strtok_r(running, ";,", &running)) != NULL) {
	                  if (strlen(param) == 0) {
	                      continue;
	                  }
	                  char paramIdentifier[3];
	                  char paramValue[10];
	                  strncpy(paramIdentifier, param, 2);
	                  paramIdentifier[2] = '\0';
	                  strncpy(paramValue, param + 2, 9);
	                  paramValue[9] = '\0';
	                  sprintf(msg, "%s%s ", paramIdentifier, paramValue);
	                  HAL_UART_Transmit(&huart2, (uint8_t*)msg, strlen(msg), HAL_MAX_DELAY);
	              }
	          } else if (strcmp(token, "*") == 0) {
	              sprintf(msg, "\nMeasurement completed.\n");
	              HAL_UART_Transmit(&huart2, (uint8_t*)msg, strlen(msg), HAL_MAX_DELAY);
	              sprintf(msg, "%d data point(s) received.\n", package_nr);
	              HAL_UART_Transmit(&huart2, (uint8_t*)msg, strlen(msg), HAL_MAX_DELAY);
	          } else {
	              sprintf(msg, "%s", token);
	              HAL_UART_Transmit(&huart2, (uint8_t*)msg, strlen(msg), HAL_MAX_DELAY);
	          }
	          token = strtok(NULL, "\n");
	      }
	      return 0;


	}

Then I got this result in the serial monitor:

Response begin
              e!0006Receiving measurement responses:
                                                    1.

So what can be source of the problem?
Thank you,

[Pavel A.]

Configure your terminal to handle \n as end of line (automatically add CR).

[AE104]

Thank you @Pavel A. ! I set it CR, Both NL and CR but it didn't work. I also recgonized that after Rx_Counter reaches to 8, the code goes and stuck at Hardfault function.

[Tesla DeLorean]

Code snippet really doesn't convey broader context as to how you accumulate and hand-off the reception buffer.

The transmit calls here will block, consider creating an output ring buffer / FIFO

On the receive do this also with interrupts so you don't miss characters from you multi-character time blocking code.

Instrument HardFault_Handler() to output actionable data.

Likely faults due to exceeding buffers or pointers, or data buffers being out-of-scope.

Perhaps assumption buffers are NUL terminated when not, and using string functions.

[AE104]

Thank you @Tesla DeLorean , here is the callback function to accumulate the raw data:

void HAL_UART_RxCpltCallback(UART_HandleTypeDef *huart)
{

  if (huart->Instance == USART3)  // check if it's USART3 interrupt
  {
    RX_Buffer[RX_Counter++] = RX_Data[0];  // Store received data in buffer
    HAL_UART_Receive_IT(huart, RX_Data, 1); // Ready to receive next byte
  }

}

Can you give me an example for creating an output ring buffer / FIFO?

[Tesla DeLorean]

Providing source code is a bit beyond the scope of my participation here.

You should perhaps bounds check your RX_Counter, and look at how you determine a full/usable condition which you can flag, and move the content to a processing buffer, and restart your buffer acquisition here.

[Karl Yamashita]

The problem with you receiving 1 byte at a time is that you're already checking for a message that isn't complete. For instance, when you receive "P", you don't have the rest of the string yet, so you're going to run into issues not finding ";," characters when you try to use strtok_r. You should separate each string received that has a linefeed into it's own buffer queue to make it easier to parse.

I created and uploaded this project on github. I used most of your parsing routine but I am using a queue buffer for transmitting over UART which is non-blocking.  The only real change is when parsing for "P".  I used a token for each value. It makes it easier to get each value and to combine in a string. You can see how I queue each string received by looking for the "\n" character. I then check in the while loop if there is a new message in the queue and parse it. You can see the image of Docklight printing what your C compiler had output.

https://github.com/karlyamashita/ParseRawData/wiki 

[AE104]

Thank you for creating this GitHub! Where do you use "UART_AddByteToBuffer" function in the project? Could you upload the main.c contents in the same GitHub folder?

[Karl Yamashita]

All the files are there in that repository including main.c. I don't write any code in main.c because ST already generates a lot of code and writing more code just makes it more clustered. So i have PollingInit and PollingRoutine called before/within main while loop, so PollingRoutine.c is really my main.

Just clone the project, import it into STM32CubeIDE. You can find each function that is being called if you place your cursor on the function and press F3 on your keyboard. That will jump you to that function.

[AE104]

Thank you @Karl Yamashita ! I just downloaded the codes and checking now. I have one more question is that you use LPUART, is that critical to use it? I am not sure my microcontroller has it. In addition, I use two UARTs, USART3 is for collecting data, and USART2 is for data transmission. So should I I replace "NotifyUser" functions with "HAL_UART_Transmit(&huart2..."?