IAP over GSM: UART and Flash programming

[gmate1]

Posted on January 07, 2016 at 16:29

Hi,

I'm developing an IAP application for firmware upgrade over GSM network. I've a SIM800H module connected to STM32 over UART interface and I'm able to establish GSM connection, setup HTTP GET request and on wire (I'm listening on SIM800-STM32 UART tx/rx lines) I see that HTTP GET works fine (I see that the data matches the firmware hosted on server). I would like to write this hex format firmware image into main flash memory, starting from an area reserved for application firmware and to do so, In my IAP driver code, I unlock the flash and start programming the flash memory by using APIs from STM32 library, however there is a mismatch between data (firmware hex file) received over UART and what's actually written in the memory. Please advice. More information below:

Snip from UART IRQ handler: file1:: uart.c
-----------------------------------------
volatile 
int
start_capture = 0;
void
process_reception(uint16_t data)
{
static
volatile 
int
rx_index = 0;
static
volatile uint8_t rx_buffer[4];
volatile uint32_t val = 0;
static
volatile uint32_t flashaddress = 0x08004000;
static
volatile 
int
seen = 0;
/* Special case for firmware hex file download */
if
(start_capture) 
/* this flag is set after successful response for HTTPREAD execute command */
{
/* Though we are told to start capturing the firmware hex file,
* we should wait until a particular character arrives
*/
if
(data == 0x3A) 
/* hex format firmware starts with this character */
seen = 1;
if
(seen)
{
if
(rx_index == 3)
{
/* ***: Since the programming is done either a word basis or
* half word basis by the STM API, Is it correct way of
* using the API ? */
val = (rx_buffer[0] << 24) + (rx_buffer[1] << 16) + (rx_buffer[2] << 8) + (rx_buffer[3]);
if
((FLASH_ProgramWord(flashaddress, val) == FLASH_COMPLETE))
{
flashaddress += 4;
}
rx_index = 0;
}
else
{
rx_buffer[rx_index++] = data;
}
}
else
{
; 
/* do nothing, wait for the special character */
}
} 
/* end of start capture */
/* some more code here, not relevant here */
}
void
USART1_IRQHandler(
void
)
{
/*
* some code for TX, not relevant here
*/
if
(USART_GetITStatus(peripheral->USARTx, USART_IT_RXNE) != RESET)
{
process_reception((
char
) USART_ReceiveData(USART1);
}
}

void
uart_init(
void
)
{
USART_InitTypeDef USART_InitStructure;
/* Restore UART to it's default state */
USART_DeInit(USART1);
/* configured as follow:
- BaudRate = 9600 baud
- Word Length = 8 Bits
- One Stop Bit
- No parity
- Hardware flow control disabled (RTS and CTS signals)
- Receive and transmit enabled
*/
USART_InitStructure.USART_BaudRate = 9600;
USART_InitStructure.USART_WordLength = USART_WordLength_8b;
USART_InitStructure.USART_StopBits = USART_StopBits_1;
USART_InitStructure.USART_Parity = USART_Parity_No;
USART_InitStructure.USART_HardwareFlowControl = USART_HardwareFlowControl_None;
USART_InitStructure.USART_Mode = USART_Mode_Rx | USART_Mode_Tx;
USART_Init(USART1, &USART_InitStructure);
USART_ITConfig(USART1, USART_IT_RXNE, ENABLE);
USART_Cmd(USART1, ENABLE);
}
Initiating the firmare capture from a different routine (file2:: app.c)
-----------------------------------------------------------------------------------
extern
volatile 
int
start_capture;
void
foo(
void
)
{
/*
* checking the response after sending AT+HTTPREAD execute command
*/
if
(strncmp(rsp, 
''+HTTPREAD:''
, 10) == 0)
{
start_capture = 1;
}
}

snip from AT+Command logs ------------------------------------ +HTTPACTION: 0,200,4437 AT+HTTPREAD +HTTPREAD: 4437 :020000040800F2 :1040000070060020D5400008F5410008D1410008A5 :1040100000000000000000000000000000000000A0 /*goes on */ ............... OK AT+HTTPTERM OK #stm32 #stm32 #usart #iap #iap #flash #flash #stm32f0 #solved

[Tesla DeLorean]

Posted on January 07, 2016 at 19:09

however there is a mismatch between data (firmware hex file) received over UART and what's actually written in the memory.

Can you be specific? Not the same how? Holes, shifted data?You don't seem to differentiate different types of HEX records.The write time is long, are you sure you aren't losing data on the serial link without flow control of some fashion.Is the code running from RAM so as to deal with the stalling if run from FLASH?

[pkumar1883]

Posted on January 08, 2016 at 09:24

Hi sidekick,

You can debug your process in some simple steps-- 1. Receive file(byte by byte) from GSM and dump this data to some other USART. Connect this USART to your PC.Capture this dump into a file (using Realterm or some other utility).Ccompare this file with the original firmware binary file. 2. check if flash in not write protected. 3. verify flash word after writing as shown below- if ((FLASH_ProgramWord(flashaddress, val) == FLASH_COMPLETE)){ if(*(u32*)flashaddress != *(u32*))val) { //verification falied } flashaddress+=4; }

[gmate1]

Posted on January 11, 2016 at 17:39

Thank you all for your help. I did few changes, most notable are:

1: Instead of collecting the entire firmware image in one shot, I now use AT+ command to download only a portion of fimrware image over GSM network and do NOT program flash in the UART ISR. This way, I'm more sure that I do not drop any packets while I try to access the flash memory during the write operation. AT+ command supports the provision of providing offset and length with the HTTPREAD execution command, so basically, I start with offset 0 and keep on incrementing the offset by another 45 bytes until the offset reaches the total firmware length(in my case, it is 4437). The reason for selecting 45 bytes is that at the moment I see that the hex file generated using KEIL uVision5, has a format something like this (see below)

:020000040800F2DA
:1040000070060020D5400008F5410008D1410008A5DA
:1040100000000000000000000000000000000000A0DA
<----------------------45--------------------> ('
'(0xA) followed by '
'(0xD) as shown above)

each line starts with ':' which is equal to 0x3A and ends with ' ' followed by ' '. I could try a bigger buffer than LINEMAX(45) here, however, i think looking for consecutive occurance of 0xD 0xA seemed like a good idea to parse the hex firmware image. I see that with the following code in my USART ISR routine, I'm able to successfully parse the hex firmware image in 45 bytes chunks. 2: In the UART ISR, I now use a local buffer, in which I store the data received over UART and If i receive a special delimeter (' ' followed by ' '), It means the end of one line (which is in total 45 bytes), I copy the contentes of this buffer into a global buffer and set a flag to notify the consumer that flash memory can now be programmed. Consumer writes into the flash memory, resets the flag and then once again invoke AT+ command to read the next 45 bytes. This goes on until the offset reaches the total length For the time being, let's assume that GSM network is very stable, no security issues, and we are living in a perfect world. However, there are still problems with proper parsing the image, and also writing this portion of image into flash memory. The problems are Problem 1: stm32 library has APIs(FLASH_If_Write(), as part of sample IAP code over UART) for writing to memory, and I use those API to write to flash memory location, once the receive buffer gets full ( receive buffer is 45 bytes in length, that includes ' '' '), however, this simple memory write test fails (note that in the memory test code, ive set the buffer size to be 48 which i thought should be okay as it can be evenly divided into word size), so there is a fundamental memory write problem in my code. snip: memory write test: ----------------------------

#define LINEMAX 45
#define APPLICATION1_ADDRESS (uint32_t)0x08004000
uint32_t FLASH_If_Write(__IO uint32_t* FlashAddress, uint32_t* Data ,uint16_t DataLength)
{
uint32_t i = 0;
for
(i = 0; (i < DataLength) && (*FlashAddress <= (USER_FLASH_END_ADDRESS-4)); i++)
{
/* the operation will be done by word */
if
(FLASH_ProgramWord(*FlashAddress, *(uint32_t*)(Data+i)) == FLASH_COMPLETE)
{
/* Check the written value */
if
(*(uint32_t*)*FlashAddress != *(uint32_t*)(Data+i))
{
/* Flash content doesn't match SRAM content */
return
(2);
}
/* Increment FLASH destination address */
*FlashAddress += 4;
}
else
{
/* Error occurred while writing data in Flash memory */
return
(1);
}
}
return
(0);
}
void
FLASH_If_Init(
void
)
{
/* Unlock the Program memory */
FLASH_Unlock();
/* Clear all FLASH flags */
FLASH_ClearFlag(FLASH_FLAG_EOP|FLASH_FLAG_WRPERR | FLASH_FLAG_PGERR | FLASH_FLAG_BSY);
}
void
simple_mem_test()
{
uint32_t flashaddress = APPLICATION1_ADDRESS;
int
ret;
char
fw[48] = 
'':104600000000000001020304060708090000000082

''
;
FLASH_If_Init(); 
/* unlocks flash */
ret = FLASH_If_Write(&flashaddress, (uint32_t *)fw, (uint16_t)48/4);
if
(ret == 0) {
SerialPutString(
''

MEM TEST PASSED

''
);
} 
else
{
sprintf
(dbg, 
''

------------FAILED with %d

''
, ret);
SerialPutString(dbg);
}
}
int
main(
void
)
{
/* initialize relevant clocks, peripherals, GPIOs, IRQs */
simple_mem_test();
while
(1)
{
/* do something */
}
}

The above test fails with error code 1, which means, something went wrong while writing. Problem 2: Though the firmware hex image starts with as mentioned above, later towards the end, the size of each line is no more uniform 45 bytes that i saw earlier(see below), so my parsing routine gets messed up, how should I reliably capture the entire firmware hex image. Parsing related code: ------------------------------- file: uart.c -------------------------------

extern
volatile 
char
message[];
void process_reception(uint16_t data, 
int
flag)
{
static
volatile 
int
rx_index = 0;
static
volatile uint8_t rx_buffer[LINEMAX] = {0};
static
volatile uint16_t prev_data = 
'
'
;
/* Special case for firmware hex file download */
if
(flag)
{
/* each line ends with newline character */
if
(rx_index == LINEMAX -1) 
/* Is this an end-of-line condition? */
{
memcpy((
void
*)message, (
void
*)rx_buffer, rx_index);
rx_index = 0; 
/* reset index */
line_valid = 1; 
/* inform consumer about the valid data in buffer */
}
else
{
/* each line of hex file starts with this special character */
if
(data == 
'
'
&& prev_data == 
'
'
) {
rx_buffer[rx_index];
rx_index = 0;
prev_data = 0;
}
prev_data = data;
/* start filling up the buffer */
rx_buffer[rx_index++] = data;
}
return
;
}
/* other part of code, not relevant here, during normal operation */
}
void
UART_IRQ_process(
int
flag)
{
/*
* some code for TX, not relevant here
*/
/* transmit data register not empty */
if
(USART_GetITStatus(USART1, USART_IT_RXNE) != RESET)
{
process_reception(USART_ReceiveData(USARTx), flag);
}
}
void
USART1_IRQHandler(
void
)
{
UART_IRQ_process(start_capture);
}

firmware image (hex file) towards the end:

:1045F0009405000036440008006CDC020000000056DA
:104600000000000001020304060708090000000082DA
:0446100000000000A6DA
:04000005080040C1EEDA
:00000001FFDA

snip: AT+ command for HTTPREAD and flash programming ------------------------------------------- file: test.c -------------------------------------------

extern
__IO 
int
start_capture; 
/* initiate firmware capturing in buffer */
extern
__IO 
int
line_valid; 
/* we have a valid data in buffer */
volatile
char
message[LINEMAX];
int
test_gsm(
int
len)
{
int
ret = SUCCESS;
volatile
uint32_t flashaddress = APPLICATION1_ADDRESS;
__IO 
int
offset = 0;
while
(offset < FW_IMAGE_LEN)
{
ret = gsm_read(offset, LINEMAX);
if
(ret != SUCCESS)
break
;
offset += LINEMAX;
if
(line_valid)
{
if
(FLASH_If_Write(&flashaddress, (uint32_t *)message, LINEMAX) == 0) {
} 
else
{
SerialPutString(
''

Error writing image

''
);
return
ERROR;
}
line_valid = 0;
}
}
return
ret;
}

Overall, below is a typical GSM session logs:

AT+HTTPREAD=15,45 <-- command that i send (starting from offset 15 here)
+HTTPREAD: 45 <-- response from GSM
:1040000070060020D5400008F5410008D1410008A5 <-- Response from GSM
OK <-- response from GSM
AT+HTTPREAD=60,45 <-- Next command that i send and so on
+HTTPREAD: 45
:1040100000000000000000000000000000000000A0
OK
AT+HTTPREAD=105,45
+HTTPREAD: 45

[gmate1]

Posted on January 11, 2016 at 17:47

''1. Receive file(byte by byte) from GSM and dump this data to some other USART. Connect this USART to your PC.Capture this dump into a file (using Realterm or some other utility).Ccompare this file with the original firmware binary file.''

I'm already listening on the USART line between STM and SIM800H (GSM module) and dumping the contents on serial console and it is indeed matching the firmware image.2. check if flash in not write protected. Flash is not write protected.

[Tesla DeLorean]

Posted on January 11, 2016 at 18:02

Ok, the Intel .HEX is a very simple format, that has variable record sizes, people were decoding these files in the 80's using assembler. It doesn't have the inherent complexity of object file formats like OMF or ELF.

You are writing the raw .HEX file lines directly into memory, seems a bit odd, but at that point why do the line breaks matter, just read in 512 or 1K, or whatever and write that big block into memory, and go fetch the next, until you get the final terminal block which might be some amount less than that. At this point you're just staging the data, so the format is not consequential. The HTTP format inherently provides a total byte count for the payload data. Use flow control if you can't handle the rate. In validation here we CRC the data on the server side as it goes out, and generate the same thing for the data written to flash, and manually confirm both sides of the comms link is in fact functional, and where corruption is occurring in the chain, if that happens.Flash writes will fail if the memory is not erased, 32-bit writes will fail if not aligned.When we process .HEX files here, we pull in a line delimited by CR/LF, and decode the line. Depending on the generation tool you typically see up to 16 or 32 bytes described per line, 80 characters should suffice for each, but the code filling your line buffer should be aware of the maximal length supported. The encoding of the line also describes the type of record, and the size of the payload.

[gmate1]

Posted on January 11, 2016 at 18:31

Thanks Clive,

Flash writes will fail if the memory is not erased, 32-bit writes will fail if not aligned. - Ohh, completely forgot about the erase part. I'll try with bigger buffer(say 512 bytes). The problem that I had was the problem of parsing the actual data, since sometimes I saw it getting intermixed with HTTP command response, but I see your point, I'll try.You are writing the raw .HEX file lines directly into memory, seems a bit odd. - So, do you mean, I should use the .bin file instead ?

[Tesla DeLorean]

Posted on January 11, 2016 at 19:46

Well the

https://en.wikipedia.org/wiki/Intel_HEX

is a method of transporting binary data using 7-bit ASCII representation, common of the late 70's early 80's when the serial/modem links, etc were not 8-bit, and where systems enroute might have different characters sets (

https://en.wikipedia.org/wiki/EBCDIC

)The .HEX form is not what gets written into the Flash for your application to run, you need to parse and extract the binary data from the records, and write that. The .HEX record reflects an address, length and type of message, and one takes the address, and writes the payload data to that address.When pulling binary data via HTTP I tend to package it with some size and checksum validation, so I can check the integrity of it before jumping to it. There's also the opportunity to compress and encrypt them. But HTTP is certainly capable of pulling 8-bit .BIN filesThe .HEX is going to typically be about 2.5x the size of an equivalent .BIN, but it does have a basic line level integrity check.

[gmate1]

Posted on January 13, 2016 at 17:34

Thank Clive1 for so many pointers.

Instead of trying with hex file, I went back to .bin file and was able to successfully update the application firmware over GSM network. However this exercise is far from over as there are many more things to consider and implement. For understanding purposes (i.e. HTTP firmware download, programming the flash memory and then jumping to the address etc) and just to make it work, I used a bigger buffer to capture the entire firmware (I tested with a small firmware that just blinks the LED on my custom board, which has STM32F072C8 as the main SOC) and did some parsing to figure out the correct start of firmware data in the receive buffer and then wrote the entire firmware image, aligned on word boundary and jumped to the new firmware and it worked :) However, for practical purposes, I break the HTTP data (since the firmware binary file is around 24kB) into chunks (A minimalist flow control) and try to segregate the firmware data from HTTP responses(OK, +HTTPDATA: <data_len>, carriage return(0xD), line feed(0xA) etc. However as of now, I haven't found a reliable way to parse the receive data. A sample UART data stream duing the GSM session is as below: AT+HTTPREAD=0,512CRLF+HTTPREAD:SPACE512CRLF<ACTUAL FIRMWARE DATA>CRLFOKCRLFAT+HTTPREAD=512,512CRLF+HTTPREAD:SPACE512CRLF<ACTUAL FIRMWARE DATA>CRLFOKCRLFAT+HTTPREAD=1024,16CRLF+HTTPREAD:SPACE16CRLF<ACTUAL FIRMWARE DATA>CRLFOKCRLF My UART ISR is something like this: snip from UART ISR:

if
(flag) 
/* this flag set before sending the AT+HTTPREAD command */
{
/* copy the local buffer into a consumer buffer.
* we reached the end of GSM response
*/
if
(rx_index == GSM_HTTP_DATA_SIZE) {
memcpy(message, (
void
*)rx_buffer, GSM_HTTP_DATA_SIZE);
line_valid = 1; 
/* Inform the consumer that the data is now available */
rx_index = 0;
}
if
(data_prev == 0xD && data == 0xA) {
;
} 
else
if
(data_prev == 
'O'
&& data == 
'K'
) {
;
} 
else
{
rx_buffer[rx_index++] = data ;
}
data_prev = data;
/* ***: exiting here at the moment */
return
;
} 
/* end of start_capture */

1: As you can see here, this will not work, since i'm only avoiding the LINE FEED, CARRIAGE RETURN (if these two were seen as consecutive characters) and 'OK' message, but the other responses (+HTTPREAD, payload size) also gets saved in the buffer. NOTE: 1: AT+HTTPREAD=0,512 is the AT+ command that I send to read a payload of 512 characters. 2: CR = Carriage return (ASCII code: 0xD), LF = Line feed (Ascii code: 0xA) SPACE = space (Ascii code: 0x20)

[Tesla DeLorean]

Posted on January 13, 2016 at 18:13

Well you're going to have to have to manage the stream. It sends you a <CR><LF> delimited message telling you how many byte follow. You should read those, and then switch back to line parsing.