Field updating of STM32 with data in external qspi flash

[imtiaz]

Posted on July 05, 2017 at 02:51

Hi All,

We are using an STM32F7 processor.

We have a typical application that will store LCD images and other asset type data in qspi flash. The QSPI flash will be memory mapped. We also have external memory mapped RAM.

In the code the images will be specified to go into the external qspi flash section , I guess using a scatter file definition of memory regions and &sharppragma in the code.

Could anyone give me some direction as to how we could update this software in the field - where we are not only updating the internal 2Mb flash , but also the external qspi flash. Currently I am using the ST system bootloader to update the internal flash.

I am assuming that a customised bootloader will be needed based on the address of the data , internal flash starting at 0x08000000 and external 0x90000000   it will have two different API's to write to either internal or external memory.

Does that mean that I will need to work with a hex file rather than a binary?

The setup of our hardware is very similar to ST's F769 eval and discovery boards. However these are programmed using dedicated programming interfaces like swd/jtag and stlink etc which have customised loaders.

In our case we will be doing OTA updates.

Highly appreciate pointer in the right direction.

Best Regards

Imtiaz 

#qspi-flash-memory-mapped-bootload-stm32f7

[Jeroen3]

Posted on July 05, 2017 at 07:52

Yes, you will have to run your own OTA upgrade software. I do not know if there is anything available.

What exactly is unclear?

If you have external memory, you can:

Download the full application image to it.

Verify the image.

Reboot in upgrade mode, copy from external to internal flash.

Reboot in download mode.

Download updated external flash data.

Reboot to normal application.

[imtiaz]

Posted on July 05, 2017 at 23:55

Hi All,

This is my current plan , in theory :

1. From the hex file created - create two binaries , one for internal flash and the other for external flash based on the address in the hex record
2. To upgrade the internal flash area use the system boot-loader as normal (uart / SPI tested , DFU)
3. On application start up - a special command will be sent to update the external flash if needed. Application will wait for 100 msec before continuing.
4. If required QSPI flash will be erased , written and verifed and then 
5. Application start up continues and memory maps the qspi flash
6. ... and Bobs your uncle

Would appreciate any holes in this theory before I start implementing.

The advantage of this method that I can see is that there is no custom bootloader , chips can be programmed in production without any programmer. Internal flash and external flash do not always have to be updated if there is no change in one of them.

Disadvantage - if the internal and external flash code get out of sync there will be a hardfault.

Regards

Imtiaz

[Jeroen3]

Posted on July 06, 2017 at 07:22

That is not really OTA (Over The Air). But it should work.

What you can do instead of the 100ms wait and the risk on a hardfault is using CRC on the external memory.

And use something like

char *version = '__DATE__' '__TIME__';

to create a string you can compare on boot to see if both memories match the same firmware build.

http://srecord.sourceforge.net/

  should be able to split your hex file to binaries, and compute and store STM32 hardware crc compatible checksum.

Are you able to program both memories using jtag/swd in one go already?

[Jack Peacock]

Posted on July 06, 2017 at 11:33

Where an over-the-air update gets complicated is in recovering from an incomplete download.  What happens if you lose communications halfway through an update?  You only have a partial firmware image, or a corrupted QSPI image.  What happens if it is started?  Don't count on a fault, it may just go into a loop.

Using the system ROM bootloader is easy but doesn't leave you any way to recover.  The usual solution is a custom bootloader that can reestablish communications and recover from a failed update.  Many of the STM32L parts have dual bank flash for just this reason: if an update fails the app can be recovered back to previous version.  On a restart the custom bootloader does a checksum to verify images and will detect a failure on the current bank, so it falls back to previous.

Same problem with the QSPI.  You need enough space for two copies, new and backup.  If this is an OTA update in the field it has to cover just about any contingency since there's no one to do it manually.  As for syncing QSPI to firmware, all you need is a tag file with the same checksum as the firmware.  If they don't match you know there's a fault.

A bulletproof update procedure is not a trivial problem, that's why you don't see a lot of example code.  Communications can be lost at any point.  If the update is large, you need to track progress and recover from a known good point, otherwise comm costs can be significant (think of OTA updates over a cell modem, in the middle of the Mojave Desert with unreliable coverage, where a 1MB image has to be restarted 5-10 times before it finishes without an error).

  Jack Peacock

[Jeroen3]

Posted on July 06, 2017 at 11:55

That is why I suggested to download the new application image for internal memory to the external memory first. 

Then only proceed with flashing internal memory if the downloaded image is verified.

[imtiaz]

Posted on July 06, 2017 at 21:59

Thanks guys for your feedback.

I have a fairly good idea how to do this now. My main gaps were how to work with a program that relies on the external flash for the program to run and how to separate the two binaries.

@

Peacock.Jack

In regards to failure from recovery - I have a wifi chip that downloads the firmware , stores it in its internal flash and verifies the MD5 hash of the file. Only then does it proceed to update the STM32F7 firmware . The only issue is that it does not have sufficientmemory to keep both internal/external image at once. So it has to be done in two parts. But this is quite good since most of the time the data in the external flash wont be changing.

Also- _ yes I will be having some kind of a CRC / Hash to match up the two images.

@

Lodder.Jeroen

I will look into SRecord , my plan was to write my own script , thanks.

@

Lodder.Jeroen

'

Are you able to program both memories using jtag/swd in one go already?' - not yet as we have to write a QSPI loader for our flash chip for the external programming tool and the debug IDE

Regards

Imtiaz

[Jeroen3]

Posted on July 07, 2017 at 07:35

abdullah.abdullah

‌ I might be able to help you with SRecord, I have used it before to split and checksum a full.hex. If you open a topic about it, mention me.

[imtiaz]

Posted on July 10, 2017 at 04:10

Hi Jeroen,

Can I take you up on the offer to help in using srecord .. just a few quick

pointers in how I can split the hex file into a couple of binaries.

I have downlaoded srecord-1.6.3-win32

Thanks

Imtiaz

[Jeroen3]

Posted on July 10, 2017 at 07:37

Sure. But 'split hex into binaries' is more than one operation. It took me a while to the hang of it.

First, call srec like this:

http://srecord.sourceforge.net/man/man1/srec_input.html

.exe @arguments.srec

And in those argument files, you put the commands. With comments starting with #.

http://srecord.sourceforge.net/man/man1/srec_examples.html#CROPPING THE DATA

of the file. Do this twice, for each region you want.

.\build\srec\input.hex # Take this file
-intel
-crop 0x08000000 0x0803E800 # Extract this region
-o .\build\srec\output.hex # Store it here
-intel
--obs=16�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?

Convert to

http://srecord.sourceforge.net/man/man1/srec_examples.html#BINARY FILES

:

.\build\srec\output.hex 
-intel
-fill 0xFF 0x08000000 0x0803E800 
-offset - 0x08000000
-o output.bin
-Binary�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?

Other examples:

If you want to add a checksum to a file, you can. This adds STM32F103 hardware checksum to 2_program.hex at location0x0803E7FC.

But you have to

http://srecord.sourceforge.net/man/man1/srec_examples.html#FILLING THE BLANKS

first.

�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?.\build\source.hex #input file name
-Intel #read input file in intel hex format
-fill 0xFF 0x08000000 0x0803F000 #fill gaps from with value 0xFF
-Output_Block_Size=16 #generate hex records with 16 byte (default 32 byte)
-address-length=4 #generate extended adress records type 04
-o .\build\srec\output.hex #output file name
-Intel #generate intel hex output file�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?

Crc:

.\build\srec\input.hex 
-intel 
-crop 0x08000000 0x0803E7FC
-STM32 0x0803E7FC
-o .\build\srec\output.hex 
-intel 
--obs=16
�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?

You can even merge the two hex files. Optionally later -unfill to optimize the size.

.\build\srec\file1.hex 
-intel
.\build\srec\file2.hex 
-intel
-o .\build\srec\file1_and_file2.hex 
-intel�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?�?

[AVI-crak]

Posted on July 10, 2017 at 06:12

There is a big difference in the algorithms for the programmer, and in the algorithms for the device user.

For the programmer, quick debugging is required, in such cases, every update for 10 minutes is very difficult. It is necessary to perform separate compilation of the code - for the microprocessor and for external memory.

To separate the compilation, you must clearly define your desires. Collect all the data for writing in qspi in one place, into one huge static structure. And place this structure in the qspi section.

A static structure may have data padding, or may not have it - from the point of view of the program code, it does not matter. However, there is a very big difference in the compilation speed of the code - 30 seconds or 2 hours (+ qspi 256MB). You just need to put the data to be filled into the processing of the re-processor by two values - process / ignore. I have a lot of small data, I use the connection files, otherwise it's very easy to get confused.

What we get:

Accessing data through a structure, bypassing magic addresses. GCC very optimally optimizes the start addresses of accesses to external memory. Even if you have a tenfold nesting of the fields of the structure - the starting address will be direct.

Free editing of the main program, except for new graphic data.

The headache bonus is an executable file for firmware 2 gigabytes in size. This is solved by the GCC configuration commands - exclude the qspi section from the firmware image. The appeal to memory is preserved, the time is shortened, the development will get acceleration.

The firmware for the user has a different structure.

A handheld bootloader in the initial addresses of the microprocessor, and a freely distributable firmware for the user. I strongly advise against taking the bootloader code from popular free resources - write your own. The worse your code is, the more difficult it will be for the Chinese to fake your device.

Algorithms for data validation are a great many. But you have the advantage of not being killed by the bootloader. Which by the way can not be copied to qspi before updating, everything else is possible - but it can not !!!!

By the way, the loss of power during the firmware update on the device is a user problem. This is exactly what is written in the instructions. Brick exchange or refund is not. If you ever give a slack - then you can say goodbye to your finances.

[Tesla DeLorean]

Posted on July 11, 2017 at 03:17

Here we create a binary package file, a bit like a .ZIP or .DFU file, in a form that describes the data and provides integrity checking. If appropriate we'll compress and encrypt the data. Simple tools will take the .AXF/ELF files as input, or .HEX depending on what makes the most sense.

The project builds in the normal way, so no special scatter file or linker script. A post link step take the object file output and translates to the deliverable image. The firmware, or loader, determines where this data is parked (ie High Flash, QSPI, MicroSD), and the update process blanks the areas of flash it needs and writes the new content, if the integrity of the flashed application is good we execute it, otherwise it stays in the boot loader, so we don't brick our devices. The integrity checking is multi-layered so the checks and signing added by the image generation process means it's not going to start the process with a bogus image. Failure therefore points more to errors introduced by the developer/compiler, or a physical failure of the flash sub-system.

[imtiaz]

Posted on July 11, 2017 at 03:31

Hi Clive,

In that case , how would you specify while writing the code the location of your data. Example if you had images or fonts etc that were too large for your internal flash?

Thanks

Imtiaz 

[Tesla DeLorean]

Posted on July 11, 2017 at 04:04

You'd just build your project as if you were flashing it directly from Keil, or whatever.

You'd take the output .HEX, or .ELF, and create, for example a .DFU file (see UM0391 for format specification). I've build a loader here for our micro-python implementation that reads the .DFU off a microSD card and writes it into memory, you'd need code that knows where internal FLASH is, and how to write into it, and similar routines for QSPI. But you could basically park the image in whatever linear memory have available, the image is 'address-less', its like a file, just a stream of data, with random access.

The .DFU can describe multiple memory regions, can break those regions into multiple blocks, and has a CRC over the entire image.

I'll link to UM0391 in another post so this doesn't get stuck in moderation.  

The .DFU is a simplified object file format, it strips a lot of symbols, debug, relocation, and other junk that is in the .AXF/.ELF object.

Take the .DFU file an zip it, this will give you some idea if compression would help, or not.

[Tesla DeLorean]

Posted on July 11, 2017 at 04:07

https://drive.google.com/open?id=0B7OY5pub_GfIR2ZBcDk3UXNjdzg

 

[AVI-crak]

Posted on July 11, 2017 at 06:36

Arm-none-eabi-objcopy -O binary -j .section_name filename.elf sw.bin

Collect the binary file for external memory from the firmware image file, using the section name as the target.

The image file '* .elf' contains a lot of additional information intended for debugging, linear space is segmented.

A binary file has a linear space, does not contain anything superfluous. It is intended for absolutely stupid programmers, like CH

When you need active protection and the ability to update the user - everything else should already work. Otherwise, the project will not end. Separate compilation allows you to complete the project quickly.

I found the shortest version of the demonstration, it is suitable for an example.

• list.h

  2

________________

Attachments :

list.h.zip : https://st--c.eu10.content.force.com/sfc/dist/version/download/?oid=00Db0000000YtG6&ids=0680X000006HyZa&d=%2Fa%2F0X0000000b9i%2FAGfpaw3_i.SL.SC4dSwYo_20ewXL_UV.q6zeXvqzfdc&asPdf=false