Keep track of current record location for an external flash memory

[romey]

Posted on May 30, 2014 at 17:33

I am working on STM32F103 with an external flash memory chip via SPI. I use a very simple data format for storing data onto the flash memory, each record is 16 bytes, all records are saved sequentially. The problem is how to keep track of the current record location. I need to keep the data as long as possible, regardless resets and power cycling. When it reaches the full capacity it can overwrite the oldest records.

I can't use backup registers because no backup battery available in my case. I can try to implement a file system but my data structure is very simple, so a file system will be my last resort. I can keep the current record address at a fixed location on the flash memory, and update it every time I write a new record. But the flash memory has a limit of 100,000 program/erase cycles. This seems to be a classical problem with flash memory, I wonder if there is a simple solution.  

[Tesla DeLorean]

Posted on May 30, 2014 at 20:39

One of the simpler solutions is to write through the whole memory in a loop, with sequence numbers in your records, when you wrap you erase the oldest data, and when you start you implement a bisection algorithm on the erase block size to find the block with the newest data in it, and start from there with your next sequence number write.

Unless you have a file system that performs some wear leveling, or is flash aware, you are likely to destroy singular blocks

[romey]

Posted on May 30, 2014 at 23:48 Many thanks, Clive. Just one question, what to do with this sequence number when it overflows. I might have to use more bytes for it to be safe, then I am concerned about the overhead for each record may become too much. Still, this is a great idea and I am eager to try. I just came across AN2594 and would like to give it a try, too. I am also wondering if it is worth the trouble to implement this emulated EEPROM on the external flash instead of the on-chip flash.

[Tesla DeLorean]

Posted on May 31, 2014 at 00:33

You should look closely at the pseudo-EEPROM implementation, and the 10K (min) cycle life. It might not do what you want/expect.

How large is your proposed SPI Flash device? If you use a 32-bit sequence number, what are the chances your flash can hold 4 billion records? You should be able to determine if the records you have wrapped a some point. ie if you find a 0xFFFFxxxx record any subsequent record less than 0x80000000 is clearly newer rather than older. Find the highest and lowest record in the flash, if the highest is 0xFFFFFFFF then find the highest below 0x80000000. All the records will be written in a sequential order, and you'd probably want to have at least one erased block. When you start writing a new block your erase the next block ahead discarding the oldest data stored there. You can also create a structure in the block indicating the cycle count for the block, this combined with a 32-bit sequence number in the record, are not going to wrap within the lifetime of the device.

[Tesla DeLorean]

Posted on May 31, 2014 at 00:36

Also be aware that writing and erasing the flash using the pseudo-EEPROM can stall the processor in an undesirable manner.

[jpeacock2399]

Posted on May 31, 2014 at 01:04

The trick is recovering the next record pointer at reset.  The sequence numbers help to detect the break between head and tail, since they won't be sequential.  You might look at a binary search over the whole memory to locate the break.  It's very fast (Log2 * N) compared to a sequential search.  Or just look at the first record on every erase block until you see sequence number out of sequence or an erased block, then you located the block where the head/tail are.

Old timers might remember the DEC VMS Spiralog file system, worked on a similar principle where data blocks were written sequentially in a circular ''spiral''.  Extremely fast write, so-so read from disk since it had to scatter-gather the blocks in the spiral.   Jack Peacock