Updating device remotely

Let's say you have a small-ish embedded device that communicates with remote server via network connection. It would be nice to update its software remotely, but since it is, well, remote, you generally want that the such updates are as fool-proof as possible. As in, no change (or very marginal one) of bricking the device.

This obviously means that the updates can't be done over active connection. Any issue with said connection might cause your update to be interrupted, causing it to fail and leaving system in unpredictable and likely corrupt state.
Solution for this is to load the software fully to local storage and to verify it against known checksums before starting the update, to make sure that everything is received properly.

Also, you generally do not want to software to perform full self-updates. If software is one monolithic block, any failure during update will easily leave it in corrupt state.
For this, the software must actually be in two parts; first is the bootloader, which in perfect world is immutable and never touched outside device production. And the second part is the main application software that can do everything but to actually perform the update (including downloading and initial verification of it).

The update process, then, is something like this:

• Main application contacts server and receives a notification of update
• Main application downloads update to local, nonvolatile storage reserved specifically for updates.
• Main application verifies that the update is valid, uncorrupt and optionally properly authenticated
• Main application sets nonvolatile system state to "perform update" mode, and starts system again
• Bootloader, on startup, checks system state and notices that update should be done, causing it to branch to update, instead of starting main application as would be normal operation
• Optionally bootloader verifies the update data again, to make sure everything is proper
• Bootloader then loads update data from nonvolatile memory and replaces main application data with the updated version. 
• If this process is interrupted (power loss for example), system is still in update mode and can resume update.
• After update is done, bootloader verifies it once again, and when satisfied that everything is correct, updates system state to normal operating mode and starts main application.

Depending on how far you want to go to make sure everything keeps rolling no matter what, there may also be two copies of software, so you can go back to old version in case new version somehow doesn't work. There is also possibility of encrypting the software when it is in transit, so that only the bootloader can decrypt it during update, to retain secrecy of your code.

Now, if you ever wanted to update the bootloader... Well, it can be done with this setup, too, but generally you don't want to do that, as failure there may be unrecoverable, or at best, difficult to recover from. Generally speaking, though, you want to avoid this as much as possible.

You may notice that I haven't written a word on how to actually implement this stuff. That's right. Every single system is slightly different here, so there isn't single solution for this, and without knowing your exact setup I can't really give any ideas. Other than ones provided above.

To turn off the radio...

I got a new car - Hyundai Kona electric - and as usual, wife isn't happy about changes. Things working differently and so on. This car, being quite different than previous ones we've had has its own quirks that you have to get to know first.

For example, to unplug charging cable, you have to unlock the car first. Even if it wasn't locked, you have to click unlock button on remote to be able to remove the plug (this is to prevent someone from just stealing your charging cable). It took me a while to figure this out, too.

However, next call was a bit more amusing.

"How do I turn off radio?!?"
"Open the door."
"..."

When you turn the car off, some parts keep being powered for a while longer, including the radio. When you open the door to step out, though, car powers these devices off (still keeping lights on for few minutes longer). Not exactly obvious behavior, but again, it makes sense when you think of it. You may, after all, want to sit in car, for example waiting for something, in which case you'd want the things keep being powered on until you step out (i.e. open the door).

Unexpected behavior is ... well, unexpected, no matter how much it makes sense afterwards.

Remember to store your data

When you are designing your small electronics project from ground up, starting with Arduino or Raspberry Pi is tempting option. Processor board does all the hard stuff and all you have to put on custom board is your application-specific electronics.

There's nothing wrong with this, of course, and for some projects is is absolutely great way to go. I have one project myself where I should have gone this route, but feature creep crept up on me.

However, if you want to mass-produce these things cheaply, this path may not be great, especially if the design is doing only something small that doesn't require processing power or included peripherals that something like RPi might offer, so designing the board around small microcontroller starts making sense.

Not to say you couldn't build a quick prototype around RPi first and then, when you find the concept feasible, designing custom board for that. That is actually a great way to do such things.

Microcontrollers offer lots of things, but memory for volatile data isn't one of them. With RPi you might use the SD card, but a simple microcontroller needs something else. While you could use internal program flash area for data, the storage isn't designed for this and you will wear out flash quickly, especially if data changes often. Most microcontrollers don't even offer EEPROM area for such data, and while some have battery-backed SRAM, the amount of data you can store there is tiny - often just tens of bytes.

So you will need external storage.

I have written about this topic few times before (here, here and here), but this time with other focus.

I like my memory chips small and easy to use. For that purpose an I2C-memory in small SO8 package is great and very simple to use. I2C memories have two huge drawbacks however; speed and size. The memory access time is limited by maximum I2C bitrate, typically around 400kHz (although higher speed devices do exist), and it is very difficult to find I2C memories with memory size larger than 256kbit (32kbytes). Possible, yes, but your options very quickly turn out very limited.

Again, for a small project this might not be a problem. Do the math as soon as you have any idea what and how much of it you will need to store, then proceed from there.

Lately, data size has become an issue for me, so I have moved mainly to SPI memories, still on SO8 (or similar) packaging. SPI is much, much faster than I2C (80MHz capable chips are common) and come in larger sizes. But they also are more difficult to use - especially if you want to use them to the max.

For example, writing to I2C is simple; send address, write indication and data. Done. There are some gotchas, like paging limitations with EEPROMs and so on but protocol-wise it's just that simple.

With SPI, you first have to first unlock pages first, send "write enable" command and only then you can write data in. And then you lock the pages again and disable writes - or at least you should. So simple write turns quickly into half a dozen separate commands, including polling write completion status and such.

Some time ago I ran onto this issue. I had tested previous versions of a board with a specific 64Mbit flash chip (sorry, exact details withheld) and it was working fine. I found out however that this specific chip was obsolete, so I had to look for replacement. And there was a new chip available, so I of course ordered few to test it.

And I couldn't write onto it. Everything else worked perfectly, but writes did nothing.

Did I mention that datasheet of this one memory chip has around 150 pages? Yes, this simple flash chip is very, very complex beast if you want to dig in, offering massive amount of features (of which I wanted to just the simplest writes and reads.)  Mass of these features turns into massive headache when you are trying to figure out why it isn't working for you, as previous chip offered no similar troubles.

Eventually I figured out the issue; the previous chip was a bit lax with Write Enable Latch bit (which enables write to chip, and is affected by abovementioned "write enable/disable" commands, but new one was more strict, disabling it automatically after every single operation. So by the time I was trying to write the actual data to the chip, Write Enable command sent a bit earlier had expired. Reordering commands a bit fixed the issue, but figuring this out cost me way too many hours of work.

This kind of behavior is exactly the reason I like those simple I2C chips so much. They, for the most part, function as dumb memory arrays, offering no such issues. But, as is common, these days I just need larger memory space and faster memory, so - for now - SPI is way to go. Despite the troubles they sometimes give.

Waiting for GPS fix...

Continuing on previous post.

You've got your GPS module working and it's returning the location, great! Now, the only problem is that whenever power goes out (for whatever reason) the module loses its location fix and it takes really long for it to regain it afterwards. Which may or may not be a major problem for your application.

GPS, while a nice system overall, has a weakness there. A receiver (and I do mean that literally; GPS is completely listen-only system; no data is ever transmitted out to satellites) receives data from satellites, but when receiver has no history information available, it takes a long time to find itself. I haven't studied the details myself, so it's a bit fuzzy for me too, so please, just take my word for it for now.

The more you know about where or when you are, the faster GPS can lock on.

Depending on module you are using this might be less an issue. Network-assisted GPS, like phones are using, can fall back to network provided data to start, so this generally isn't an issue.

Many GPS modules also remember their history, even after power loss, so they have rough idea where and/or when they are. Unfortunately, some lose their data after some time (week or so without power - using capacitor as back-up power source) and then you're in square one again.

If your system has its own clock and non-volatile storage, you can also hand this data to GPS module you're using yourself. Many GPS chips can take location and/or time data in to make the initial fix quicker, generally in format of NMEA message. In this specific case, I'm using SirfStar chip, which has proprietary $PSRF101 message for this specific purpose. Provide the chip with initial location and time data and the time-to-fix will drop down significantly!

Of course, every chip has a bit different commands for this, and for example the command above uses ECEF coordinates and GPS week/seconds numbers, instead of lat/lon pairs and real time you might be tempted to use. But at this points, it is just data transformation and easily searchable so I don't provide any links here. Especially since your module might use completely different chipset with different commands.

May your module find itself in record time now!