deeaitch/ota-reference-design
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
be12197ac07d7c9ca57089ec887ba6280edd0067
ota-reference-design/docs/02-update_strategies.md
deeaitch 2b49cbe010 Update Strategies
2026-08-03 16:02:00 -04:00

910 lines
28 KiB
Markdown
Raw Blame History

# 02. Choosing an OTA Update Strategy: Full Images, Packages, or Something Else?
In the previous chapter, we defined an OTA update as the controlled delivery and application of a new version of software or data to a remote device.
The next natural question is:
> How should such an update actually be performed?
At first glance, the answer seems simple. There are full-image updates, packages, binary deltas, containers, atomic filesystem trees, and other well-known approaches. All that remains is to compare them and choose the best one.
But this framing starts too late.
Before choosing a technology, we need to answer a more fundamental question:
> **What exactly are we trying to update?**
The entire system? The operating system? A single application? A set of files? Configuration? Certificates? Maps? A machine-learning model?
The answer determines almost everything else: the update unit, storage requirements, verification method, activation mechanism, rollback capability, and acceptable downtime.
In addition, some commonly compared approaches describe different aspects of the process. A full image or a package defines **what the update unit is**. An A/B layout defines **where the new version is prepared and how the system switches between versions**. A binary delta defines **how the amount of transferred data is reduced**. A cryptographic signature defines **how the origin of the update is verified**.
These are not necessarily mutually exclusive choices.
For example, a device may use:
- full system images as the update unit;
- an A/B layout for safe activation;
- binary deltas to reduce network traffic;
- cryptographic signatures to verify authenticity;
- a separate mechanism for configuration updates.
Therefore, choosing an OTA strategy is not selecting one item from a short list. It is the design of several related mechanisms.
---
## 1. What Exactly Is Being Updated?
OTA is often associated exclusively with device firmware. For a small microcontroller, this may be a sufficiently accurate description: new firmware may indeed replace almost all executable content on the device.
An embedded Linux system is usually more complex.
It may include:
```text
Bootloader
Kernel
Device tree
Root filesystem
System libraries
System services
Applications
Configuration
Persistent data
Certificates
Content
```
These components have different lifecycles.
The kernel may be updated rarely. A user application may be updated every month. Configuration may change several times a day. A certificate may be replaced only before it expires. Maps or machine-learning models may have their own independent release cycle.
The first step is therefore to identify the possible update targets.
### 1.1. The Complete System
In this case, an update is treated as a transition from one complete device version to another:
```text
System version N
|
v
System version N + 1
```
The new version may include the kernel, root filesystem, system libraries, services, and applications.
The defining property of this approach is that the system is delivered and tested as a whole. The device does not assemble the new version itself from many independently changing components. It receives a predefined state.
This improves reproducibility:
> The system version installed on the device should match the version built and tested by the build system.
This is the level at which full-image updates, A/B layouts, recovery partitions, and atomic switching between versions are usually considered.
### 1.2. The Operating System
Sometimes the system layer must be updated without replacing the entire contents of the device.
Update targets may include:
- the kernel;
- system libraries;
- system services;
- drivers and modules;
- individual distribution packages.
This approach often relies on a package manager and the existing infrastructure of a Linux distribution.
In this case, the device does not receive a completely prepared new state. It transforms the current system into a new one by installing, removing, or replacing individual components.
### 1.3. Applications
In many products, the base platform changes rarely while application software has a much shorter release cycle.
For example:
```text
Embedded Linux
├── Device manager
├── Communication service
├── Web interface
├── User interface
└── Diagnostics
```
An update may affect only one service or application.
This avoids replacing the entire system for a small application-level change. However, it introduces a new question: is the new application version compatible with the installed libraries, configuration, data schema, and other services?
The smaller the update unit, the smaller the change, but the larger the number of version combinations that may need to be supported.
### 1.4. Data and Content
OTA can update more than executable code.
Examples include:
- configuration;
- calibration data;
- certificates and key material;
- maps;
- voice packages;
- dictionaries;
- machine-learning models;
- filtering rules;
- static user-interface resources;
- parameter databases or diagnostic-code databases.
Such data often cannot reasonably be included in a full system image every time it changes. It may require its own versioning, verification, compatibility, and rollback rules.
For example, a new machine-learning model may require a specific runtime version. A new configuration may be incompatible with an older application. A new certificate may require coordinated changes on both the client and server sides.
Therefore, even a data update remains an architectural problem rather than a simple file transfer.
### 1.5. Multiple Levels at the Same Time
A real product usually has more than one update unit.
For example:
```text
Operating system -> full system image
Applications -> packages or containers
Maps and ML models -> separate artifacts
Configuration -> small atomic documents
Certificates -> dedicated rotation mechanism
```
The operating system may be updated several times per year, applications monthly, data weekly, and configuration whenever necessary.
The question is therefore not:
> How is the device updated?
It is:
> **Which parts of the device are updated, how independently, and how often?**
---
## 2. What Decisions Make Up an OTA Strategy?
The term *update strategy* is often used too broadly. As a result, full images, A/B layouts, deltas, containers, and signatures appear in the same list even though they solve different problems.
It is useful to separate an OTA design into several independent questions.
### 2.1. Update Unit
What is the installable artifact?
- a full image;
- a filesystem;
- a package;
- an application;
- a container;
- an individual file;
- a data set.
### 2.2. Delivery Method
How is the artifact transferred to the device?
- in full;
- compressed;
- as a binary delta;
- in blocks;
- as a stream;
- through a local gateway or intermediate cache.
### 2.3. Installation Method
How is the new version prepared?
- written to an inactive partition;
- applied directly to the running filesystem;
- assembled as a new filesystem tree;
- unpacked into a separate directory;
- imported as a new container image.
### 2.4. Activation Method
When does the new version become active?
- immediately after a file is replaced;
- after a service restart;
- after a device reboot;
- after an atomic symlink or partition switch;
- after confirmation by an external system.
### 2.5. Recovery Method
What happens if the update fails?
- return to the previous partition;
- boot into a recovery system;
- roll back a transaction;
- reinstall the previous package;
- return to the previous filesystem tree;
- require manual service.
These decisions are related, but they are not the same.
For example, a full-image update may or may not use an A/B layout. A package update may be transferred in full or as a delta. A container may be only the application delivery mechanism, while the base operating system is updated with a separate system image.
---
## 3. Full-Image Updates
With a full-image update, the device receives a prebuilt image of the system or one of its major partitions.
The process usually looks like this:
```text
Build system
|
| produces tested image
v
Update server
|
| delivers image
v
Device
|
| writes image
v
New system version
```
Instead of changing individual components sequentially, the device receives a complete version.
### 3.1. Primary Advantage
The main advantage is a reproducible state.
If the image was built and tested as a whole, the device should end up in exactly that state after a successful installation.
With a correct implementation, this reduces the number of possible version combinations:
```text
Version A
Version B
Version C
```
instead of:
```text
Kernel A + library B + service C + application D
Kernel A + library C + service C + application D
Kernel B + library C + service D + application D
...
```
The more independently updated components a system has, the faster the number of possible states grows.
### 3.2. Advantages
- predictable and reproducible system state;
- convenient system-level testing;
- no dependency resolution on the device;
- natural compatibility with an A/B layout;
- relatively clear rollback mechanism;
- the system partition can remain immutable during normal operation;
- easier identification of the exact installed component set.
### 3.3. Disadvantages
- large artifact size;
- increased storage requirements;
- long downloads over slow connections;
- transfer of unchanged data;
- user data and configuration require separate handling;
- a small application change may require releasing a new system image.
### 3.4. Typical Uses
This approach is especially natural for appliance-like devices where the entire system is treated as one product:
- industrial controllers;
- network devices;
- automotive control units;
- medical devices;
- terminals;
- specialized embedded Linux systems.
It is particularly attractive where reliability and reproducibility matter more than minimizing update size.
---
## 4. Package-Based Updates
With a package-based approach, the system is updated by installing or replacing individual packages.
These may use standard Linux mechanisms such as:
- `deb`;
- `rpm`;
- `opkg`;
or a custom package format created for a specific product.
A package usually contains:
- files;
- metadata;
- a version;
- dependencies;
- installation instructions;
- sometimes pre-installation and post-installation scripts.
### 4.1. Primary Advantage
Packages allow only the changed components to be updated.
If one service has been fixed, there is no need to transfer and replace the entire filesystem.
This is especially convenient when the device is already based on a maintained Linux distribution and its components have relatively independent lifecycles.
### 4.2. Advantages
- smaller update size;
- reuse of an existing ecosystem;
- independent component release cycles;
- only required changes need to be installed;
- a familiar model for Linux engineers;
- convenient dependency management when the repository is tightly controlled.
### 4.3. Disadvantages
- a large number of possible system states;
- dependence on correct dependency resolution;
- installation scripts may fail after partial execution;
- system-wide atomicity is more difficult;
- rolling back a package does not always restore the previous state;
- data and configuration migrations may be irreversible;
- the result may depend on the initial state of the device.
The last point is especially important.
If two devices begin the update from different states, the same sequence of package operations does not necessarily produce exactly the same final state.
### 4.4. A Package Manager Does Not Exclude OTA
Using a package manager does not make a mechanism “not real OTA.”
OTA describes the remote delivery and controlled application of an update. A package manager can absolutely be part of such a system.
The critical questions are not whether the implementation uses `apt`, `rpm`, `opkg`, or a custom installer, but:
- who controls the repository;
- how package authenticity is verified;
- which version transitions are allowed;
- what happens if power is lost;
- how partial installation is detected;
- whether a working state can be recovered;
- how every supported update path is tested.
A package manager alone solves only part of the OTA problem.
---
## 5. Updating Individual Applications
Application updates resemble package-based updates, but the managed unit is a specific application or service rather than an operating-system component.
An application may be delivered as:
- an executable file;
- an archive containing libraries and resources;
- an application bundle;
- a custom package;
- a separate filesystem tree;
- a container image.
### 5.1. Advantages
- small update size;
- fast release cycle;
- independence from base operating-system releases;
- only one service may need to restart;
- different teams can manage different components.
### 5.2. Disadvantages
- compatibility with the platform must be managed explicitly;
- interacting services may become version-incompatible;
- stable APIs or ABIs are required;
- data migrations complicate rollback;
- the overall product no longer has one obvious version.
Instead of a simple system version such as `3.2.0`, the product may have a combination such as:
```text
Base OS: 3.2
Device service: 7.4
User interface: 5.1
Communication stack: 2.8
Configuration schema: 4
```
This is not necessarily bad, but the complexity must be accepted and controlled deliberately.
---
## 6. Container-Based Updates
A container image may be used as the delivery and execution unit for an application.
In this model, the base operating system provides a container runtime, while application components are delivered separately.
```text
Host operating system
├── Container A
├── Container B
└── Container C
```
Updating an application may consist of downloading a new image and switching to it.
### 6.1. Advantages
- the application is delivered with a significant part of its dependencies;
- strong component isolation;
- convenient versioning;
- relatively simple return to a previous image;
- reuse of existing build and registry infrastructure;
- applications can have independent release cycles.
### 6.2. Disadvantages
- the container runtime becomes part of the trusted platform;
- storage and memory requirements increase;
- another operational layer is introduced;
- application state still requires separate management;
- container updates do not update the kernel, drivers, or base operating system;
- desktop or cloud practices cannot automatically be transferred to a constrained embedded device.
Containers do not replace OTA architecture. They may become one layer of it.
For example:
```text
Base OS -> A/B system images
Applications -> containers
Configuration -> signed documents
```
---
## 7. Atomic Filesystem Trees and Versioned System States
Between full images and traditional packages are approaches in which the system is represented as a versioned filesystem tree.
A new version is assembled separately, after which the device atomically switches to it. Unchanged objects may be reused, so a fully independent image does not always need to be transferred or stored.
Concepts in this class include:
- content-addressed storage;
- immutable filesystem trees;
- snapshot-based deployments;
- OSTree-like models.
The central idea is:
> The update is neither an in-place set of file changes nor necessarily a complete block image. It is a complete versioned filesystem state.
### 7.1. Advantages
- atomic switching between versions;
- reproducible system state;
- reuse of unchanged content;
- convenient rollback to a previous tree;
- fewer changes to the running system;
- a useful compromise between images and packages.
### 7.2. Disadvantages
- more complex storage model;
- additional build-tooling requirements;
- garbage collection and multi-version management;
- state outside the managed tree requires separate handling;
- integration with the bootloader and early boot may be non-trivial;
- the approach may be excessive for a simple device.
This mechanism can be attractive, but its benefits appear only when the entire system lifecycle is genuinely designed around immutable, versioned states.
---
## 8. Binary Delta Updates
A binary delta contains the difference between a known old version and a new version rather than the complete new artifact.
Conceptually:
```text
Old artifact + Delta = New artifact
```
This may significantly reduce the amount of transferred data, especially when only a small portion of the artifact has changed.
However, a delta is not a separate answer to the question of what is being updated.
It may be applied to:
- a system image;
- a partition;
- a package;
- a container layer;
- an individual file;
- a model or data set.
It is therefore more accurate to treat a binary delta as a **delivery optimization**, not as an update unit.
### 8.1. Advantages
- lower network traffic;
- faster download over slow connections;
- reduced mobile or satellite data cost;
- the ability to update large artifacts through a constrained channel.
### 8.2. Disadvantages
- the delta depends on a specific source version;
- multiple update paths may need to be stored or generated;
- corruption of the source artifact may make application impossible;
- the device requires resources to reconstruct the new version;
- both server-side and device-side complexity increase;
- the delta must be verified as strictly as the full artifact;
- savings may be small when changes are poorly localized.
For example, updating from version `1.0` to `1.1` may require one delta, while updating from `0.8` to `1.1` requires another.
With many supported versions, update paths multiply:
```text
0.8 -> 1.1
0.9 -> 1.1
1.0 -> 1.1
1.0 -> 1.0.1 -> 1.1
```
The more possible paths there are, the harder they are to create, test, and maintain.
Reduced network traffic is therefore purchased with additional system complexity.
---
## 9. Updating Individual Files and Data
The smallest update unit is an individual file or a small group of files.
This is a natural approach for:
- configuration;
- certificates;
- rules;
- calibration data;
- static content;
- models;
- maps;
- user-interface resources.
At first glance, such an update seems simple: download a file and replace the old one.
A reliable implementation still needs to answer:
- how the version is checked;
- how authenticity is verified;
- how integrity is verified;
- how compatibility is verified;
- how a partially written file is avoided;
- how replacement is made atomic;
- what happens if power is lost;
- how the previous version is preserved;
- who confirms successful activation.
Even a small file can render the device unusable if it contains critical configuration.
Artifact size does not determine the cost of failure.
---
## 10. Hybrid Strategies
In most complex products, the most reasonable solution is a combination of several mechanisms.
For example:
```text
Bootloader and base OS
-> signed full images
-> A/B installation
-> reboot-based activation
Applications
-> packages or containers
-> independent release cycle
Maps and ML models
-> separate artifacts
-> optional delta delivery
Configuration
-> small signed documents
-> atomic replacement
Certificates
-> dedicated rotation protocol
```
This allows different properties to be applied to different components.
For the base system, the priority may be reliability and reproducibility. For maps, it may be minimal network traffic. For applications, rapid release. For certificates, strict coordination and limited validity periods.
### 10.1. Advantages
- each data type receives an appropriate mechanism;
- the entire system does not need to be updated for a small change;
- reliability, bandwidth, and release speed can be optimized independently;
- the design reflects the real lifecycle of each component.
### 10.2. Disadvantages
- multiple mechanisms must be designed, tested, and maintained;
- compatibility modelling becomes more complex;
- identifying the complete product version becomes more difficult;
- dependencies may exist between separate update channels;
- rolling back one component may require rolling back others;
- observability and diagnostics become more complex.
A hybrid strategy must not become an accidental collection of unrelated mechanisms.
Each independent update channel should exist for a concrete reason:
- a different lifecycle;
- a different transfer cost;
- a different cost of failure;
- different activation requirements;
- a need for organizational independence.
Without such a reason, an additional mechanism only increases system complexity.
---
## 11. Why There Is No Best Strategy
The question:
> Which OTA strategy is best?
has no universal answer.
A full image may be an excellent choice for an industrial controller and too expensive for a device using satellite connectivity.
Packages may fit naturally into a Linux gateway and create an unacceptable number of states in a safety-critical system.
Containers may simplify independent application releases while adding a pointless operational layer to a small device.
A binary delta may dramatically reduce transfer cost while introducing more complexity than the rest of the updater.
The correct question is:
> **What constraints does our system have, and which update properties matter most?**
### 11.1. Reliability
What happens if power is lost in the middle of installation?
Must the device always return to the previous version?
Is manual recovery acceptable?
The more expensive physical access is, the more valuable atomic installation and automatic rollback become.
### 11.2. Storage Capacity
Is there enough space for two complete system images?
Can the device store the downloaded artifact while also constructing the new version?
How many previous versions must be retained?
Limited storage may exclude some approaches or require a more complex streaming installation process.
### 11.3. Connectivity Cost and Quality
Does the device use Ethernet, Wi-Fi, mobile, or satellite connectivity?
Is the connection stable or frequently interrupted?
Is traffic expensive?
Can downloads safely resume?
Over an expensive link, the complexity of delta updates may be justified. On a cheap local network, a full image may be simpler and cheaper to operate.
### 11.4. Update Frequency
Is the system updated once per year or several times per day?
Rare system updates and frequent data updates naturally require different mechanisms.
The more frequently a component changes, the more valuable an independent release cycle becomes.
### 11.5. Component Independence
Can the components really be updated independently?
Are their interfaces stable?
Is compatibility between versions maintained?
Independent delivery does not imply independent operation. If application version `5` requires library version `3`, the two must still be coordinated.
### 11.6. Cost of Failure
What happens if an update is unusable?
Does the device temporarily lose a minor feature? Stop a production line? Make a vehicle unavailable? Require a technician visit?
The same technical failure may have completely different consequences in different products.
### 11.7. Downtime Requirements
Can the device be rebooted?
How long may version activation take?
Is restarting one service acceptable?
Some systems may update overnight with a full reboot. Others must preserve their primary function almost continuously.
### 11.8. Support Lifetime
How many years will the device remain in service?
Must the system support updates from very old versions?
Will a team still be available to maintain a complex mechanism ten years from now?
The OTA architecture must not only be technically possible today. It must remain maintainable throughout the entire product lifecycle.
### 11.9. Team Capabilities
A complex solution does not become a good solution merely because it is technically elegant.
The team must be able to:
- build artifacts;
- test update paths;
- operate server infrastructure;
- investigate failed updates;
- manage cryptographic keys;
- maintain compatibility;
- recover devices.
If potential traffic savings require a mechanism the team cannot operate reliably, the optimization may increase the total cost of the product.
---
## 12. Strategy Is a Consequence of Constraints
OTA selection should not begin with a technology name.
It should begin with a model of the product:
```text
What is updated?
|
v
How often is it updated?
|
v
How expensive is failure?
|
v
What resources are available?
|
v
What recovery guarantees are required?
|
v
Which mechanism satisfies these constraints?
```
Therefore:
> **An OTA strategy is a consequence of system constraints, not a choice of the newest or most popular technology.**
The engineering process should look approximately like this:
1. Identify every independently updateable component.
2. Describe the lifecycle of each component.
3. Determine the cost of a failed update.
4. Record storage, memory, power, and connectivity constraints.
5. Define atomicity, activation, and rollback requirements.
6. Select an update unit for each component.
7. Only then choose specific technologies and tools.
The order matters.
If the process starts with a tool, the architecture will be forced to fit the tool's capabilities. If it starts with constraints, the tool can be selected for the actual problem.
---
## 13. Preliminary Comparison
The following table is not yet a complete decision matrix. It only summarizes the primary properties of the approaches discussed in this chapter.
| Approach | Primary Unit | Main Strength | Main Cost |
|---|---|---|---|
| Full image | System or partition | Reproducibility and predictable rollback | Artifact size and storage requirements |
| Packages | System component | Independent component updates | Many possible states and difficult rollback |
| Application | Individual service or bundle | Fast independent releases | Platform compatibility |
| Container | Application with dependencies | Isolation and controlled delivery | Additional runtime and operational complexity |
| Filesystem tree | Complete filesystem state | Atomicity with content reuse | More complex storage model |
| Binary delta | Difference between artifact versions | Reduced network traffic | Dependence on the source version |
| Individual files | Configuration or data | Small and targeted changes | Atomicity and compatibility must be designed explicitly |
| Hybrid model | Multiple update units | Optimization for different lifecycles | Overall system complexity |
There is intentionally no *best option* column.
The same approach may be strong or weak depending on context.
---
## 14. What Comes Next
This chapter describes the space of possible solutions, but it does not declare a single winner.
The next step will be a series of Architecture Decision Records in which each option is evaluated in the context of specific constraints.
Each ADR should answer not:
> What is this technology?
but:
> **Under which conditions would I choose it, and which consequences of that decision would I be prepared to accept?**
The following topics will be considered separately:
- when to choose full-image updates;
- when to choose package-based updates;
- when to update individual applications;
- when to use containers;
- when binary deltas are justified;
- when versioned filesystem trees are appropriate;
- how to design a hybrid strategy.
The results will then be summarized in a decision matrix.
The matrix should not select a solution automatically. Its purpose is to show which constraints push the architecture toward a particular approach and where trade-offs begin to conflict.
---
## Conclusion
An OTA update cannot be reduced to a choice between full images and packages.
A real system must separately define:
- what is updated;
- how independently it is updated;
- how it is delivered;
- where it is prepared;
- how it is activated;
- how it is verified;
- how it is rolled back;
- what happens when something fails.
Full images, packages, containers, filesystem trees, and individual files define different units of system management. A/B layouts, deltas, signatures, integrity checks, and recovery mechanisms add the properties required around those units.
A good OTA design therefore does not begin with a technology.
It begins with constraints, the product lifecycle, and an honest answer to the question:
> **What device state must we guarantee after every possible update outcome?**
Reference in New Issue View Git Blame Copy Permalink