204 lines
11 KiB
Markdown
204 lines
11 KiB
Markdown
# 04. Starting an Updated System or Service
|
|
|
|
The previous chapter described who prepares, installs, and checks an update. Now we will follow what happens when the new software starts.
|
|
|
|
Installing files is only part of the work. The new software must start and do its job before the device decides to keep it.
|
|
|
|
We will use two examples: a system update with two system slots, and a package update for one service. These are examples of possible designs. Each needs a recovery mechanism suited to its platform.
|
|
|
|
## 1. From Power-On to the Application
|
|
|
|
A Linux device starts in several steps. A simple view is:
|
|
|
|
```text
|
|
Power on or reset
|
|
|
|
|
v
|
|
Early platform code
|
|
|
|
|
v
|
|
Bootloader selects and loads a system
|
|
|
|
|
v
|
|
Linux kernel starts
|
|
|
|
|
v
|
|
Root filesystem becomes available
|
|
|
|
|
v
|
|
Init starts services
|
|
|
|
|
v
|
|
Main application starts
|
|
```
|
|
|
|
The exact steps depend on the hardware. Early platform code prepares enough hardware to continue loading software. The **bootloader** loads the operating system and passes control to it.
|
|
|
|
The **kernel** is the core of Linux. It manages hardware and provides the basic functions that programs use. The **root filesystem** contains the files and directories needed by the system. **Init** is the first program Linux starts; it arranges the startup of services. A **service** is a program that runs to provide a function, such as communication or diagnostics.
|
|
|
|
The OTA manager starts later in this chain. If Linux cannot reach that point, the manager cannot repair the device. Recovery for an early failure must therefore work before the manager starts.
|
|
|
|
## 2. Which Software Will Start?
|
|
|
|
A system update may change the kernel, the root filesystem, or both. Some platforms also use a **device tree**, a description of hardware that helps the kernel use the device.
|
|
|
|
These parts must work together. Selecting a new root filesystem while loading an incompatible kernel can prevent the system from working.
|
|
|
|
The design must say which parts belong to a system version and how they are selected together. Later chapters will describe where these parts are stored.
|
|
|
|
A package update has a different starting point. Linux is already running. The device replaces an application package and starts the updated service on the existing system platform. It may need a service restart rather than a device reboot.
|
|
|
|
## 3. Installed Does Not Mean Working
|
|
|
|
Consider four different events:
|
|
|
|
| Event | What we know |
|
|
|---|---|
|
|
| Installed | The installation step has finished. |
|
|
| Started | The new system or service has started. |
|
|
| Checked | The required checks have passed. |
|
|
| Confirmed | The device has decided to keep the new version. |
|
|
|
|
A process can start and then crash. It can also keep running while failing to answer requests or read a required sensor.
|
|
|
|
A **health check** tests whether the software can do the work the device needs. For example, it might check that the main service answers a local request and can read a required device.
|
|
|
|
The checks need a time limit. Some services need time to become ready, but the device must not wait forever. The design must define what to check, how long to wait, and what counts as failure.
|
|
|
|
In these examples, the checks can run locally. Losing the server connection alone does not make the new software unhealthy. Reports can wait until the connection returns.
|
|
|
|
## 4. Example: Starting a New System in an A/B Layout
|
|
|
|
An **A/B layout** provides two places, called slots, for system software. In this example, slot A contains the working version. The installer prepares the new version in slot B while A keeps running.
|
|
|
|
The new version first gets a **trial boot**: a chance to start and pass the checks before it becomes the normal choice.
|
|
|
|
```text
|
|
A is running
|
|
|
|
|
Install new system in B
|
|
|
|
|
Select B for a trial boot
|
|
|
|
|
Restart into B
|
|
|
|
|
Check essential functions
|
|
|
|
|
+-- Pass --> Confirm B
|
|
|
|
|
+-- Fail --> Return to A and check it
|
|
```
|
|
|
|
The boot control mechanism limits the number of unconfirmed attempts. If B never reaches the program that confirms it, a later boot must be able to select A again.
|
|
|
|
This requires more than a choice between two slots. If B hangs, something must cause another reset. A **watchdog** is a timer that can reset the device when software stops showing that it is alive. Its setup must cover the failures the design intends to recover from.
|
|
|
|
The boot attempt state must be saved so a reset cannot erase the fact that an attempt was made. The device also needs a clear record of whether B has been confirmed.
|
|
|
|
Returning to a previous version is called **rollback**. Keeping A available makes rollback possible, but does not prove that A will still work. Shared data changed by B may also affect A.
|
|
|
|
[ADR-014: Trial Boot and Confirmation for System Updates](../adr/ADR-014-trial-boot-and-confirmation.md) explains when to choose this policy and what the platform must support.
|
|
|
|
The state diagram separates an explicit health-check failure from a startup failure that prevents the checks from running. It also shows the checks after returning to A.
|
|
|
|
|
|
|
|
[PlantUML source](../diagrams/plantuml/ab-trial-boot.puml) · [Runnable simulation and failure scenarios](../examples/simulations/ab-trial-boot/readme.md)
|
|
|
|
## 5. Example: Starting an Updated Service
|
|
|
|
Now imagine a device with a stable system platform and an independently updated communication service. Version 1 works. Version 2 arrives as a package.
|
|
|
|
A separate update supervisor controls this change. It can be part of the OTA manager. It must stay able to run when the communication service fails.
|
|
|
|
Before installation, the device keeps the exact version 1 package available locally, with the information needed to verify it. This example assumes that both versions work on the current platform and that the service's data remains usable by version 1.
|
|
|
|
```text
|
|
Service version 1 is running
|
|
|
|
|
Install version 2 and restart the service
|
|
|
|
|
Check that version 2 can do its job
|
|
|
|
|
+-- Pass --> Keep version 2 and report success
|
|
|
|
|
+-- Fail --> Block another automatic attempt at version 2
|
|
|
|
|
Stop the failed service
|
|
|
|
|
Reinstall version 1
|
|
|
|
|
Start and check version 1
|
|
|
|
|
Save and report the result
|
|
```
|
|
|
|
Reinstalling an older package is called a **downgrade**. Here, a **forced downgrade** means that the supervisor explicitly requests the previous version, even though the normal update policy prefers newer versions. It still verifies the package and checks compatibility.
|
|
|
|
The package manager must support this operation. The supervisor must also account for installation scripts and dependencies. Keeping an old package does not by itself make downgrade safe.
|
|
|
|
After recovery, the supervisor applies a **lock** that prevents another automatic installation of the failed version. Here, lock means a saved update rule, not a temporary lock used while writing files. The failure block is saved before recovery starts, so a restart cannot immediately trigger the same failed update again.
|
|
|
|
For this example, the rule blocks version 2 of this package on this device. Other designs may hold the package at version 1 until an operator reviews the failure. The scope of the lock and the conditions for removing it must be explicit.
|
|
|
|
If version 1 also fails its checks, recovery has failed. The supervisor records that outcome and follows the device's recovery policy. It must not report success just because the downgrade command finished.
|
|
|
|
[ADR-005: Package-Based Updates](../adr/ADR-005-package-based-updates.md) describes the conditions under which independent packages fit a stable system platform.
|
|
|
|
[ADR-015: Service Checks and Package Downgrade](../adr/ADR-015-service-checks-and-package-downgrade.md) describes the recovery policy used here, including the scope of the update lock and the conditions for removing it.
|
|
|
|
## 6. Who Handles Each Failure?
|
|
|
|
| Failure | Who handles it in these examples? |
|
|
|---|---|
|
|
| New system cannot reach Linux services | Boot control uses the saved trial state to select the previous system on a later boot. |
|
|
| New system hangs | A watchdog causes a reset; boot control then uses the trial state. |
|
|
| Linux starts, but an essential function fails | Health checks request system recovery. |
|
|
| Updated service crashes or fails its checks | The separate supervisor downgrades it and blocks the failed version. |
|
|
| Previous system or service also fails | The device follows its recovery policy and records that recovery failed. |
|
|
| Server connection is lost | Local checks and recovery continue; reporting waits. |
|
|
|
|
These mechanisms must be connected deliberately. Two system slots alone do not detect a hang. A service manager that restarts a crashed process does not, by itself, restore the previous package.
|
|
|
|
## 7. Remembering What Happened
|
|
|
|
Both examples need information that survives a restart. This is called **persistent state**.
|
|
|
|
The device needs enough information to answer questions such as:
|
|
|
|
- Which version was working before the update?
|
|
- Which version is being tried now?
|
|
- Has it passed the required checks?
|
|
- Has recovery started or finished?
|
|
- Is this version blocked from another automatic attempt?
|
|
- Which result still needs to be reported?
|
|
|
|
Saving this information is part of the design. Power can fail while it is being written. After a restart, the device must be able to read a valid state and decide what to do next.
|
|
|
|
A useful report includes the failed version, the reason for failure, the recovery result, the version now running, and any update lock. If the device is offline, it saves the report and sends it later.
|
|
|
|
## 8. Recovering Software and Data
|
|
|
|
Returning to old software does not automatically restore old data.
|
|
|
|
For example, version 2 may change a database into a format that version 1 cannot read. The old system slot or package can still be present, yet recovery can fail.
|
|
|
|
The update design must keep data compatible, provide a tested way to restore it, or define another recovery path. The same question applies to configuration and changes made by package installation scripts.
|
|
|
|
## 9. Related Architecture Decisions
|
|
|
|
These Architecture Decision Records (ADRs) explain when to choose the approaches used in this chapter:
|
|
|
|
| Question | Further reading |
|
|
|---|---|
|
|
| When should the software system be updated as one image? | [ADR-004: Full-System Image Updates](../adr/ADR-004-full-system-image-updates.md) |
|
|
| When do independent packages fit a stable system platform? | [ADR-005: Package-Based Updates](../adr/ADR-005-package-based-updates.md) |
|
|
| When should a new system get a trial boot before it is confirmed? | [ADR-014: Trial Boot and Confirmation for System Updates](../adr/ADR-014-trial-boot-and-confirmation.md) |
|
|
| When should a failed service update trigger downgrade and an update lock? | [ADR-015: Service Checks and Package Downgrade](../adr/ADR-015-service-checks-and-package-downgrade.md) |
|
|
|
|
## 10. What Comes Next?
|
|
|
|
Both examples follow the same pattern: start the new version, check its required functions, then keep it or recover. They use different mechanisms because a system boot and a service restart fail in different ways.
|
|
|
|
Next, we will look at [storage layout](05-storage-layout.md): where software versions, device data, and update state live, and what must remain available when an update fails.
|