06 - atomic update
This commit is contained in:
deeaitch committed
1 parent
6d63457737
commit
2bdacaf37a
21 files changed
+1302
-365
No files matched your search
@@ -0,0 +1,65 @@
|
||||
BasedOnStyle: LLVM
|
||||
AccessModifierOffset: -4
|
||||
AlignAfterOpenBracket: Align
|
||||
AlignConsecutiveAssignments: None
|
||||
AlignOperands: Align
|
||||
AllowAllArgumentsOnNextLine: false
|
||||
AllowAllConstructorInitializersOnNextLine: false
|
||||
AllowAllParametersOfDeclarationOnNextLine: false
|
||||
AllowShortBlocksOnASingleLine: Always
|
||||
AllowShortCaseLabelsOnASingleLine: false
|
||||
AllowShortFunctionsOnASingleLine: All
|
||||
AllowShortIfStatementsOnASingleLine: Always
|
||||
AllowShortLambdasOnASingleLine: All
|
||||
AllowShortLoopsOnASingleLine: true
|
||||
AlwaysBreakAfterReturnType: None
|
||||
AlwaysBreakTemplateDeclarations: Yes
|
||||
BreakBeforeBraces: Custom
|
||||
BraceWrapping:
|
||||
AfterCaseLabel: false
|
||||
AfterClass: false
|
||||
AfterControlStatement: Never
|
||||
AfterEnum: false
|
||||
AfterFunction: false
|
||||
AfterNamespace: false
|
||||
AfterUnion: false
|
||||
BeforeCatch: false
|
||||
BeforeElse: false
|
||||
IndentBraces: false
|
||||
SplitEmptyFunction: false
|
||||
SplitEmptyRecord: true
|
||||
BreakBeforeBinaryOperators: None
|
||||
BreakBeforeTernaryOperators: true
|
||||
BreakConstructorInitializers: BeforeColon
|
||||
BreakInheritanceList: BeforeColon
|
||||
ColumnLimit: 0
|
||||
CompactNamespaces: false
|
||||
ContinuationIndentWidth: 8
|
||||
IndentCaseLabels: true
|
||||
IndentPPDirectives: None
|
||||
IndentWidth: 4
|
||||
KeepEmptyLinesAtTheStartOfBlocks: true
|
||||
MaxEmptyLinesToKeep: 2
|
||||
NamespaceIndentation: All
|
||||
ObjCSpaceAfterProperty: false
|
||||
ObjCSpaceBeforeProtocolList: true
|
||||
PointerAlignment: Right
|
||||
ReflowComments: false
|
||||
SpaceAfterCStyleCast: true
|
||||
SpaceAfterLogicalNot: false
|
||||
SpaceAfterTemplateKeyword: false
|
||||
SpaceBeforeAssignmentOperators: true
|
||||
SpaceBeforeCpp11BracedList: false
|
||||
SpaceBeforeCtorInitializerColon: true
|
||||
SpaceBeforeInheritanceColon: true
|
||||
SpaceBeforeParens: ControlStatements
|
||||
SpaceBeforeRangeBasedForLoopColon: false
|
||||
SpaceInEmptyParentheses: false
|
||||
SpacesBeforeTrailingComments: 0
|
||||
SpacesInAngles: false
|
||||
SpacesInCStyleCastParentheses: false
|
||||
SpacesInContainerLiterals: false
|
||||
SpacesInParentheses: false
|
||||
SpacesInSquareBrackets: false
|
||||
TabWidth: 4
|
||||
UseTab: Never
|
||||
+10
-1
@@ -1,9 +1,18 @@
|
||||
# Changelog
|
||||
|
||||
Reader-facing changes to the articles, architecture decisions, diagrams, and practical examples, newest first. Entries are grouped by commit date and summarize the Git history; issue references are included where recorded in commit messages. File links open the current material, while commit links show the historical changes.
|
||||
Reader-facing changes to the articles, architecture decisions, diagrams, and practical examples, newest first. New work is listed by change date; older entries summarize the Git history by commit date. Issue references are included where recorded in commit messages. File links open the current material, while commit links show the historical changes.
|
||||
|
||||
When adding or revising material, add a short entry under the relevant date with links to the affected files and any related issue or commit.
|
||||
|
||||
## 2026-10-20
|
||||
|
||||
- Added [chapter 06: Atomic Updates](docs/06-atomic-updates.md), covering preparation before activation, durable state, gaps between records, restart decisions, and the limits of package recovery.
|
||||
- Added [ADR-018: Prepare and Verify Before Activation](adr/ADR-018-prepare-and-verify-before-activation.md) and [ADR-019: Recover Interrupted Updates From Durable State](adr/ADR-019-recover-interrupted-updates-from-durable-state.md).
|
||||
- Added PlantUML sources for the [installation sequence](diagrams/plantuml/ab-installation-sequence.puml) and [restart decisions](diagrams/plantuml/ab-installation-recovery.puml). No new image previews are included.
|
||||
- Added a [C++17 interrupted installation model](examples/simulations/interrupted-installation/readme.md), with reset points, three supplied boot-record write outcomes, recovery checks, and CMake/CTest support. It models completed durable operations in memory, not real power-loss-safe storage.
|
||||
- Applied the root [.clang-format](.clang-format) to the new model and the existing [A/B trial boot example](examples/simulations/ab-trial-boot/readme.md). Existing C++ changes are formatting only.
|
||||
- Updated the [README](readme.md), [series overview](docs/00-series-overview.md), [chapter 05 continuation](docs/05-storage-layout.md), and [diagram index](diagrams/readme.md).
|
||||
|
||||
## 2026-10-11
|
||||
|
||||
### Storage layout
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# ADR-018: Prepare and Verify Before Activation
|
||||
|
||||
## Context
|
||||
|
||||
This example uses two system slots. A is confirmed and running. B is inactive and may be replaced. There is no pending trial, and the installer has exclusive access to B.
|
||||
|
||||
Power may stop during installation. A and shared device data must remain usable. Boot control provides an interface for saving a trial request separately from normal installation progress.
|
||||
|
||||
## Decision
|
||||
|
||||
Prepare B without changing the normal boot choice A. Save the target and release identity before writing. Complete the required storage operations and verify the installed contents against trusted release information before saving prepared state.
|
||||
|
||||
Request a trial through boot control only after the software and required preparation state have been saved. Boot control owns the trial target and attempts. Its storage method must have a defined result after interruption.
|
||||
|
||||
If an operation fails or its result is unknown, inspect saved state before retrying. After a restart, verify the installed release again before making a new activation request. Do not rewrite a slot already selected for a trial or running.
|
||||
|
||||
Keep confirmation separate. A prepared release can still fail startup checks. Preserve A and the data it needs during the supported recovery period.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Replace the Running Slot
|
||||
|
||||
This uses less storage, but an interrupted installation may damage the only runnable system. It requires another installation and recovery design.
|
||||
|
||||
### Select B Before Preparation Ends
|
||||
|
||||
This permits boot control to select incomplete software. It does not meet this example's recovery requirement.
|
||||
|
||||
### Treat Installation Success as Confirmation
|
||||
|
||||
Installation checks do not show that services start and work on the device. A trial remains necessary here.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Interrupted preparation leaves A available.
|
||||
- Activation has a clear precondition.
|
||||
- Installation checks and startup checks have separate roles.
|
||||
|
||||
### Negative
|
||||
|
||||
- Two system slots require space.
|
||||
- Storage completion, verification, and state writes need explicit ordering.
|
||||
- Shared data and shared boot components still need their own recovery rules.
|
||||
|
||||
## Applicability and Limits
|
||||
|
||||
Use this approach when a complete release can be prepared away from the running system. The platform must define write completion and boot-state recovery. This ADR does not select a filesystem or prove power-loss safety.
|
||||
|
||||
A package installed into the running filesystem does not gain this property without a separate preparation and activation mechanism.
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [Chapter 06: Atomic Updates](../docs/06-atomic-updates.md)
|
||||
- [ADR-014: Trial Boot and Confirmation](ADR-014-trial-boot-and-confirmation.md)
|
||||
- [ADR-019: Recover Interrupted Updates From Durable State](ADR-019-recover-interrupted-updates-from-durable-state.md)
|
||||
@@ -0,0 +1,59 @@
|
||||
# ADR-019: Recover Interrupted Updates From Durable State
|
||||
|
||||
## Context
|
||||
|
||||
Boot control and the OTA manager save different records. A restart can happen after one record changes but before another does. For example, a saved trial request can exist while installation progress still says prepared.
|
||||
|
||||
This example must continue locally without a server connection. Both retained software versions must understand the records needed during recovery.
|
||||
|
||||
## Decision
|
||||
|
||||
Assign one owner to each state transition. Boot control owns boot selection, trial attempts, and confirmation. The OTA manager owns installation progress and pending reports. Progress does not override boot state.
|
||||
|
||||
Save recovery information outside replaceable software as required by ADR-017. Define record validation, supported formats, and the action taken when a record is invalid. Use a storage method with a stated interruption contract and check write errors.
|
||||
|
||||
Before continuing an interrupted operation, read authoritative boot state and inspect the target release. Reconcile old progress with a saved boot request or outcome. Do not rewrite a pending or running target, reset trial attempts, or begin a fresh trial merely because a progress record is old.
|
||||
|
||||
For each operation, define whether to repeat it, verify its result, abandon it, or enter recovery. Bind progress to the release and target. A flag without this identity is not enough.
|
||||
|
||||
If no valid boot record remains, use the platform's separate recovery policy. Never infer confirmation from unreadable state. Retain results until reporting and recovery policies allow removal.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Restart Installation Without Reading Boot State
|
||||
|
||||
This may overwrite a selected target or reset the trial limit. It is unsuitable for the separate-record design used here.
|
||||
|
||||
### Put Every State Change in One Transaction
|
||||
|
||||
This can work if all readers and writers share suitable storage and transaction support. It is not assumed here because boot control and Linux may use different storage interfaces.
|
||||
|
||||
### Ask the Server What to Do After Every Restart
|
||||
|
||||
The server can help with deployment policy, but connectivity may be absent during boot and recovery. Local decisions must remain possible.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- A gap between state writes has a defined recovery path.
|
||||
- Old progress does not undo a saved boot decision.
|
||||
- Interrupted operations can be handled offline.
|
||||
|
||||
### Negative
|
||||
|
||||
- Record compatibility and ownership must be maintained across releases.
|
||||
- Recovery needs more than a single progress flag.
|
||||
- Storage behavior and interruption handling need platform tests.
|
||||
|
||||
## Applicability and Limits
|
||||
|
||||
Use this approach when an update spans several persistent records or operations. It does not make those writes one atomic transaction.
|
||||
|
||||
Checksums, redundant records, and file replacement are possible parts of a storage design. None is a complete power-loss solution without defined storage assumptions and tests. Package scripts and data migrations need their own repeat and recovery rules.
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [Chapter 06: Atomic Updates](../docs/06-atomic-updates.md)
|
||||
- [ADR-017: Keep Recovery State Outside Replaceable Software](ADR-017-keep-recovery-state-outside-replaceable-software.md)
|
||||
- [ADR-018: Prepare and Verify Before Activation](ADR-018-prepare-and-verify-before-activation.md)
|
||||
@@ -0,0 +1,42 @@
|
||||
@startuml
|
||||
title A/B installation and restart decisions
|
||||
skinparam shadowing false
|
||||
skinparam monochrome true
|
||||
state "A confirmed\nB inactive; no pending trial" as Idle
|
||||
state "Intent saved\nWriting B" as Writing
|
||||
state "B fully written\nNot yet verified" as Written
|
||||
state "B verified\nPrepared state saved" as Prepared
|
||||
state "Trial request saved\nOTA progress may still be old" as Requested
|
||||
state "Inspect saved boot state\nand target release" as Inspect
|
||||
state "Separate recovery policy" as Recovery
|
||||
state "Trial boot lifecycle\nSee existing trial boot diagram" as Trial
|
||||
|
||||
[*] --> Idle
|
||||
Idle --> Writing : save release and target
|
||||
Writing --> Written : complete storage writes
|
||||
Written --> Prepared : verify expected contents; save prepared state
|
||||
Prepared --> Requested : boot control saves trial request
|
||||
Requested --> Trial : restart; consume attempt through boot control
|
||||
Writing --> Inspect : restart
|
||||
Written --> Inspect : restart
|
||||
Prepared --> Inspect : restart
|
||||
Requested --> Inspect : restart before entering trial lifecycle
|
||||
Inspect --> Writing : A selected; B incomplete\nrestart preparation under policy
|
||||
Inspect --> Prepared : A selected; B complete\nverify again; save prepared state
|
||||
Inspect --> Requested : saved B trial found\nrepair progress; preserve attempt count
|
||||
Inspect --> Recovery : no valid boot state
|
||||
Written --> Idle : verification fails; abandon B
|
||||
Recovery --> [*]
|
||||
Trial --> [*]
|
||||
|
||||
note right of Inspect
|
||||
Read boot state before writing B.
|
||||
Never infer confirmation from progress.
|
||||
A saved boot request wins over old progress.
|
||||
end note
|
||||
legend bottom
|
||||
One A-to-B installation. No shared data migration or shared boot update.
|
||||
Writes assume the platform's durability and record validation contract.
|
||||
Trial outcomes and later updates belong to the trial boot lifecycle.
|
||||
endlegend
|
||||
@enduml
|
||||
@@ -0,0 +1,48 @@
|
||||
@startuml
|
||||
title A/B preparation and activation
|
||||
skinparam shadowing false
|
||||
skinparam monochrome true
|
||||
participant "OTA manager" as Manager
|
||||
participant Installer
|
||||
database "Slot B" as B
|
||||
database "OTA progress" as Progress
|
||||
participant "Boot control" as Boot
|
||||
|
||||
note over Manager, Boot
|
||||
A is confirmed. B is inactive. No earlier trial is pending.
|
||||
One installer owns B. A and shared data are preserved.
|
||||
end note
|
||||
Manager -> Progress : Save intent: release and target B
|
||||
Manager -> Installer : Prepare B
|
||||
Installer -> B : Write release
|
||||
Installer -> B : Complete required storage operations
|
||||
Installer -> B : Read and verify installed contents
|
||||
Installer --> Manager : Verified expected release
|
||||
note over Manager, Boot
|
||||
Reset before activation: boot A.
|
||||
Inspect B and verify again before a new activation request.
|
||||
end note
|
||||
Manager -> Progress : Save prepared release B
|
||||
Manager -> Boot : Save trial request for B
|
||||
Boot -> Boot : Persist target and attempt limit
|
||||
note right of Boot
|
||||
Interrupted state write:
|
||||
read valid old or new state,
|
||||
or enter defined recovery.
|
||||
end note
|
||||
Boot --> Manager : Trial request saved
|
||||
note over Manager, Boot
|
||||
Reset here: boot control may start B.
|
||||
Old OTA progress must not cause a rewrite or fresh attempts.
|
||||
end note
|
||||
Manager -> Progress : Save trial-requested result
|
||||
Manager -> Boot : Restart into trial
|
||||
ref over Manager, Boot
|
||||
Existing A/B trial boot, checks, and confirmation diagram
|
||||
end ref
|
||||
legend bottom
|
||||
Success path only. Any error stops forward progress and requires state inspection.
|
||||
Arrows assume completed durable operations under the platform storage contract.
|
||||
A saved trial request is not confirmation.
|
||||
endlegend
|
||||
@enduml
|
||||
+4
-2
@@ -1,12 +1,14 @@
|
||||
# OTA diagrams
|
||||
|
||||
PlantUML sources accompany chapters 03–05. The diagrams describe the illustrative A/B design, not a required platform or partition table.
|
||||
PlantUML sources accompany chapters 03–06. The diagrams describe the example A/B design, not a required platform or partition table.
|
||||
|
||||
| View | Source | Preview |
|
||||
|---|---|---|
|
||||
| Roles and the server/device boundary | [PlantUML](plantuml/ota-architecture.puml) | [PNG](rendered/ota-architecture.png) |
|
||||
| Trial boot, confirmation and recovery | [PlantUML](plantuml/ab-trial-boot.puml) | [PNG](rendered/ab-trial-boot.png) |
|
||||
| Storage areas and access responsibilities | [PlantUML](plantuml/ab-storage-layout.puml) | [PNG](rendered/ab-storage-layout.png) |
|
||||
| Preparation and activation sequence | [PlantUML](plantuml/ab-installation-sequence.puml) | Source only |
|
||||
| Installation states and restart decisions | [PlantUML](plantuml/ab-installation-recovery.puml) | Source only |
|
||||
|
||||
From the repository root, regenerate previews with a local PlantUML installation and Graphviz:
|
||||
|
||||
@@ -14,4 +16,4 @@ From the repository root, regenerate previews with a local PlantUML installation
|
||||
JAVA_TOOL_OPTIONS=-Djava.awt.headless=true plantuml -tpng -o ../rendered diagrams/plantuml/*.puml
|
||||
```
|
||||
|
||||
Keep sources and previews together when changing a diagram. No external rendering service is needed. State transitions assume valid durable records; these pictures do not establish power-loss safety. The [A/B simulation](../examples/simulations/ab-trial-boot/readme.md) explores the same logical transitions.
|
||||
Chapters 03–05 include existing previews. The chapter 06 diagrams are provided as sources only; generate previews locally when needed. No external rendering service is needed. These diagrams do not establish power-loss safety. The [A/B simulation](../examples/simulations/ab-trial-boot/readme.md) covers trial boot. The [interrupted installation model](../examples/simulations/interrupted-installation/readme.md) covers preparation, old progress, and supplied boot-record write outcomes.
|
||||
@@ -83,6 +83,7 @@ ota-reference-design/
|
||||
│ ├── 03-system-architecture.md
|
||||
│ ├── 04-starting-an-updated-system-or-service.md
|
||||
│ ├── 05-storage-layout.md
|
||||
│ ├── 06-atomic-updates.md
|
||||
│ ├── ...
|
||||
│
|
||||
├── diagrams/
|
||||
@@ -91,7 +92,8 @@ ota-reference-design/
|
||||
│
|
||||
├── examples/
|
||||
│ └── simulations/
|
||||
│ └── ab-trial-boot/
|
||||
│ ├── ab-trial-boot/
|
||||
│ └── interrupted-installation/
|
||||
│
|
||||
├── adr/
|
||||
│ ├── ADR-001-repository-name.md
|
||||
@@ -112,6 +114,7 @@ The chapters are designed to build on each other. The current chapters are:
|
||||
- [03. OTA System Architecture](03-system-architecture.md)
|
||||
- [04. Starting an Updated System or Service](04-starting-an-updated-system-or-service.md)
|
||||
- [05. Storage Layout](05-storage-layout.md)
|
||||
- [06. Atomic Updates](06-atomic-updates.md)
|
||||
|
||||
To catch up on additions and revisions, see the [changelog](../CHANGELOG.md), with dated summaries and links to chapters, diagrams, simulations, and related history.
|
||||
|
||||
@@ -198,7 +201,7 @@ Where possible, theoretical discussions are accompanied by practical material, i
|
||||
|
||||
The objective is to connect high-level architecture with practical implementation.
|
||||
|
||||
Available now: [PlantUML diagrams and PNG previews](../diagrams/readme.md) and a [C++ A/B trial boot simulation](../examples/simulations/ab-trial-boot/readme.md). Raspberry Pi and QEMU demonstrations are planned.
|
||||
Available now: [PlantUML diagrams](../diagrams/readme.md), a [C++ A/B trial boot simulation](../examples/simulations/ab-trial-boot/readme.md), and a [C++ interrupted installation model](../examples/simulations/interrupted-installation/readme.md). Chapters 03–05 have PNG previews; chapter 06 currently has sources only. Raspberry Pi and QEMU demonstrations are planned.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -217,4 +217,4 @@ This table describes requirements. A storage diagram does not prove they are met
|
||||
|
||||
A useful storage layout makes the boundaries clear: what the update replaces, what it preserves, and what recovery can still read.
|
||||
|
||||
Next, we will look at atomic updates. We will ask how to prepare and activate a new version without exposing an incomplete one, and how the device makes a safe decision after an interrupted operation.
|
||||
Next, [chapter 06: Atomic Updates](06-atomic-updates.md) explains how to prepare and activate a new version without exposing an incomplete one, and how the device makes a safe decision after an interrupted operation.
|
||||
@@ -0,0 +1,154 @@
|
||||
# 06. Atomic Updates: Preparation, Activation, and Interrupted Operations
|
||||
|
||||
[Chapter 05](05-storage-layout.md) described where software, device data, and recovery state live. Now we need to decide in which order to change them.
|
||||
|
||||
The main question is simple: **if power stops between two operations, what will the device do when it starts again?**
|
||||
|
||||
We will use the same A/B system and independent service examples. They show possible designs, rather than rules for every platform.
|
||||
|
||||
## 1. What Does Atomic Mean?
|
||||
|
||||
An **atomic operation** has one visible result: the old state or the new state. A reader must not see a partly applied result.
|
||||
|
||||
An OTA update includes many operations. Downloading, writing software, changing boot state, and checking the new system do not become one atomic operation just because the device has two slots.
|
||||
|
||||
It helps to separate three questions:
|
||||
|
||||
| Question | Mechanism in this A/B example |
|
||||
|---|---|
|
||||
| Can the installer prepare B without changing running system A? | Write only to the inactive slot. |
|
||||
| Can boot control select a complete system after an interrupted state change? | Use a defined state storage and recovery method. |
|
||||
| Does B work on this device? | Trial boot, local checks, and confirmation. |
|
||||
|
||||
**Activation** means allowing the new version to start. **Confirmation** means accepting it after checks. They are different events. B can be complete and safe to try, yet still fail at startup.
|
||||
|
||||
The aim is to make each visible switch safe and to define recovery between switches.
|
||||
|
||||
## 2. Prepare B While A Remains Available
|
||||
|
||||
Assume A is confirmed, B is inactive, and no earlier trial is pending. The installer has exclusive control of B. Shared boot code and persistent application data are outside its write target.
|
||||
|
||||
Before writing B, check the update policy, target hardware, available space, and trusted update information. Then:
|
||||
|
||||
1. Save the installation intent, including the release and target slot.
|
||||
2. Write the new software to B. Keep A as the normal boot choice.
|
||||
3. Complete the platform's required storage writes and check their results.
|
||||
4. Verify the installed contents against the expected release.
|
||||
5. Save that this release is prepared in B.
|
||||
6. Ask boot control to save a trial request for B.
|
||||
7. Save the local result of that request for later reporting.
|
||||
|
||||
The expected release information must come from a trusted source. Comparing two untrusted hashes does not establish trust. Signature policy will be covered with security.
|
||||
|
||||
A check of the downloaded file alone does not prove that writing B succeeded. Define how the installed contents are checked. If the installer decompresses or changes the artifact, the check must match the resulting format.
|
||||
|
||||
The platform must also define when writes have reached persistent storage. A read from a cache can match the expected bytes while required storage work is still pending.
|
||||
|
||||
If preparation fails, keep B unavailable for trial. Preserve A and enough state to retry or stop safely. Do not replace a slot that is running, selected for a pending trial, or still required for recovery.
|
||||
|
||||
This ordering is described in [ADR-018: Prepare and Verify Before Activation](../adr/ADR-018-prepare-and-verify-before-activation.md).
|
||||
|
||||
## 3. The Activation Boundary
|
||||
|
||||
In this example, boot control owns the trial target, attempt count, and confirmation. The OTA manager uses its interface; it does not maintain a second boot selector in a progress file.
|
||||
|
||||
Before boot control accepts B, the installed release and the state needed to try it must already be saved. A successful request means that boot control has saved the request according to its storage contract. It does not mean that B is confirmed.
|
||||
|
||||
If the request returns an error or its result is lost during a restart, the OTA manager must read boot state before retrying. An error does not always prove that nothing changed.
|
||||
|
||||
[Preparation and activation sequence — PlantUML source](../diagrams/plantuml/ab-installation-sequence.puml)
|
||||
|
||||
There is a gap between steps 6 and 7: boot control may have saved the request while the OTA manager still says “prepared.” After a restart, boot control can start B. The OTA manager must accept that saved boot decision and repair its progress record. It must not overwrite B or reset the trial attempt count.
|
||||
|
||||
Two separate records do not form one transaction. Safe ordering and recovery rules handle the states left between them.
|
||||
|
||||
## 4. Save State With an Explicit Storage Contract
|
||||
|
||||
**Durable state** is state that remains available after the failures covered by the design. A successful ordinary file write is not enough to establish this property.
|
||||
|
||||
For a Linux file, one possible replacement procedure is:
|
||||
|
||||
1. Create a temporary file in the same directory as the destination.
|
||||
2. Write the complete new record and check for errors.
|
||||
3. Use `fsync()` on the temporary file and check the result.
|
||||
4. Rename it over the destination.
|
||||
5. Use `fsync()` on the containing directory and check the result.
|
||||
|
||||
The Linux [fsync manual](https://man7.org/linux/man-pages/man2/fsync.2.html) explains that synchronizing a file does not by itself make its directory entry durable. This procedure needs a filesystem and storage stack that support the required operations. It is not a general recipe for raw flash or bootloader state.
|
||||
|
||||
After an interrupted operation, a reader needs a valid old record, a valid new record, or a defined recovery action. Records can include a format version, release identity, generation number, and a checksum. A checksum can help detect damage; it does not prove authenticity or make writes atomic.
|
||||
|
||||
Another option is redundant records. Keep the previous valid record while saving a new one, validate both after restart, and apply a defined selection rule. The storage design must explain how interrupted writes affect each copy. Two copies are not sufficient if one erase operation can destroy both.
|
||||
|
||||
When no valid boot record remains, use the platform's separate recovery policy. Do not interpret missing or unreadable state as confirmation. Do not invent a normal slot from an OTA progress file that boot control cannot trust.
|
||||
|
||||
## 5. Continue After a Restart
|
||||
|
||||
Before any new write, read the records owned by boot control and inspect the target. Check the release identity, rather than only a flag such as `ready = true`.
|
||||
|
||||
| State found after restart | Action in this example |
|
||||
|---|---|
|
||||
| A selected; B partly written | Keep A; restart preparation of B from the beginning. |
|
||||
| A selected; B contains the expected complete release; progress is old | Verify B again, then save prepared state. |
|
||||
| A selected; B prepared; no trial requested | Keep A until policy allows activation. |
|
||||
| B trial requested; progress still says prepared | Keep the boot request; repair progress without writing B. |
|
||||
| Trial ended or B was confirmed while reporting was interrupted | Read the saved boot outcome; do not create a fresh trial. |
|
||||
| Boot state is unreadable | Stop ordinary installation and use the platform recovery policy. |
|
||||
|
||||
An operation is **idempotent** if repeating it has the same intended effect as doing it once. Reading state and verifying the same release can usually be repeated. Rewriting all of B can be repeated here only while B is inactive and has no pending trial.
|
||||
|
||||
Repeating a trial request is different. If it restores the attempt count, a failed system could receive endless attempts. Read the existing trial and its result before deciding what to do.
|
||||
|
||||
Package scripts, database migrations, and external actions also need their own repeat rules. An “installation in progress” flag cannot tell whether such an action completed.
|
||||
|
||||
[Installation states and restart decisions — PlantUML source](../diagrams/plantuml/ab-installation-recovery.puml)
|
||||
|
||||
[ADR-019: Recover Interrupted Updates From Durable State](../adr/ADR-019-recover-interrupted-updates-from-durable-state.md) defines the ownership and recovery rules used here. [ADR-017](../adr/ADR-017-keep-recovery-state-outside-replaceable-software.md) defines where the records must live.
|
||||
|
||||
## 6. An Independent Service Update
|
||||
|
||||
A service package may replace files in the running filesystem. Stopping the service does not make those replacements one atomic operation.
|
||||
|
||||
If installation stops halfway, some files or scripts may have changed while others have not. Use the package manager's documented repair procedure before assuming that the previous package can be installed safely. Retain its dependencies, required inputs, and compatible data as described in [ADR-015](../adr/ADR-015-service-checks-and-package-downgrade.md).
|
||||
|
||||
For an application that supports it, another design is to prepare a complete version in a separate directory and switch a reference to it. This gives a smaller activation boundary. It still needs rules for saving that reference, stopping and starting the process, and keeping data compatible. A running process may still hold old files open after the reference changes.
|
||||
|
||||
Neither design makes database changes or external actions part of the software switch. Define their recovery separately.
|
||||
|
||||
## 7. A C++ Model of Interrupted Installation
|
||||
|
||||
The [C++17 example](../examples/simulations/interrupted-installation/readme.md) models one A-to-B installation. It keeps simulated persistent storage separate from an installer object. Recreating the installer removes its temporary verification result while keeping saved bytes and records.
|
||||
|
||||
It covers interruptions after intent, partial writing, complete writing, verification, prepared state, and trial request. It also models three possible results of an interrupted boot-state write: the old record, the new record, or no valid record.
|
||||
|
||||
Important checks include:
|
||||
|
||||
- B cannot be activated before verification and saved prepared state.
|
||||
- A remains unchanged during every installation scenario.
|
||||
- A saved trial request is preserved when progress is old.
|
||||
- A restart requires fresh verification before a new activation.
|
||||
- Invalid boot state requires separate recovery.
|
||||
|
||||
The model uses memory, expected byte strings, and explicit reset points. It does not implement a durable file store, cryptographic verification, or a real bootloader. It does not simulate interruption inside a storage write. The three boot-record outcomes are supplied assumptions, not evidence that a storage method produces them.
|
||||
|
||||
The existing [trial boot example](../examples/simulations/ab-trial-boot/readme.md) continues with attempts, health checks, confirmation, and return to A. The examples are separate programs.
|
||||
|
||||
## 8. Check the Design Against Failures
|
||||
|
||||
| Failure | Required result |
|
||||
|---|---|
|
||||
| Power stops during writing B | A remains available; B is not selected for trial. |
|
||||
| Installed contents fail verification | Stop activation; retain A. |
|
||||
| Saving prepared state fails | Do not request activation until saved state is checked. |
|
||||
| Saving the trial request is interrupted | Boot control reads a valid record or enters defined recovery. |
|
||||
| Trial request is saved but progress is old | Read boot state; do not rewrite B or refresh attempts. |
|
||||
| State storage fills up or returns a write error | Stop forward progress and inspect state before retrying. |
|
||||
| A package script stops after changing shared data | Follow its defined recovery procedure; software downgrade alone may be insufficient. |
|
||||
|
||||
Model tests check ordering. Platform tests must also interrupt real writes, resets, and recovery on the chosen storage. Test failures during recovery too. A completed write in a C++ model does not prove that hardware will retain it.
|
||||
|
||||
## 9. What Comes Next?
|
||||
|
||||
Preparation keeps incomplete software out of the boot path. Activation gives a complete version a chance to run. Confirmation still depends on checks after startup.
|
||||
|
||||
Next, we will look more closely at rollback: when to return to a previous version, what recovery needs, and when that previous version is no longer a usable choice.
|
||||
@@ -4,26 +4,27 @@
|
||||
#include <iostream>
|
||||
#include <stdexcept>
|
||||
|
||||
int main(int argc, char* argv[]) {
|
||||
int main(int argc, char *argv[]) {
|
||||
const std::string selected = argc > 1 ? argv[1] : "all";
|
||||
if (argc > 2 || selected == "--help" || selected == "-h" ||
|
||||
(selected != "all" && std::find(ota::scenarios().begin(), ota::scenarios().end(),
|
||||
selected) == ota::scenarios().end())) {
|
||||
std::ostream& output = (argc == 2 && (selected == "--help" || selected == "-h"))
|
||||
? std::cout : std::cerr;
|
||||
selected) == ota::scenarios().end())) {
|
||||
std::ostream &output = (argc == 2 && (selected == "--help" || selected == "-h"))
|
||||
? std::cout
|
||||
: std::cerr;
|
||||
output << "Usage: ab-simulate [all";
|
||||
for (const auto& scenario : ota::scenarios()) output << '|' << scenario;
|
||||
for (const auto &scenario: ota::scenarios()) output << '|' << scenario;
|
||||
output << "]\n";
|
||||
return &output == &std::cout ? 0 : 2;
|
||||
}
|
||||
try {
|
||||
for (const auto& scenario : ota::scenarios()) {
|
||||
for (const auto &scenario: ota::scenarios()) {
|
||||
if (selected != "all" && selected != scenario) continue;
|
||||
std::cout << "\n=== " << scenario << " ===\n";
|
||||
ota::TemporaryDirectory directory;
|
||||
ota::run_scenario(scenario, directory.path(), &std::cout);
|
||||
}
|
||||
} catch (const std::exception& error) {
|
||||
} catch (const std::exception &error) {
|
||||
std::cerr << "Simulation failed: " << error.what() << '\n';
|
||||
return 1;
|
||||
}
|
||||
|
||||
@@ -8,226 +8,223 @@
|
||||
#include <utility>
|
||||
|
||||
namespace ota {
|
||||
namespace {
|
||||
constexpr const char* phase_names[] = {
|
||||
"A confirmed", "Prepare B", "B prepared and verified", "B trial pending",
|
||||
"B trial running", "B confirmed", "Recovery to A pending",
|
||||
"A recovery running", "A recovery checked", "Separate recovery required"
|
||||
};
|
||||
namespace {
|
||||
constexpr const char *phase_names[] = {
|
||||
"A confirmed", "Prepare B", "B prepared and verified", "B trial pending",
|
||||
"B trial running", "B confirmed", "Recovery to A pending",
|
||||
"A recovery running", "A recovery checked", "Separate recovery required"};
|
||||
|
||||
Phase read_phase(const std::string& value) {
|
||||
for (int i = 0; i < 10; ++i) {
|
||||
if (value == phase_names[i]) return static_cast<Phase>(i);
|
||||
}
|
||||
throw std::runtime_error("Invalid saved phase");
|
||||
}
|
||||
Phase read_phase(const std::string &value) {
|
||||
for (int i = 0; i < 10; ++i) {
|
||||
if (value == phase_names[i]) return static_cast<Phase>(i);
|
||||
}
|
||||
throw std::runtime_error("Invalid saved phase");
|
||||
}
|
||||
|
||||
Slot read_slot(const std::string& value) {
|
||||
if (value == "A") return Slot::A;
|
||||
if (value == "B") return Slot::B;
|
||||
throw std::runtime_error("Invalid saved slot");
|
||||
}
|
||||
Slot read_slot(const std::string &value) {
|
||||
if (value == "A") return Slot::A;
|
||||
if (value == "B") return Slot::B;
|
||||
throw std::runtime_error("Invalid saved slot");
|
||||
}
|
||||
|
||||
void require(bool condition, const char* message) {
|
||||
if (!condition) throw std::logic_error(message);
|
||||
}
|
||||
} // namespace
|
||||
void require(bool condition, const char *message) {
|
||||
if (!condition) throw std::logic_error(message);
|
||||
}
|
||||
}// namespace
|
||||
|
||||
const char* name(Phase phase) { return phase_names[static_cast<int>(phase)]; }
|
||||
const char* name(Slot slot) { return slot == Slot::A ? "A" : "B"; }
|
||||
const char *name(Phase phase) { return phase_names[static_cast<int>(phase)]; }
|
||||
const char *name(Slot slot) { return slot == Slot::A ? "A" : "B"; }
|
||||
|
||||
Device::Device(const std::filesystem::path& directory)
|
||||
: path_(directory / "boot-state.txt") {
|
||||
if (!std::filesystem::exists(path_)) {
|
||||
save();
|
||||
return;
|
||||
}
|
||||
std::ifstream input(path_);
|
||||
std::string header, phase, normal, trial;
|
||||
int a_ready = 0, b_ready = 0;
|
||||
if (!(input >> header >> std::quoted(phase) >> normal >> trial
|
||||
>> state_.attempts_left >> a_ready >> b_ready
|
||||
>> std::quoted(state_.outcome)) || header != "OTA_AB_STATE_V1") {
|
||||
throw std::runtime_error("Unreadable saved boot state");
|
||||
}
|
||||
input >> std::ws;
|
||||
if (!input.eof() || state_.attempts_left < 0 ||
|
||||
(a_ready != 0 && a_ready != 1) || (b_ready != 0 && b_ready != 1)) {
|
||||
throw std::runtime_error("Invalid saved boot state");
|
||||
}
|
||||
state_.phase = read_phase(phase);
|
||||
state_.normal = read_slot(normal);
|
||||
if (trial != "none") state_.trial = read_slot(trial);
|
||||
state_.a_ready = a_ready != 0;
|
||||
state_.b_ready = b_ready != 0;
|
||||
}
|
||||
|
||||
void Device::save() const {
|
||||
auto temporary = path_;
|
||||
temporary += ".tmp";
|
||||
std::ofstream output(temporary, std::ios::trunc);
|
||||
output << "OTA_AB_STATE_V1\n" << std::quoted(name(state_.phase)) << '\n'
|
||||
<< name(state_.normal) << '\n'
|
||||
<< (state_.trial ? name(*state_.trial) : "none") << '\n'
|
||||
<< state_.attempts_left << '\n'
|
||||
<< state_.a_ready << ' ' << state_.b_ready << '\n'
|
||||
<< std::quoted(state_.outcome) << '\n';
|
||||
output.close();
|
||||
if (!output) throw std::runtime_error("Cannot save boot state");
|
||||
// Teaching stand-in: rename alone does not guarantee power-loss durability.
|
||||
std::filesystem::rename(temporary, path_);
|
||||
}
|
||||
|
||||
void Device::prepare(bool complete) {
|
||||
require(state_.phase == Phase::AConfirmed || state_.phase == Phase::PreparingB,
|
||||
"Cannot prepare B in this phase");
|
||||
state_.b_ready = false;
|
||||
state_.phase = Phase::PreparingB;
|
||||
save();
|
||||
if (complete) {
|
||||
// Represents complete installation AND artifact verification.
|
||||
state_.b_ready = true;
|
||||
state_.phase = Phase::BPrepared;
|
||||
save();
|
||||
}
|
||||
}
|
||||
|
||||
void Device::request_trial(int attempts) {
|
||||
require(state_.phase == Phase::BPrepared && state_.b_ready,
|
||||
"B must be prepared and verified before trial");
|
||||
require(attempts > 0, "The trial needs at least one attempt");
|
||||
state_.phase = Phase::TrialPending;
|
||||
state_.trial = Slot::B;
|
||||
state_.attempts_left = attempts;
|
||||
save();
|
||||
}
|
||||
|
||||
Slot Device::boot() {
|
||||
require(state_.phase != Phase::SeparateRecoveryRequired,
|
||||
"Separate recovery is required; do not alternate failed systems");
|
||||
if (state_.trial) {
|
||||
if (state_.attempts_left > 0) {
|
||||
--state_.attempts_left;
|
||||
state_.phase = Phase::TrialRunning;
|
||||
// Persist the consumed attempt BEFORE handing control to B.
|
||||
Device::Device(const std::filesystem::path &directory)
|
||||
: path_(directory / "boot-state.txt") {
|
||||
if (!std::filesystem::exists(path_)) {
|
||||
save();
|
||||
running_ = *state_.trial;
|
||||
return *running_;
|
||||
}
|
||||
state_.phase = Phase::RecoveryPending;
|
||||
state_.trial.reset();
|
||||
save();
|
||||
}
|
||||
if (state_.phase == Phase::RecoveryPending) {
|
||||
state_.phase = Phase::RecoveryRunning;
|
||||
save();
|
||||
}
|
||||
running_ = state_.normal;
|
||||
return *running_;
|
||||
}
|
||||
|
||||
void Device::check(bool healthy) {
|
||||
require(running_.has_value(), "Boot before checking the running system");
|
||||
require(state_.phase == Phase::TrialRunning || state_.phase == Phase::RecoveryRunning,
|
||||
"Checks require a running trial or recovery");
|
||||
if (state_.phase == Phase::TrialRunning) {
|
||||
state_.trial.reset();
|
||||
state_.attempts_left = 0;
|
||||
if (healthy) {
|
||||
state_.phase = Phase::BConfirmed;
|
||||
state_.normal = Slot::B;
|
||||
state_.outcome = "B confirmed";
|
||||
} else {
|
||||
state_.phase = Phase::RecoveryPending;
|
||||
}
|
||||
} else if (healthy) {
|
||||
state_.phase = Phase::RecoveryChecked;
|
||||
state_.outcome = "Recovered to A";
|
||||
} else {
|
||||
state_.phase = Phase::SeparateRecoveryRequired;
|
||||
state_.outcome = "Recovery A failed";
|
||||
}
|
||||
save();
|
||||
}
|
||||
|
||||
TemporaryDirectory::TemporaryDirectory() {
|
||||
std::random_device random;
|
||||
for (int attempt = 0; attempt < 100; ++attempt) {
|
||||
auto candidate = std::filesystem::temp_directory_path() /
|
||||
("ota-ab-simulation-" + std::to_string(random()) + "-" + std::to_string(random()));
|
||||
if (std::filesystem::create_directory(candidate)) {
|
||||
path_ = std::move(candidate);
|
||||
return;
|
||||
}
|
||||
std::ifstream input(path_);
|
||||
std::string header, phase, normal, trial;
|
||||
int a_ready = 0, b_ready = 0;
|
||||
if (!(input >> header >> std::quoted(phase) >> normal >> trial >> state_.attempts_left >> a_ready >> b_ready >> std::quoted(state_.outcome)) || header != "OTA_AB_STATE_V1") {
|
||||
throw std::runtime_error("Unreadable saved boot state");
|
||||
}
|
||||
input >> std::ws;
|
||||
if (!input.eof() || state_.attempts_left < 0 ||
|
||||
(a_ready != 0 && a_ready != 1) || (b_ready != 0 && b_ready != 1)) {
|
||||
throw std::runtime_error("Invalid saved boot state");
|
||||
}
|
||||
state_.phase = read_phase(phase);
|
||||
state_.normal = read_slot(normal);
|
||||
if (trial != "none") state_.trial = read_slot(trial);
|
||||
state_.a_ready = a_ready != 0;
|
||||
state_.b_ready = b_ready != 0;
|
||||
}
|
||||
throw std::runtime_error("Cannot create a unique temporary directory");
|
||||
}
|
||||
|
||||
TemporaryDirectory::~TemporaryDirectory() {
|
||||
std::error_code ignored;
|
||||
std::filesystem::remove_all(path_, ignored);
|
||||
}
|
||||
|
||||
const std::vector<std::string>& scenarios() {
|
||||
static const std::vector<std::string> names = {
|
||||
"success", "health-failure", "attempts-exhausted", "restart-during-trial",
|
||||
"interrupted-preparation", "recovery-failure"
|
||||
};
|
||||
return names;
|
||||
}
|
||||
|
||||
Device run_scenario(const std::string& scenario, const std::filesystem::path& directory,
|
||||
std::ostream* output) {
|
||||
if (std::find(scenarios().begin(), scenarios().end(), scenario) == scenarios().end()) {
|
||||
throw std::invalid_argument("Unknown scenario: " + scenario);
|
||||
void Device::save() const {
|
||||
auto temporary = path_;
|
||||
temporary += ".tmp";
|
||||
std::ofstream output(temporary, std::ios::trunc);
|
||||
output << "OTA_AB_STATE_V1\n"
|
||||
<< std::quoted(name(state_.phase)) << '\n'
|
||||
<< name(state_.normal) << '\n'
|
||||
<< (state_.trial ? name(*state_.trial) : "none") << '\n'
|
||||
<< state_.attempts_left << '\n'
|
||||
<< state_.a_ready << ' ' << state_.b_ready << '\n'
|
||||
<< std::quoted(state_.outcome) << '\n';
|
||||
output.close();
|
||||
if (!output) throw std::runtime_error("Cannot save boot state");
|
||||
// Teaching stand-in: rename alone does not guarantee power-loss durability.
|
||||
std::filesystem::rename(temporary, path_);
|
||||
}
|
||||
Device device(directory);
|
||||
auto show = [&](const std::string& event) {
|
||||
if (!output) return;
|
||||
const auto& state = device.state();
|
||||
*output << event << ": " << name(state.phase) << "; normal=" << name(state.normal)
|
||||
<< "; attempts_left=" << state.attempts_left << "; running="
|
||||
<< (device.running() ? name(*device.running()) : "none") << '\n';
|
||||
};
|
||||
auto reset = [&] {
|
||||
device = Device(directory);
|
||||
const auto selected = device.boot();
|
||||
show(std::string("Reset -> ") + name(selected));
|
||||
};
|
||||
auto message = [&](const char* text) { if (output) *output << text << '\n'; };
|
||||
|
||||
show("Initial state");
|
||||
if (scenario == "interrupted-preparation") {
|
||||
device.prepare(false);
|
||||
show("Interrupted installation: B remains ineligible");
|
||||
void Device::prepare(bool complete) {
|
||||
require(state_.phase == Phase::AConfirmed || state_.phase == Phase::PreparingB,
|
||||
"Cannot prepare B in this phase");
|
||||
state_.b_ready = false;
|
||||
state_.phase = Phase::PreparingB;
|
||||
save();
|
||||
if (complete) {
|
||||
// Represents complete installation AND artifact verification.
|
||||
state_.b_ready = true;
|
||||
state_.phase = Phase::BPrepared;
|
||||
save();
|
||||
}
|
||||
}
|
||||
|
||||
void Device::request_trial(int attempts) {
|
||||
require(state_.phase == Phase::BPrepared && state_.b_ready,
|
||||
"B must be prepared and verified before trial");
|
||||
require(attempts > 0, "The trial needs at least one attempt");
|
||||
state_.phase = Phase::TrialPending;
|
||||
state_.trial = Slot::B;
|
||||
state_.attempts_left = attempts;
|
||||
save();
|
||||
}
|
||||
|
||||
Slot Device::boot() {
|
||||
require(state_.phase != Phase::SeparateRecoveryRequired,
|
||||
"Separate recovery is required; do not alternate failed systems");
|
||||
if (state_.trial) {
|
||||
if (state_.attempts_left > 0) {
|
||||
--state_.attempts_left;
|
||||
state_.phase = Phase::TrialRunning;
|
||||
// Persist the consumed attempt BEFORE handing control to B.
|
||||
save();
|
||||
running_ = *state_.trial;
|
||||
return *running_;
|
||||
}
|
||||
state_.phase = Phase::RecoveryPending;
|
||||
state_.trial.reset();
|
||||
save();
|
||||
}
|
||||
if (state_.phase == Phase::RecoveryPending) {
|
||||
state_.phase = Phase::RecoveryRunning;
|
||||
save();
|
||||
}
|
||||
running_ = state_.normal;
|
||||
return *running_;
|
||||
}
|
||||
|
||||
void Device::check(bool healthy) {
|
||||
require(running_.has_value(), "Boot before checking the running system");
|
||||
require(state_.phase == Phase::TrialRunning || state_.phase == Phase::RecoveryRunning,
|
||||
"Checks require a running trial or recovery");
|
||||
if (state_.phase == Phase::TrialRunning) {
|
||||
state_.trial.reset();
|
||||
state_.attempts_left = 0;
|
||||
if (healthy) {
|
||||
state_.phase = Phase::BConfirmed;
|
||||
state_.normal = Slot::B;
|
||||
state_.outcome = "B confirmed";
|
||||
} else {
|
||||
state_.phase = Phase::RecoveryPending;
|
||||
}
|
||||
} else if (healthy) {
|
||||
state_.phase = Phase::RecoveryChecked;
|
||||
state_.outcome = "Recovered to A";
|
||||
} else {
|
||||
state_.phase = Phase::SeparateRecoveryRequired;
|
||||
state_.outcome = "Recovery A failed";
|
||||
}
|
||||
save();
|
||||
}
|
||||
|
||||
TemporaryDirectory::TemporaryDirectory() {
|
||||
std::random_device random;
|
||||
for (int attempt = 0; attempt < 100; ++attempt) {
|
||||
auto candidate = std::filesystem::temp_directory_path() /
|
||||
("ota-ab-simulation-" + std::to_string(random()) + "-" + std::to_string(random()));
|
||||
if (std::filesystem::create_directory(candidate)) {
|
||||
path_ = std::move(candidate);
|
||||
return;
|
||||
}
|
||||
}
|
||||
throw std::runtime_error("Cannot create a unique temporary directory");
|
||||
}
|
||||
|
||||
TemporaryDirectory::~TemporaryDirectory() {
|
||||
std::error_code ignored;
|
||||
std::filesystem::remove_all(path_, ignored);
|
||||
}
|
||||
|
||||
const std::vector<std::string> &scenarios() {
|
||||
static const std::vector<std::string> names = {
|
||||
"success", "health-failure", "attempts-exhausted", "restart-during-trial",
|
||||
"interrupted-preparation", "recovery-failure"};
|
||||
return names;
|
||||
}
|
||||
|
||||
Device run_scenario(const std::string &scenario, const std::filesystem::path &directory,
|
||||
std::ostream *output) {
|
||||
if (std::find(scenarios().begin(), scenarios().end(), scenario) == scenarios().end()) {
|
||||
throw std::invalid_argument("Unknown scenario: " + scenario);
|
||||
}
|
||||
Device device(directory);
|
||||
auto show = [&](const std::string &event) {
|
||||
if (!output) return;
|
||||
const auto &state = device.state();
|
||||
*output << event << ": " << name(state.phase) << "; normal=" << name(state.normal)
|
||||
<< "; attempts_left=" << state.attempts_left << "; running="
|
||||
<< (device.running() ? name(*device.running()) : "none") << '\n';
|
||||
};
|
||||
auto reset = [&] {
|
||||
device = Device(directory);
|
||||
const auto selected = device.boot();
|
||||
show(std::string("Reset -> ") + name(selected));
|
||||
};
|
||||
auto message = [&](const char *text) { if (output) *output << text << '\n'; };
|
||||
|
||||
show("Initial state");
|
||||
if (scenario == "interrupted-preparation") {
|
||||
device.prepare(false);
|
||||
show("Interrupted installation: B remains ineligible");
|
||||
reset();
|
||||
return device;
|
||||
}
|
||||
device.prepare();
|
||||
show("Installation and verification complete");
|
||||
device.request_trial(2);
|
||||
show("Trial requested");
|
||||
reset();
|
||||
if (scenario == "attempts-exhausted" || scenario == "restart-during-trial") {
|
||||
message("No confirmation: simulate a reset caused by power loss or watchdog");
|
||||
reset();
|
||||
}
|
||||
if (scenario == "attempts-exhausted") {
|
||||
message("B still cannot reach health checks; simulate another reset");
|
||||
reset();
|
||||
device.check(true);
|
||||
show("Recovery checks pass");
|
||||
} else if (scenario == "health-failure" || scenario == "recovery-failure") {
|
||||
device.check(false);
|
||||
show("B health checks fail");
|
||||
reset();
|
||||
device.check(scenario != "recovery-failure");
|
||||
show("Recovery checks complete");
|
||||
} else {
|
||||
device.check(true);
|
||||
show("B health checks pass; confirmation saved");
|
||||
reset();
|
||||
}
|
||||
return device;
|
||||
}
|
||||
device.prepare();
|
||||
show("Installation and verification complete");
|
||||
device.request_trial(2);
|
||||
show("Trial requested");
|
||||
reset();
|
||||
if (scenario == "attempts-exhausted" || scenario == "restart-during-trial") {
|
||||
message("No confirmation: simulate a reset caused by power loss or watchdog");
|
||||
reset();
|
||||
}
|
||||
if (scenario == "attempts-exhausted") {
|
||||
message("B still cannot reach health checks; simulate another reset");
|
||||
reset();
|
||||
device.check(true);
|
||||
show("Recovery checks pass");
|
||||
} else if (scenario == "health-failure" || scenario == "recovery-failure") {
|
||||
device.check(false);
|
||||
show("B health checks fail");
|
||||
reset();
|
||||
device.check(scenario != "recovery-failure");
|
||||
show("Recovery checks complete");
|
||||
} else {
|
||||
device.check(true);
|
||||
show("B health checks pass; confirmation saved");
|
||||
reset();
|
||||
}
|
||||
return device;
|
||||
}
|
||||
} // namespace ota
|
||||
}// namespace ota
|
||||
@@ -8,61 +8,69 @@
|
||||
|
||||
namespace ota {
|
||||
|
||||
enum class Phase {
|
||||
AConfirmed, PreparingB, BPrepared, TrialPending, TrialRunning,
|
||||
BConfirmed, RecoveryPending, RecoveryRunning, RecoveryChecked,
|
||||
SeparateRecoveryRequired
|
||||
};
|
||||
enum class Phase {
|
||||
AConfirmed,
|
||||
PreparingB,
|
||||
BPrepared,
|
||||
TrialPending,
|
||||
TrialRunning,
|
||||
BConfirmed,
|
||||
RecoveryPending,
|
||||
RecoveryRunning,
|
||||
RecoveryChecked,
|
||||
SeparateRecoveryRequired
|
||||
};
|
||||
|
||||
enum class Slot { A, B };
|
||||
enum class Slot { A,
|
||||
B };
|
||||
|
||||
const char* name(Phase phase);
|
||||
const char* name(Slot slot);
|
||||
const char *name(Phase phase);
|
||||
const char *name(Slot slot);
|
||||
|
||||
struct State {
|
||||
Phase phase = Phase::AConfirmed;
|
||||
Slot normal = Slot::A;
|
||||
std::optional<Slot> trial;
|
||||
int attempts_left = 0;
|
||||
bool a_ready = true;
|
||||
bool b_ready = false;
|
||||
std::string outcome;
|
||||
};
|
||||
struct State {
|
||||
Phase phase = Phase::AConfirmed;
|
||||
Slot normal = Slot::A;
|
||||
std::optional<Slot> trial;
|
||||
int attempts_left = 0;
|
||||
bool a_ready = true;
|
||||
bool b_ready = false;
|
||||
std::string outcome;
|
||||
};
|
||||
|
||||
class Device {
|
||||
public:
|
||||
explicit Device(const std::filesystem::path& directory);
|
||||
const State& state() const { return state_; }
|
||||
std::optional<Slot> running() const { return running_; }
|
||||
void prepare(bool complete = true);
|
||||
void request_trial(int attempts = 2);
|
||||
Slot boot();
|
||||
void check(bool healthy);
|
||||
class Device {
|
||||
public:
|
||||
explicit Device(const std::filesystem::path &directory);
|
||||
const State &state() const { return state_; }
|
||||
std::optional<Slot> running() const { return running_; }
|
||||
void prepare(bool complete = true);
|
||||
void request_trial(int attempts = 2);
|
||||
Slot boot();
|
||||
void check(bool healthy);
|
||||
|
||||
private:
|
||||
void save() const;
|
||||
State state_;
|
||||
std::filesystem::path path_;
|
||||
// Volatile: reconstructing a Device requires another boot before checks.
|
||||
std::optional<Slot> running_;
|
||||
};
|
||||
private:
|
||||
void save() const;
|
||||
State state_;
|
||||
std::filesystem::path path_;
|
||||
// Volatile: reconstructing a Device requires another boot before checks.
|
||||
std::optional<Slot> running_;
|
||||
};
|
||||
|
||||
// Own only a freshly created directory and clean it up at scope exit.
|
||||
class TemporaryDirectory {
|
||||
public:
|
||||
TemporaryDirectory();
|
||||
~TemporaryDirectory();
|
||||
TemporaryDirectory(const TemporaryDirectory&) = delete;
|
||||
TemporaryDirectory& operator=(const TemporaryDirectory&) = delete;
|
||||
const std::filesystem::path& path() const { return path_; }
|
||||
// Own only a freshly created directory and clean it up at scope exit.
|
||||
class TemporaryDirectory {
|
||||
public:
|
||||
TemporaryDirectory();
|
||||
~TemporaryDirectory();
|
||||
TemporaryDirectory(const TemporaryDirectory &) = delete;
|
||||
TemporaryDirectory &operator=(const TemporaryDirectory &) = delete;
|
||||
const std::filesystem::path &path() const { return path_; }
|
||||
|
||||
private:
|
||||
std::filesystem::path path_;
|
||||
};
|
||||
private:
|
||||
std::filesystem::path path_;
|
||||
};
|
||||
|
||||
const std::vector<std::string>& scenarios();
|
||||
Device run_scenario(const std::string& scenario,
|
||||
const std::filesystem::path& directory,
|
||||
std::ostream* output = nullptr);
|
||||
const std::vector<std::string> &scenarios();
|
||||
Device run_scenario(const std::string &scenario,
|
||||
const std::filesystem::path &directory,
|
||||
std::ostream *output = nullptr);
|
||||
|
||||
} // namespace ota
|
||||
}// namespace ota
|
||||
@@ -6,108 +6,107 @@
|
||||
#include <string>
|
||||
|
||||
namespace {
|
||||
void expect(bool condition, const char* message) {
|
||||
if (!condition) throw std::runtime_error(message);
|
||||
}
|
||||
|
||||
template <class Operation>
|
||||
void expect_rejected(Operation operation) {
|
||||
try {
|
||||
operation();
|
||||
} catch (const std::logic_error&) {
|
||||
return;
|
||||
void expect(bool condition, const char *message) {
|
||||
if (!condition) throw std::runtime_error(message);
|
||||
}
|
||||
throw std::runtime_error("Invalid operation was accepted");
|
||||
}
|
||||
|
||||
void scenario_outcomes() {
|
||||
struct Expected {
|
||||
const char* scenario;
|
||||
ota::Phase phase;
|
||||
ota::Slot normal;
|
||||
const char* outcome;
|
||||
};
|
||||
const Expected expected[] = {
|
||||
{"success", ota::Phase::BConfirmed, ota::Slot::B, "B confirmed"},
|
||||
{"health-failure", ota::Phase::RecoveryChecked, ota::Slot::A, "Recovered to A"},
|
||||
{"attempts-exhausted", ota::Phase::RecoveryChecked, ota::Slot::A, "Recovered to A"},
|
||||
{"restart-during-trial", ota::Phase::BConfirmed, ota::Slot::B, "B confirmed"},
|
||||
{"interrupted-preparation", ota::Phase::PreparingB, ota::Slot::A, ""},
|
||||
{"recovery-failure", ota::Phase::SeparateRecoveryRequired, ota::Slot::A,
|
||||
"Recovery A failed"}
|
||||
};
|
||||
for (const auto& result : expected) {
|
||||
ota::TemporaryDirectory directory;
|
||||
auto device = ota::run_scenario(result.scenario, directory.path());
|
||||
const ota::Device restored(directory.path());
|
||||
const auto& state = restored.state();
|
||||
expect(state.phase == result.phase, "Unexpected saved phase");
|
||||
expect(state.normal == result.normal, "Unexpected normal slot");
|
||||
expect(state.outcome == result.outcome, "Unexpected saved outcome");
|
||||
expect(state.a_ready, "The retained slot was modified");
|
||||
if (state.phase == ota::Phase::SeparateRecoveryRequired) {
|
||||
expect_rejected([&] { device.boot(); });
|
||||
template<class Operation>
|
||||
void expect_rejected(Operation operation) {
|
||||
try {
|
||||
operation();
|
||||
} catch (const std::logic_error &) {
|
||||
return;
|
||||
}
|
||||
throw std::runtime_error("Invalid operation was accepted");
|
||||
}
|
||||
|
||||
void scenario_outcomes() {
|
||||
struct Expected {
|
||||
const char *scenario;
|
||||
ota::Phase phase;
|
||||
ota::Slot normal;
|
||||
const char *outcome;
|
||||
};
|
||||
const Expected expected[] = {
|
||||
{"success", ota::Phase::BConfirmed, ota::Slot::B, "B confirmed"},
|
||||
{"health-failure", ota::Phase::RecoveryChecked, ota::Slot::A, "Recovered to A"},
|
||||
{"attempts-exhausted", ota::Phase::RecoveryChecked, ota::Slot::A, "Recovered to A"},
|
||||
{"restart-during-trial", ota::Phase::BConfirmed, ota::Slot::B, "B confirmed"},
|
||||
{"interrupted-preparation", ota::Phase::PreparingB, ota::Slot::A, ""},
|
||||
{"recovery-failure", ota::Phase::SeparateRecoveryRequired, ota::Slot::A,
|
||||
"Recovery A failed"}};
|
||||
for (const auto &result: expected) {
|
||||
ota::TemporaryDirectory directory;
|
||||
auto device = ota::run_scenario(result.scenario, directory.path());
|
||||
const ota::Device restored(directory.path());
|
||||
const auto &state = restored.state();
|
||||
expect(state.phase == result.phase, "Unexpected saved phase");
|
||||
expect(state.normal == result.normal, "Unexpected normal slot");
|
||||
expect(state.outcome == result.outcome, "Unexpected saved outcome");
|
||||
expect(state.a_ready, "The retained slot was modified");
|
||||
if (state.phase == ota::Phase::SeparateRecoveryRequired) {
|
||||
expect_rejected([&] { device.boot(); });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void saved_attempts() {
|
||||
ota::TemporaryDirectory directory;
|
||||
ota::Device device(directory.path());
|
||||
device.prepare();
|
||||
device.request_trial(2);
|
||||
for (int left : {1, 0}) {
|
||||
void saved_attempts() {
|
||||
ota::TemporaryDirectory directory;
|
||||
ota::Device device(directory.path());
|
||||
device.prepare();
|
||||
device.request_trial(2);
|
||||
for (int left: {1, 0}) {
|
||||
device = ota::Device(directory.path());
|
||||
expect(device.boot() == ota::Slot::B, "Expected trial boot of B");
|
||||
const ota::Device restored(directory.path());
|
||||
expect(restored.state().attempts_left == left, "Attempt was not saved before boot");
|
||||
expect(restored.state().normal == ota::Slot::A, "B was confirmed prematurely");
|
||||
}
|
||||
device = ota::Device(directory.path());
|
||||
expect(device.boot() == ota::Slot::B, "Expected trial boot of B");
|
||||
const ota::Device restored(directory.path());
|
||||
expect(restored.state().attempts_left == left, "Attempt was not saved before boot");
|
||||
expect(restored.state().normal == ota::Slot::A, "B was confirmed prematurely");
|
||||
expect(device.boot() == ota::Slot::A, "Exhausted trial did not return to A");
|
||||
expect(device.state().phase == ota::Phase::RecoveryRunning, "Recovery checks skipped");
|
||||
}
|
||||
device = ota::Device(directory.path());
|
||||
expect(device.boot() == ota::Slot::A, "Exhausted trial did not return to A");
|
||||
expect(device.state().phase == ota::Phase::RecoveryRunning, "Recovery checks skipped");
|
||||
}
|
||||
|
||||
void incomplete_slot() {
|
||||
ota::TemporaryDirectory directory;
|
||||
ota::Device device(directory.path());
|
||||
device.prepare(false);
|
||||
device = ota::Device(directory.path());
|
||||
expect(!device.state().b_ready, "Incomplete B was marked ready");
|
||||
expect_rejected([&] { device.request_trial(); });
|
||||
expect(device.boot() == ota::Slot::A, "Incomplete B was selected");
|
||||
}
|
||||
|
||||
void confirmation_requires_boot() {
|
||||
ota::TemporaryDirectory directory;
|
||||
ota::Device device(directory.path());
|
||||
device.prepare();
|
||||
expect_rejected([&] { device.request_trial(0); });
|
||||
expect_rejected([&] { device.request_trial(-1); });
|
||||
device.request_trial();
|
||||
expect_rejected([&] { device.check(true); });
|
||||
device.boot();
|
||||
ota::Device restored(directory.path());
|
||||
expect(!restored.running(), "Running slot survived reconstruction");
|
||||
expect_rejected([&] { restored.check(true); });
|
||||
device.check(true);
|
||||
expect_rejected([&] { device.check(false); });
|
||||
}
|
||||
|
||||
void unreadable_state() {
|
||||
ota::TemporaryDirectory directory;
|
||||
{
|
||||
std::ofstream output(directory.path() / "boot-state.txt");
|
||||
output << "incomplete record\n";
|
||||
void incomplete_slot() {
|
||||
ota::TemporaryDirectory directory;
|
||||
ota::Device device(directory.path());
|
||||
device.prepare(false);
|
||||
device = ota::Device(directory.path());
|
||||
expect(!device.state().b_ready, "Incomplete B was marked ready");
|
||||
expect_rejected([&] { device.request_trial(); });
|
||||
expect(device.boot() == ota::Slot::A, "Incomplete B was selected");
|
||||
}
|
||||
try {
|
||||
const ota::Device device(directory.path());
|
||||
} catch (const std::runtime_error&) {
|
||||
return;
|
||||
|
||||
void confirmation_requires_boot() {
|
||||
ota::TemporaryDirectory directory;
|
||||
ota::Device device(directory.path());
|
||||
device.prepare();
|
||||
expect_rejected([&] { device.request_trial(0); });
|
||||
expect_rejected([&] { device.request_trial(-1); });
|
||||
device.request_trial();
|
||||
expect_rejected([&] { device.check(true); });
|
||||
device.boot();
|
||||
ota::Device restored(directory.path());
|
||||
expect(!restored.running(), "Running slot survived reconstruction");
|
||||
expect_rejected([&] { restored.check(true); });
|
||||
device.check(true);
|
||||
expect_rejected([&] { device.check(false); });
|
||||
}
|
||||
throw std::runtime_error("Unreadable state was accepted");
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void unreadable_state() {
|
||||
ota::TemporaryDirectory directory;
|
||||
{
|
||||
std::ofstream output(directory.path() / "boot-state.txt");
|
||||
output << "incomplete record\n";
|
||||
}
|
||||
try {
|
||||
const ota::Device device(directory.path());
|
||||
} catch (const std::runtime_error &) {
|
||||
return;
|
||||
}
|
||||
throw std::runtime_error("Unreadable state was accepted");
|
||||
}
|
||||
}// namespace
|
||||
|
||||
int main() {
|
||||
try {
|
||||
@@ -117,7 +116,7 @@ int main() {
|
||||
confirmation_requires_boot();
|
||||
unreadable_state();
|
||||
std::cout << "All recovery invariant checks passed\n";
|
||||
} catch (const std::exception& error) {
|
||||
} catch (const std::exception &error) {
|
||||
std::cerr << "Test failed: " << error.what() << '\n';
|
||||
return 1;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
cmake_minimum_required(VERSION 3.20)
|
||||
project(interrupted_installation LANGUAGES CXX)
|
||||
set(CMAKE_CXX_EXTENSIONS OFF)
|
||||
|
||||
add_library(installation-model simulation.cpp)
|
||||
target_compile_features(installation-model PUBLIC cxx_std_17)
|
||||
if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
|
||||
target_compile_options(installation-model PRIVATE -Wall -Wextra -Wpedantic)
|
||||
endif()
|
||||
|
||||
add_executable(installation-simulate main.cpp)
|
||||
target_link_libraries(installation-simulate PRIVATE installation-model)
|
||||
|
||||
include(CTest)
|
||||
if(BUILD_TESTING)
|
||||
add_executable(installation-tests tests.cpp)
|
||||
target_link_libraries(installation-tests PRIVATE installation-model)
|
||||
add_test(NAME installation-invariants COMMAND installation-tests)
|
||||
add_test(NAME all-scenarios COMMAND installation-simulate all)
|
||||
add_test(NAME cli-help COMMAND installation-simulate --help)
|
||||
add_test(NAME cli-invalid COMMAND installation-simulate invalid)
|
||||
set_tests_properties(cli-invalid PROPERTIES WILL_FAIL TRUE)
|
||||
endif()
|
||||
@@ -0,0 +1,49 @@
|
||||
#include "simulation.hpp"
|
||||
|
||||
#include <exception>
|
||||
#include <iostream>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
|
||||
namespace model = ota::installation;
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
if (argc == 2 && std::string(argv[1]) == "--help") {
|
||||
std::cout << "Usage: installation-simulate [all|--help]\n";
|
||||
return 0;
|
||||
}
|
||||
if (argc > 2 || (argc == 2 && std::string(argv[1]) != "all")) {
|
||||
std::cerr << "Usage: installation-simulate [all|--help]\n";
|
||||
return 1;
|
||||
}
|
||||
try {
|
||||
for (int cut = 0; cut <= model::installation_steps; ++cut) {
|
||||
model::Storage storage;
|
||||
{
|
||||
model::Installer installer(storage);
|
||||
for (int step = 0; step < cut; ++step) model::perform_step(installer, step);
|
||||
}
|
||||
model::Installer restarted(storage);
|
||||
const auto action = restarted.recover();
|
||||
std::cout << "Reset after " << cut << " steps: boot=" << model::boot_choice(storage.boot)
|
||||
<< "; " << model::name(action) << '\n';
|
||||
}
|
||||
for (const auto result: {model::WriteResult::OldRecord, model::WriteResult::NewRecord, model::WriteResult::InvalidRecord}) {
|
||||
model::Storage storage;
|
||||
{
|
||||
model::Installer installer(storage);
|
||||
for (int step = 0; step < 5; ++step) model::perform_step(installer, step);
|
||||
try {
|
||||
installer.request_trial(result);
|
||||
} catch (const std::runtime_error &error) {
|
||||
std::cout << error.what() << '\n';
|
||||
}
|
||||
}
|
||||
model::Installer restarted(storage);
|
||||
std::cout << "Boot-state write outcome: " << model::name(restarted.recover()) << '\n';
|
||||
}
|
||||
} catch (const std::exception &error) {
|
||||
std::cerr << error.what() << '\n';
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
# Interrupted A/B installation model
|
||||
|
||||
This C++17 example follows [chapter 06](../../../docs/06-atomic-updates.md), [ADR-018](../../../adr/ADR-018-prepare-and-verify-before-activation.md), and [ADR-019](../../../adr/ADR-019-recover-interrupted-updates-from-durable-state.md). It models preparation and activation of B while confirmed system A remains available. It does not access devices or install files.
|
||||
|
||||
Build and run from the repository root with CMake 3.20 or later and a C++17 compiler. No third-party libraries are required.
|
||||
|
||||
```sh
|
||||
cmake -S examples/simulations/interrupted-installation -B /tmp/ota-installation-build -DCMAKE_BUILD_TYPE=Debug
|
||||
cmake --build /tmp/ota-installation-build
|
||||
/tmp/ota-installation-build/installation-simulate all
|
||||
ctest --test-dir /tmp/ota-installation-build --output-on-failure
|
||||
```
|
||||
|
||||
With no argument, the runner also runs all scenarios. `--help` prints usage. Unknown arguments return a nonzero exit status.
|
||||
|
||||
## Reset Points
|
||||
|
||||
Each run starts from fresh storage. It performs the first N steps, destroys the installer, and creates another installer using the same simulated storage.
|
||||
|
||||
| Steps completed before reset | Last operation | Recovery action |
|
||||
|---|---|---|
|
||||
| 0 | None | Keep A; start preparation. |
|
||||
| 1 | Save release and target intent | Keep A; restart preparation. |
|
||||
| 2 | Write part of B | Keep A; restart preparation. |
|
||||
| 3 | Complete writing B | Verify again; save prepared state; wait for activation policy. |
|
||||
| 4 | Verify B | Repeat verification; save prepared state; wait for activation policy. |
|
||||
| 5 | Save prepared state | Verify again; wait for activation policy. |
|
||||
| 6 | Save trial request | Preserve trial and attempts; repair old progress. |
|
||||
| 7 | Save trial-requested progress | Preserve trial and attempts. |
|
||||
|
||||
The runner also supplies three possible results of an interrupted boot-record write: retain the old record, retain the new record, or find no valid record. The last case requires separate recovery. These are assumed outcomes, not a redundant-record algorithm.
|
||||
|
||||
## Storage and Ownership
|
||||
|
||||
`Storage` represents persistent bytes and records in memory. Each modeled storage operation is already complete and durable when it returns. `Installer` holds a temporary verification result. Recreating it clears that result, so a new activation requires fresh verification.
|
||||
|
||||
Boot state owns selection and the trial limit. OTA progress can be old. When a trial request already exists, recovery repairs progress without rewriting B or renewing attempts. Recovery with A selected checks both the release identity and installed contents before accepting B as prepared.
|
||||
|
||||
Only one installer is active. A is initially confirmed; there is no earlier pending trial. B is writable only while boot state selects A. A remains unchanged. `recover()` does not activate B automatically; the caller still owns activation policy.
|
||||
|
||||
## Checks and Limits
|
||||
|
||||
[tests.cpp](tests.cpp) checks all reset points and their continuation, activation guards, loss of temporary verification, damaged contents, wrong release identity, all supplied boot-write outcomes, and repeated recovery without renewing attempts. Checks remain active in Release builds.
|
||||
|
||||
The model compares strings with fixed expected bytes. It has no signatures, real files, flushing, redundant records, or interruption inside a write. It does not model storage wear, concurrent installers, data migration, running processes, or later releases. Direct changes to `Storage` in tests represent faults between installer instances.
|
||||
|
||||
The boot record is simplified to A selected, B trial, or invalid. Attempts are recorded but not consumed here. The printed boot choice is the next lifecycle destination, not a real bootloader decision. Confirmation, attempt exhaustion, and recovery health checks remain in the separate [A/B trial boot example](../ab-trial-boot/readme.md); the programs are not integrated.
|
||||
|
||||
Real storage needs a defined durability contract and failure-injection tests on the target platform. This model checks ordering and recovery decisions only.
|
||||
|
||||
## Sources and Diagrams
|
||||
|
||||
- [simulation.hpp](simulation.hpp) and [simulation.cpp](simulation.cpp): storage, installer, and recovery rules.
|
||||
- [main.cpp](main.cpp): scenario output.
|
||||
- [CMakeLists.txt](CMakeLists.txt): build and CTest integration.
|
||||
- [Preparation and activation sequence](../../../diagrams/plantuml/ab-installation-sequence.puml).
|
||||
- [Installation recovery states](../../../diagrams/plantuml/ab-installation-recovery.puml).
|
||||
|
||||
Format C++ sources with the repository's root `.clang-format`.
|
||||
@@ -0,0 +1,154 @@
|
||||
#include "simulation.hpp"
|
||||
|
||||
#include <stdexcept>
|
||||
|
||||
namespace ota::installation {
|
||||
|
||||
const std::string &expected_contents() {
|
||||
static const std::string contents = "release-2:kernel+device-tree+rootfs";
|
||||
return contents;
|
||||
}
|
||||
|
||||
const char *name(RecoveryAction action) {
|
||||
switch (action) {
|
||||
case RecoveryAction::RestartPreparation:
|
||||
return "restart preparation; keep A";
|
||||
case RecoveryAction::ReadyToActivate:
|
||||
return "B verified again; wait for activation policy";
|
||||
case RecoveryAction::PreserveTrial:
|
||||
return "preserve saved B trial and attempts";
|
||||
case RecoveryAction::SeparateRecovery:
|
||||
return "stop; separate recovery required";
|
||||
}
|
||||
throw std::logic_error("Unknown recovery action");
|
||||
}
|
||||
|
||||
const char *boot_choice(BootRecord record) {
|
||||
switch (record) {
|
||||
case BootRecord::ASelected:
|
||||
return "A";
|
||||
case BootRecord::BTrial:
|
||||
return "B trial (attempt handling is outside this model)";
|
||||
case BootRecord::Invalid:
|
||||
return "separate recovery";
|
||||
}
|
||||
throw std::logic_error("Unknown boot record");
|
||||
}
|
||||
|
||||
void Installer::require_writable() const {
|
||||
if (storage_.boot != BootRecord::ASelected) {
|
||||
throw std::logic_error("B is selected or boot state is invalid");
|
||||
}
|
||||
}
|
||||
|
||||
void Installer::begin() {
|
||||
require_writable();
|
||||
verified_ = false;
|
||||
storage_.release = "release-2";
|
||||
storage_.progress = Progress::Writing;
|
||||
}
|
||||
|
||||
void Installer::write_partial() {
|
||||
require_writable();
|
||||
if (storage_.progress != Progress::Writing || storage_.release != "release-2") {
|
||||
throw std::logic_error("Save matching intent before writing");
|
||||
}
|
||||
verified_ = false;
|
||||
storage_.b_contents = expected_contents().substr(0, 8);
|
||||
}
|
||||
|
||||
void Installer::finish_write() {
|
||||
require_writable();
|
||||
if (storage_.progress != Progress::Writing || storage_.release != "release-2") {
|
||||
throw std::logic_error("Save matching intent before writing");
|
||||
}
|
||||
verified_ = false;
|
||||
storage_.b_contents = expected_contents();
|
||||
}
|
||||
|
||||
void Installer::verify() {
|
||||
require_writable();
|
||||
verified_ = false;
|
||||
if (storage_.release != "release-2" || storage_.b_contents != expected_contents()) {
|
||||
throw std::runtime_error("Installed release does not match expected contents");
|
||||
}
|
||||
verified_ = true;
|
||||
}
|
||||
|
||||
void Installer::save_prepared() {
|
||||
require_writable();
|
||||
if (!verified_) throw std::logic_error("Verify B before saving prepared state");
|
||||
storage_.progress = Progress::Prepared;
|
||||
}
|
||||
|
||||
void Installer::request_trial(WriteResult result) {
|
||||
require_writable();
|
||||
if (!verified_ || storage_.progress != Progress::Prepared) {
|
||||
throw std::logic_error("Verify B and save prepared state before activation");
|
||||
}
|
||||
// Supplied outcomes stand for the storage contract of boot control.
|
||||
// OldRecord leaves all fields intact; NewRecord saves one complete record.
|
||||
if (result == WriteResult::OldRecord) {
|
||||
throw std::runtime_error("Interrupted request; old record remains");
|
||||
}
|
||||
if (result == WriteResult::InvalidRecord) {
|
||||
storage_.boot = BootRecord::Invalid;
|
||||
throw std::runtime_error("Interrupted request; no valid record remains");
|
||||
}
|
||||
storage_.boot = BootRecord::BTrial;
|
||||
storage_.attempts_left = 2;
|
||||
}
|
||||
|
||||
void Installer::save_requested() {
|
||||
if (storage_.boot != BootRecord::BTrial) {
|
||||
throw std::logic_error("No saved B trial request");
|
||||
}
|
||||
storage_.progress = Progress::TrialRequested;
|
||||
}
|
||||
|
||||
RecoveryAction Installer::recover() {
|
||||
verified_ = false;
|
||||
// Boot state takes priority over progress. Never write B or renew attempts
|
||||
// when a trial is already selected. Invalid boot state is not confirmation.
|
||||
if (storage_.boot == BootRecord::Invalid) return RecoveryAction::SeparateRecovery;
|
||||
if (storage_.boot == BootRecord::BTrial) {
|
||||
save_requested();
|
||||
return RecoveryAction::PreserveTrial;
|
||||
}
|
||||
if (storage_.release != "release-2" || storage_.b_contents != expected_contents()) {
|
||||
return RecoveryAction::RestartPreparation;
|
||||
}
|
||||
verify();
|
||||
save_prepared();
|
||||
return RecoveryAction::ReadyToActivate;
|
||||
}
|
||||
|
||||
void perform_step(Installer &installer, int step) {
|
||||
switch (step) {
|
||||
case 0:
|
||||
installer.begin();
|
||||
break;
|
||||
case 1:
|
||||
installer.write_partial();
|
||||
break;
|
||||
case 2:
|
||||
installer.finish_write();
|
||||
break;
|
||||
case 3:
|
||||
installer.verify();
|
||||
break;
|
||||
case 4:
|
||||
installer.save_prepared();
|
||||
break;
|
||||
case 5:
|
||||
installer.request_trial();
|
||||
break;
|
||||
case 6:
|
||||
installer.save_requested();
|
||||
break;
|
||||
default:
|
||||
throw std::logic_error("Unknown installation step");
|
||||
}
|
||||
}
|
||||
|
||||
}// namespace ota::installation
|
||||
@@ -0,0 +1,60 @@
|
||||
#pragma once
|
||||
|
||||
#include <string>
|
||||
|
||||
namespace ota::installation {
|
||||
|
||||
enum class Progress { Idle,
|
||||
Writing,
|
||||
Prepared,
|
||||
TrialRequested };
|
||||
enum class BootRecord { ASelected,
|
||||
BTrial,
|
||||
Invalid };
|
||||
enum class WriteResult { OldRecord,
|
||||
NewRecord,
|
||||
InvalidRecord };
|
||||
enum class RecoveryAction { RestartPreparation,
|
||||
ReadyToActivate,
|
||||
PreserveTrial,
|
||||
SeparateRecovery };
|
||||
|
||||
// This object stands for persistent storage. Each assignment is a completed
|
||||
// durable operation in the model, not a real storage implementation.
|
||||
struct Storage {
|
||||
std::string a_contents = "release-1";
|
||||
std::string b_contents;
|
||||
std::string release;
|
||||
Progress progress = Progress::Idle;
|
||||
BootRecord boot = BootRecord::ASelected;
|
||||
int attempts_left = 0;
|
||||
};
|
||||
|
||||
const std::string &expected_contents();
|
||||
const char *name(RecoveryAction action);
|
||||
const char *boot_choice(BootRecord record);
|
||||
|
||||
class Installer {
|
||||
public:
|
||||
explicit Installer(Storage &storage) : storage_(storage) {}
|
||||
void begin();
|
||||
void write_partial();
|
||||
void finish_write();
|
||||
void verify();
|
||||
void save_prepared();
|
||||
void request_trial(WriteResult result = WriteResult::NewRecord);
|
||||
void save_requested();
|
||||
RecoveryAction recover();
|
||||
|
||||
private:
|
||||
void require_writable() const;
|
||||
Storage &storage_;
|
||||
// Lost when the installer is reconstructed after a simulated reset.
|
||||
bool verified_ = false;
|
||||
};
|
||||
|
||||
// Ordered operations; cut points are before step 0 and after each step.
|
||||
constexpr int installation_steps = 7;
|
||||
void perform_step(Installer &installer, int step);
|
||||
|
||||
}// namespace ota::installation
|
||||
@@ -0,0 +1,148 @@
|
||||
#include "simulation.hpp"
|
||||
|
||||
#include <iostream>
|
||||
#include <stdexcept>
|
||||
|
||||
namespace model = ota::installation;
|
||||
|
||||
namespace {
|
||||
|
||||
void expect(bool condition, const char *message) {
|
||||
if (!condition) throw std::runtime_error(message);
|
||||
}
|
||||
|
||||
template<class Operation>
|
||||
void expect_rejected(Operation operation) {
|
||||
try {
|
||||
operation();
|
||||
} catch (const std::logic_error &) {
|
||||
return;
|
||||
}
|
||||
throw std::runtime_error("Unsafe operation accepted");
|
||||
}
|
||||
|
||||
void cut_points() {
|
||||
for (int cut = 0; cut <= model::installation_steps; ++cut) {
|
||||
model::Storage storage;
|
||||
{
|
||||
model::Installer installer(storage);
|
||||
for (int step = 0; step < cut; ++step) model::perform_step(installer, step);
|
||||
}
|
||||
model::Installer restarted(storage);
|
||||
const auto action = restarted.recover();
|
||||
expect(storage.a_contents == "release-1", "A changed during installation");
|
||||
if (cut < 3) {
|
||||
expect(action == model::RecoveryAction::RestartPreparation, "Incomplete B accepted");
|
||||
expect(storage.boot == model::BootRecord::ASelected, "Incomplete B selected");
|
||||
expect_rejected([&] { restarted.request_trial(); });
|
||||
// Exercise the documented restart path, not only its decision.
|
||||
for (int step = 0; step < model::installation_steps; ++step) model::perform_step(restarted, step);
|
||||
} else if (cut < 6) {
|
||||
expect(action == model::RecoveryAction::ReadyToActivate, "Complete B not reverified");
|
||||
expect(storage.boot == model::BootRecord::ASelected, "Recovery activated B without policy");
|
||||
restarted.request_trial();
|
||||
restarted.save_requested();
|
||||
} else {
|
||||
expect(action == model::RecoveryAction::PreserveTrial, "Saved trial lost");
|
||||
expect(storage.progress == model::Progress::TrialRequested, "Old progress not repaired");
|
||||
}
|
||||
expect(storage.boot == model::BootRecord::BTrial, "Continuation did not reach trial");
|
||||
expect(storage.b_contents == model::expected_contents(), "Selected B is incomplete");
|
||||
expect(storage.a_contents == "release-1", "Continuation changed A");
|
||||
storage.attempts_left = 1;// Boot control has already consumed an attempt.
|
||||
model::Installer second_restart(storage);
|
||||
expect(second_restart.recover() == model::RecoveryAction::PreserveTrial, "Repeated recovery lost trial");
|
||||
expect(storage.attempts_left == 1, "Recovery renewed trial attempts");
|
||||
expect_rejected([&] { second_restart.begin(); });
|
||||
expect_rejected([&] { second_restart.write_partial(); });
|
||||
expect_rejected([&] { second_restart.finish_write(); });
|
||||
expect_rejected([&] { second_restart.request_trial(); });
|
||||
}
|
||||
}
|
||||
|
||||
void activation_guards() {
|
||||
model::Storage storage;
|
||||
model::Installer installer(storage);
|
||||
expect_rejected([&] { installer.finish_write(); });
|
||||
expect_rejected([&] { installer.save_requested(); });
|
||||
installer.begin();
|
||||
installer.finish_write();
|
||||
expect_rejected([&] { installer.save_prepared(); });
|
||||
installer.verify();
|
||||
expect_rejected([&] { installer.request_trial(); });
|
||||
installer.save_prepared();
|
||||
model::Installer restarted(storage);
|
||||
expect_rejected([&] { restarted.request_trial(); });
|
||||
expect(restarted.recover() == model::RecoveryAction::ReadyToActivate, "Fresh verification failed");
|
||||
restarted.request_trial();
|
||||
}
|
||||
|
||||
void bad_contents_and_identity() {
|
||||
for (bool wrong_release: {false, true}) {
|
||||
model::Storage storage;
|
||||
{
|
||||
model::Installer installer(storage);
|
||||
for (int step = 0; step < 5; ++step) model::perform_step(installer, step);
|
||||
}
|
||||
if (wrong_release) storage.release = "another-release";
|
||||
else
|
||||
storage.b_contents = "damaged";
|
||||
model::Installer restarted(storage);
|
||||
expect(restarted.recover() == model::RecoveryAction::RestartPreparation, "Stale prepared record trusted");
|
||||
try {
|
||||
restarted.verify();
|
||||
throw std::logic_error("Bad release passed verification");
|
||||
} catch (const std::runtime_error &) {
|
||||
// Expected content or identity failure; logic_error escapes.
|
||||
}
|
||||
expect_rejected([&] { restarted.request_trial(); });
|
||||
expect(storage.boot == model::BootRecord::ASelected, "Bad release selected");
|
||||
}
|
||||
}
|
||||
|
||||
void interrupted_boot_write() {
|
||||
for (const auto result: {model::WriteResult::OldRecord, model::WriteResult::NewRecord, model::WriteResult::InvalidRecord}) {
|
||||
model::Storage storage;
|
||||
{
|
||||
model::Installer installer(storage);
|
||||
for (int step = 0; step < 5; ++step) model::perform_step(installer, step);
|
||||
bool failed = false;
|
||||
try {
|
||||
installer.request_trial(result);
|
||||
} catch (const std::runtime_error &) {
|
||||
failed = true;
|
||||
}
|
||||
expect(failed == (result != model::WriteResult::NewRecord), "Wrong request result");
|
||||
}
|
||||
model::Installer restarted(storage);
|
||||
const auto action = restarted.recover();
|
||||
if (result == model::WriteResult::OldRecord) {
|
||||
expect(action == model::RecoveryAction::ReadyToActivate, "Old boot record not retained");
|
||||
expect(storage.attempts_left == 0, "Old record partly changed");
|
||||
} else if (result == model::WriteResult::NewRecord) {
|
||||
expect(action == model::RecoveryAction::PreserveTrial, "New boot record not retained");
|
||||
expect(storage.attempts_left == 2, "New trial limit lost");
|
||||
} else {
|
||||
expect(action == model::RecoveryAction::SeparateRecovery, "Invalid boot state accepted");
|
||||
expect(storage.progress == model::Progress::Prepared, "Invalid boot state changed progress");
|
||||
expect_rejected([&] { restarted.begin(); });
|
||||
expect_rejected([&] { restarted.request_trial(); });
|
||||
}
|
||||
expect(storage.a_contents == "release-1", "Boot record failure changed A");
|
||||
}
|
||||
}
|
||||
|
||||
}// namespace
|
||||
|
||||
int main() {
|
||||
try {
|
||||
cut_points();
|
||||
activation_guards();
|
||||
bad_contents_and_identity();
|
||||
interrupted_boot_write();
|
||||
std::cout << "All installation recovery checks passed\n";
|
||||
} catch (const std::exception &error) {
|
||||
std::cerr << "Test failed: " << error.what() << '\n';
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
A practical reference design for reliable and secure over-the-air updates on embedded Linux.
|
||||
|
||||
This repository is intended as an educational reference design. It demonstrates one possible approach to designing and implementing a reliable OTA update system for embedded Linux.
|
||||
This repository demonstrates one possible approach to designing and implementing a reliable OTA update system for embedded Linux.
|
||||
|
||||
The project focuses on the engineering decisions behind OTA systems, including:
|
||||
|
||||
@@ -22,7 +22,7 @@ This repository is not intended to be a production-ready framework. Instead, it
|
||||
|
||||
The repository is being developed incrementally as a structured series of articles, diagrams, architecture decisions, and practical examples.
|
||||
|
||||
Start with the [series overview](docs/00-series-overview.md). The current practical material includes [PlantUML diagrams](diagrams/readme.md) and an [A/B trial boot simulation](examples/simulations/ab-trial-boot/readme.md) with startup and recovery failure scenarios.
|
||||
Start with the [series overview](docs/00-series-overview.md). The current practical material includes [PlantUML diagrams](diagrams/readme.md), an [A/B trial boot simulation](examples/simulations/ab-trial-boot/readme.md), and an [interrupted installation model](examples/simulations/interrupted-installation/readme.md). [Chapter 06: Atomic Updates](docs/06-atomic-updates.md) connects preparation, activation, and recovery after interruption.
|
||||
|
||||
See the [changelog](CHANGELOG.md) for dated updates to the chapters, architecture decisions, diagrams, and simulations, with links to the material and related history.
|
||||
|
||||
|
||||
Reference in new issue
Block a user