next chapter

This commit is contained in:
deeaitch committed 2026-10-25 08:31:08 -04:00
1 parent 9385583955
commit c98eccc057
21 files changed
+1054 -11

No files matched your search

+11 -1
View File
@@ -4,11 +4,21 @@ Reader-facing changes to the articles, architecture decisions, diagrams, and pra
When adding or revising material, add a short entry under the relevant date with links to the affected files and any related issue or commit. When adding or revising material, add a short entry under the relevant date with links to the affected files and any related issue or commit.
## 2026-10-24
- Added [chapter 07: Rollback and Recovery](docs/07-rollback-and-recovery.md), covering recovery target eligibility, shared-data compatibility, approved restore and data loss, restart rules, bounded recovery, failed-release blocks, and the rollback window.
- Added [ADR-020: Define Rollback Eligibility and Retention](adr/ADR-020-define-rollback-eligibility-and-retention.md), [ADR-021: Preserve Data Compatibility During Trial](adr/ADR-021-preserve-data-compatibility-during-trial.md), and [ADR-022: Bound Recovery and Block Failed System Releases](adr/ADR-022-bound-recovery-and-block-failed-system-releases.md).
- Added a [C++17 rollback policy simulation](examples/simulations/rollback-policy/readme.md) with nine scenarios and CMake/CTest checks. It models completed durable transitions and supplied eligibility and health outcomes, not real storage, backup restoration, bootloader integration, or a rescue environment.
- Added PlantUML sources for [rollback policy](diagrams/plantuml/rollback-policy.puml), [recovery coordination](diagrams/plantuml/rollback-recovery-sequence.puml), and the [rollback window](diagrams/plantuml/rollback-window.puml). Added PNG previews for [rollback policy](diagrams/rendered/rollback-policy.png), [recovery coordination](diagrams/rendered/rollback-recovery-sequence.png), and the [rollback window](diagrams/rendered/rollback-window.png), rendered from those sources.
- Updated the [README](readme.md), [series overview](docs/00-series-overview.md), [chapter 06 continuation](docs/06-atomic-updates.md), and [diagram index](diagrams/readme.md). Linked the existing chapter 06 PNG previews and corrected the outdated source-only descriptions.
## 2026-10-20 ## 2026-10-20
- Added PNG previews for the [installation sequence](diagrams/rendered/ab-installation-sequence.png) and [restart decisions](diagrams/rendered/ab-installation-recovery.png) after the initial chapter 06 addition. History: [9385583](https://git.himro.ca/deeaitch/ota-reference-design/commit/9385583).
- Added [chapter 06: Atomic Updates](docs/06-atomic-updates.md), covering preparation before activation, durable state, gaps between records, restart decisions, and the limits of package recovery. - Added [chapter 06: Atomic Updates](docs/06-atomic-updates.md), covering preparation before activation, durable state, gaps between records, restart decisions, and the limits of package recovery.
- Added [ADR-018: Prepare and Verify Before Activation](adr/ADR-018-prepare-and-verify-before-activation.md) and [ADR-019: Recover Interrupted Updates From Durable State](adr/ADR-019-recover-interrupted-updates-from-durable-state.md). - Added [ADR-018: Prepare and Verify Before Activation](adr/ADR-018-prepare-and-verify-before-activation.md) and [ADR-019: Recover Interrupted Updates From Durable State](adr/ADR-019-recover-interrupted-updates-from-durable-state.md).
- Added PlantUML sources for the [installation sequence](diagrams/plantuml/ab-installation-sequence.puml) and [restart decisions](diagrams/plantuml/ab-installation-recovery.puml). No new image previews are included. - Added PlantUML sources for the [installation sequence](diagrams/plantuml/ab-installation-sequence.puml) and [restart decisions](diagrams/plantuml/ab-installation-recovery.puml). The initial chapter addition included sources only; previews were added in the follow-up above.
- Added a [C++17 interrupted installation model](examples/simulations/interrupted-installation/readme.md), with reset points, three supplied boot-record write outcomes, recovery checks, and CMake/CTest support. It models completed durable operations in memory, not real power-loss-safe storage. - Added a [C++17 interrupted installation model](examples/simulations/interrupted-installation/readme.md), with reset points, three supplied boot-record write outcomes, recovery checks, and CMake/CTest support. It models completed durable operations in memory, not real power-loss-safe storage.
- Applied the root [.clang-format](.clang-format) to the new model and the existing [A/B trial boot example](examples/simulations/ab-trial-boot/readme.md). Existing C++ changes are formatting only. - Applied the root [.clang-format](.clang-format) to the new model and the existing [A/B trial boot example](examples/simulations/ab-trial-boot/readme.md). Existing C++ changes are formatting only.
- Updated the [README](readme.md), [series overview](docs/00-series-overview.md), [chapter 05 continuation](docs/05-storage-layout.md), and [diagram index](diagrams/readme.md). - Updated the [README](readme.md), [series overview](docs/00-series-overview.md), [chapter 05 continuation](docs/05-storage-layout.md), and [diagram index](diagrams/readme.md).
@@ -0,0 +1,56 @@
# ADR-020: Define Rollback Eligibility and Retention
## Context
An A/B device retains A while trying B. Keeping A's bytes does not guarantee that A can still run: shared data, configuration, boot components, or security policy may have changed. Confirmation and later slot reuse also affect what the product can promise.
The reference design needs local recovery without downloading the old release. Boot control and Linux must understand the retained recovery state.
## Decision
Define an explicit rollback window and the conditions that make its target eligible. Retain the target release, matching boot artifacts, required dependencies, verification information, and a supported data path for that window. Record release identities and the window state outside replaceable software.
Admit a trial only when its permitted operations preserve the early-boot recovery path. The eligibility contract covers artifact validity, shared components, configuration, data, recovery-state formats, and the platform's security policy. Reassess these conditions before a supervisor initiates recovery where possible. Early boot recovery must rely on a contract established before trial, not on checks that require the failed Linux system.
Confirmation alone neither guarantees nor closes a post-confirmation rollback window. The product must state whether such a window exists. Save its closure before an incompatible change or slot reuse removes the promised recovery path. An interrupted closure may conservatively leave rollback unavailable; it must not leave a promise to use a removed target.
If eligibility cannot be established, stop ordinary rollback and use the separate recovery policy. Check essential functions after starting the retained release before reporting success.
## Alternatives Considered
### Treat Any Retained Slot as Recoverable
Simple, but ignores shared data and changes outside the slot. The slot may boot while the application remains unusable.
### Keep Every Previous Release Indefinitely
This needs more storage and still does not preserve compatibility or permission to run old software.
### Download Recovery Software When Needed
This can supplement local recovery, but cannot satisfy this design's offline requirement.
## Consequences
### Positive
- The recovery promise has an explicit lifetime and prerequisites.
- Cleanup and migrations cannot silently invalidate a retained target.
- Confirmation, data compatibility, and slot retention can be reviewed separately.
### Negative
- Retained artifacts and compatible data consume resources.
- Releases must maintain compatible recovery-state formats during the window.
- Closing the window may require an alternative recovery path.
## Applicability and Limits
Use when a product promises a return to retained software. This decision defines policy, not an artifact verifier or a durable journal implementation. Hardware failure and damaged shared boot components need separate recovery support. Security policy can prohibit an otherwise operational old release.
## Related Documents
- [Chapter 07: Rollback and Recovery](../docs/07-rollback-and-recovery.md)
- [ADR-014: Trial Boot and Confirmation](ADR-014-trial-boot-and-confirmation.md)
- [ADR-017: Keep Recovery State Outside Replaceable Software](ADR-017-keep-recovery-state-outside-replaceable-software.md)
- [ADR-021: Preserve Data Compatibility During Trial](ADR-021-preserve-data-compatibility-during-trial.md)
@@ -0,0 +1,53 @@
# ADR-021: Preserve Data Compatibility During Trial
## Context
The reference A/B design shares persistent application data and configuration between A and B. Boot control may return to A before a Linux recovery supervisor can run. A trial that makes shared data unreadable by A would invalidate that path.
This example can keep a compatible format during trial. It does not require an incompatible migration before B can pass its checks.
## Decision
Keep shared application data and configuration usable by both retained releases throughout trial. Review writes as well as reads: B's output must remain usable by A. Include interrupted application writes in the platform's data-recovery tests.
Reject or defer a trial requiring an incompatible mutation without another supported recovery path. Delay such a migration until confirmation and explicit closure of the old rollback window. If rollback remains promised after confirmation, preserve compatibility for that period too.
Save window closure before beginning an incompatible migration. The migration still needs its own interruption and failure procedure; closing the window does not make the migration safe.
## Alternatives Considered
### Separate Data Copy for B
Can isolate trial writes, but needs more space, a durable selection procedure, and a policy for merging or losing trial records. Suitable when compatible shared writes are impractical.
### Restore a Backup Before Returning to A
Can recover an older format, but requires a validated backup, interruption-safe restoration, and explicit authorization for any lost records. Early failures require a recovery environment that can restore data before A uses it. This is not assumed in the shared-data trial design.
### Always Reverse the Migration
A reverse migration may exist, but can fail or lose information. It must be implemented and tested for the specific schema; it cannot be assumed from the presence of an old image.
## Consequences
### Positive
- Boot control can return to A without first running a data restore service.
- Trial records remain available after rollback when both versions support them.
- Compatibility becomes an admission condition for a release.
### Negative
- New features may need a staged schema change.
- Compatibility tests must cover both versions' readers and writers.
- A later incompatible migration ends this rollback path and needs another recovery plan.
## Applicability and Limits
Use when retained releases share data and compatible operation during trial is feasible. This does not protect against corruption or guarantee that a migration after confirmation succeeds. Products needing incompatible trial writes should choose and test a different data recovery design explicitly.
## Related Documents
- [Chapter 07: Rollback and Recovery](../docs/07-rollback-and-recovery.md)
- [ADR-016: Separate System Software and Persistent Device Data](ADR-016-separate-system-software-and-persistent-device-data.md)
- [ADR-020: Define Rollback Eligibility and Retention](ADR-020-define-rollback-eligibility-and-retention.md)
@@ -0,0 +1,59 @@
# ADR-022: Bound Recovery and Block Failed System Releases
## Context
Returning to A can fail, and a server may offer failed B again after a successful return. Restarting a coordinator must not give either path unlimited new attempts. The device must recover locally and retain results while offline.
Boot control owns boot selection and attempt accounting. A supervisor coordinates recovery checks and keeps policy records outside the slots.
## Decision
Save the failed system release's unambiguous identity and reason, its automatic retry block, and recovery intent before recovery actions. Apply the block on every automatic installation path. It survives reset and rediscovery of the same release. A different release is subject to normal admission checks. An authorized operator may clear the block after review, with the action recorded.
Use a finite recovery budget and a defined check deadline. Boot control accounts for each recovery boot attempt durably before handing control to the target. A restart does not refresh this budget. The reset mechanism must cover hangs that would otherwise prevent progress.
Reconcile the supervisor's progress with authoritative boot state after interruption. Exclude competing installs while recovery owns the target and data. Report recovery success only after the restored system passes its essential local checks.
If the target is ineligible, checks fail, or the recovery budget expires, stop the ordinary A/B path. Enter a separately defined recovery policy rather than alternating releases. The platform must specify the available recovery environment or physical service path, behavior of affected functions, and diagnostics that remain possible.
Retain the failed release, failure reason, recovery result, running release, active block, and any approved data loss for later reporting. Losing connectivity does not prevent local recovery.
## Alternatives Considered
### Retry Until One Slot Works
Can leave a device switching indefinitely without providing its required functions or reaching a diagnosable terminal state.
### Let Rediscovery Clear the Failure
Normal polling could immediately repeat a known failing installation. A new offer alone is not evidence that the failure was corrected.
### Block All Future Releases
May fit stricter products, but prevents an otherwise acceptable corrective release from being considered. This example blocks the exact failed release on this device.
## Consequences
### Positive
- Failed updates cannot automatically repeat without policy intervention.
- Recovery has a finite outcome and continues offline.
- Reports distinguish an update failure from a failed recovery.
### Negative
- Boot control and supervisor records need coordination and compatible formats.
- Retry limits, check deadlines, and watchdog coverage need platform validation.
- A separate recovery path and a block-removal process must be maintained.
## Applicability and Limits
Use for automatic system rollback with locally retained state. This extends the package retry-block pattern from ADR-015 to system release identities. It does not implement a rescue environment or a fleet rollout pause. The block is operational policy; it does not replace signature verification or security anti-rollback rules.
## Related Documents
- [Chapter 07: Rollback and Recovery](../docs/07-rollback-and-recovery.md)
- [ADR-014: Trial Boot and Confirmation](ADR-014-trial-boot-and-confirmation.md)
- [ADR-015: Service Checks and Package Downgrade](ADR-015-service-checks-and-package-downgrade.md)
- [ADR-019: Recover Interrupted Updates From Durable State](ADR-019-recover-interrupted-updates-from-durable-state.md)
- [ADR-020: Define Rollback Eligibility and Retention](ADR-020-define-rollback-eligibility-and-retention.md)
+52
View File
@@ -0,0 +1,52 @@
@startuml
title Ordinary rollback policy after a failed system update
skinparam shadowing false
skinparam monochrome true
start
:Save failed release, reason, retry block,
recovery intent and finite budget;
:Exclude competing installation;
if (Rollback window open and
target eligible?) then (yes)
if (Recovery boot budget remains?) then (yes)
if (Current data usable by A?) then (yes)
else (no)
if (Valid compatible backup and
restore explicitly authorized?) then (yes)
:Restore through the defined recovery procedure;
:Retain any data-loss result;
else (no)
:Save failure; enter separate recovery policy;
stop
endif
endif
:Boot control saves consumed attempt
and selects A before handoff;
if (A reaches checks and passes
within the deadline?) then (yes)
:Save recovery success and pending report;
:Retain failed-release block;
stop
else (no)
:Explicit check failure goes to separate recovery;
:Failure to reach checks uses reset path
and remaining saved budget;
:Budget exhaustion goes to separate recovery;
stop
endif
else (no)
:Save exhausted budget; enter separate recovery;
stop
endif
else (no)
:Save ineligible target; enter separate recovery;
stop
endif
legend bottom
Target eligibility includes artifacts, shared components, configuration,
recovery-state compatibility, and permission under security policy.
Early boot recovery relies on the contract established before trial.
A backup restore before A boots requires a capable recovery environment.
This diagram describes policy, not storage atomicity or a rescue implementation.
endlegend
@enduml
@@ -0,0 +1,46 @@
@startuml
title Recovery coordination and restart after boot handoff
skinparam shadowing false
skinparam monochrome true
participant "Recovery supervisor" as Supervisor
database "Recovery journal" as Journal
participant "Data recovery procedure" as Data
participant "Boot control" as Boot
participant "Retained system A" as A
Supervisor -> Journal : Save failed release, reason, block,\nrecovery intent and reporting obligation
Supervisor -> Supervisor : Exclude installs; establish target eligibility
opt Approved compatible backup needed
Supervisor -> Data : Restore under its interruption contract
Data --> Supervisor : Compatible data; loss accounting
Supervisor -> Journal : Save completed restore and data-loss result
end
Supervisor -> Boot : Request recovery boot of A
Boot -> Boot : Save selection and consumed recovery attempt
Boot --> Supervisor : Saved state / request result
... reset can occur before supervisor saves progress ...
Boot -> A : Handoff under saved boot state
A -> Supervisor : Start local recovery coordinator
Supervisor -> Boot : Read authoritative selection and budget
Boot --> Supervisor : A selected; attempt already consumed
Supervisor -> Journal : Reconcile progress; do not replenish budget
Supervisor -> A : Check required local functions with deadline
alt Checks pass
A --> Supervisor : Healthy
Supervisor -> Journal : Save recovery success, running release,\ndata loss and pending report; retain block
else Checks fail
A --> Supervisor : Failed
Supervisor -> Journal : Save recovery failure
Supervisor -> Supervisor : Enter separate recovery policy
end
note over Boot,A
If A cannot reach checks, the reset path and saved budget
bound further boots. Exhaustion enters separate recovery.
end note
note over Supervisor,Data
Supervisor available here is an assumption. If restoration must
precede A's startup, a capable recovery environment must perform it.
Each component needs its own durable write and restart contract.
The sequence is not one transaction. Reporting may remain offline.
end note
@enduml
+43
View File
@@ -0,0 +1,43 @@
@startuml
title Example rollback window across a release lifecycle
skinparam shadowing false
skinparam monochrome true
participant "Release lifecycle" as Release
participant "Rollback policy" as Policy
participant "Shared data" as Data
participant "Slot A" as A
Release -> Policy : Admit B's trial; retain eligible A
Policy -> A : Preserve A and matching artifacts
Policy -> Data : Require A/B-compatible reads and writes
note over Policy,A
Rollback window open during trial.
end note
Release -> Release : B passes local checks
Release -> Policy : Confirm B
note over Policy,Data
Confirmation accepts B. It does not itself prove that later
data changes remain compatible or define a retention period.
end note
opt Product promises rollback after confirmation
Policy -> A : Continue retaining eligible A
Policy -> Data : Continue preserving A compatibility
end
Release -> Policy : End promised rollback period
Policy -> Policy : Durably close A's rollback window
note over Policy,A
If closure is interrupted, never infer an open window
after its prerequisites have been removed.
end note
opt Incompatible migration required
Release -> Data : Migrate under a separate recovery procedure
end
opt A becomes the target of a later update
Release -> A : Reuse under the next update's admission policy
end
note over Policy,A
A is no longer promised as an ordinary recovery choice.
Separate recovery and security policy still apply.
Closure, migration, and slot reuse are distinct events.
end note
@enduml
+7 -4
View File
@@ -1,14 +1,17 @@
# OTA diagrams # OTA diagrams
PlantUML sources accompany chapters 03–06. The diagrams describe the example A/B design, not a required platform or partition table. PlantUML sources accompany chapters 03–07. The diagrams describe the example A/B design, not a required platform or partition table.
| View | Source | Preview | | View | Source | Preview |
|---|---|---| |---|---|---|
| Roles and the server/device boundary | [PlantUML](plantuml/ota-architecture.puml) | [PNG](rendered/ota-architecture.png) | | Roles and the server/device boundary | [PlantUML](plantuml/ota-architecture.puml) | [PNG](rendered/ota-architecture.png) |
| Trial boot, confirmation and recovery | [PlantUML](plantuml/ab-trial-boot.puml) | [PNG](rendered/ab-trial-boot.png) | | Trial boot, confirmation and recovery | [PlantUML](plantuml/ab-trial-boot.puml) | [PNG](rendered/ab-trial-boot.png) |
| Storage areas and access responsibilities | [PlantUML](plantuml/ab-storage-layout.puml) | [PNG](rendered/ab-storage-layout.png) | | Storage areas and access responsibilities | [PlantUML](plantuml/ab-storage-layout.puml) | [PNG](rendered/ab-storage-layout.png) |
| Preparation and activation sequence | [PlantUML](plantuml/ab-installation-sequence.puml) | Source only | | Preparation and activation sequence | [PlantUML](plantuml/ab-installation-sequence.puml) | [PNG](rendered/ab-installation-sequence.png) |
| Installation states and restart decisions | [PlantUML](plantuml/ab-installation-recovery.puml) | Source only | | Installation states and restart decisions | [PlantUML](plantuml/ab-installation-recovery.puml) | [PNG](rendered/ab-installation-recovery.png) |
| Rollback eligibility and recovery decisions | [PlantUML](plantuml/rollback-policy.puml) | [PNG](rendered/rollback-policy.png) |
| Recovery coordination and restart | [PlantUML](plantuml/rollback-recovery-sequence.puml) | [PNG](rendered/rollback-recovery-sequence.png) |
| Rollback window and release lifecycle | [PlantUML](plantuml/rollback-window.puml) | [PNG](rendered/rollback-window.png) |
From the repository root, regenerate previews with a local PlantUML installation and Graphviz: From the repository root, regenerate previews with a local PlantUML installation and Graphviz:
@@ -16,4 +19,4 @@ From the repository root, regenerate previews with a local PlantUML installation
JAVA_TOOL_OPTIONS=-Djava.awt.headless=true plantuml -tpng -o ../rendered diagrams/plantuml/*.puml JAVA_TOOL_OPTIONS=-Djava.awt.headless=true plantuml -tpng -o ../rendered diagrams/plantuml/*.puml
``` ```
Chapters 03–05 include existing previews. The chapter 06 diagrams are provided as sources only; generate previews locally when needed. No external rendering service is needed. These diagrams do not establish power-loss safety. The [A/B simulation](../examples/simulations/ab-trial-boot/readme.md) covers trial boot. The [interrupted installation model](../examples/simulations/interrupted-installation/readme.md) covers preparation, old progress, and supplied boot-record write outcomes. Chapters 03–07 include PNG previews alongside their editable PlantUML sources. No external rendering service is needed. These diagrams do not establish power-loss safety. The [A/B simulation](../examples/simulations/ab-trial-boot/readme.md) covers trial boot. The [interrupted installation model](../examples/simulations/interrupted-installation/readme.md) covers preparation, old progress, and supplied boot-record write outcomes. The [rollback policy simulation](../examples/simulations/rollback-policy/readme.md) covers recovery eligibility, data loss, bounded attempts, and failed-release blocks.
Binary file not shown.

After

Width:  |  Height:  |  Size: 83 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 91 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 62 KiB

+7 -4
View File
@@ -71,7 +71,7 @@ The emphasis is always on understanding the underlying design rather than memori
## Repository Structure ## Repository Structure
The structure below includes the current documentation, PlantUML diagrams, and A/B simulation. Hardware demonstrations and later chapters will be added as the practical material develops. The structure below includes the current documentation, PlantUML diagrams, and C++ simulations. Hardware demonstrations and later chapters will be added as the practical material develops.
```text ```text
ota-reference-design/ ota-reference-design/
@@ -84,6 +84,7 @@ ota-reference-design/
│ ├── 04-starting-an-updated-system-or-service.md │ ├── 04-starting-an-updated-system-or-service.md
│ ├── 05-storage-layout.md │ ├── 05-storage-layout.md
│ ├── 06-atomic-updates.md │ ├── 06-atomic-updates.md
│ ├── 07-rollback-and-recovery.md
│ ├── ... │ ├── ...
│ │
├── diagrams/ ├── diagrams/
@@ -93,7 +94,8 @@ ota-reference-design/
├── examples/ ├── examples/
│ └── simulations/ │ └── simulations/
│ ├── ab-trial-boot/ │ ├── ab-trial-boot/
│ └── interrupted-installation/ │ ├── interrupted-installation/
│ └── rollback-policy/
│ │
├── adr/ ├── adr/
│ ├── ADR-001-repository-name.md │ ├── ADR-001-repository-name.md
@@ -115,6 +117,7 @@ The chapters are designed to build on each other. The current chapters are:
- [04. Starting an Updated System or Service](04-starting-an-updated-system-or-service.md) - [04. Starting an Updated System or Service](04-starting-an-updated-system-or-service.md)
- [05. Storage Layout](05-storage-layout.md) - [05. Storage Layout](05-storage-layout.md)
- [06. Atomic Updates](06-atomic-updates.md) - [06. Atomic Updates](06-atomic-updates.md)
- [07. Rollback and Recovery](07-rollback-and-recovery.md)
To catch up on additions and revisions, see the [changelog](../CHANGELOG.md), with dated summaries and links to chapters, diagrams, simulations, and related history. To catch up on additions and revisions, see the [changelog](../CHANGELOG.md), with dated summaries and links to chapters, diagrams, simulations, and related history.
@@ -139,7 +142,7 @@ Storage Layout
Atomic Updates Atomic Updates
│ │
▼ ▼
Rollback Rollback and Recovery
│ │
▼ ▼
Security Security
@@ -201,7 +204,7 @@ Where possible, theoretical discussions are accompanied by practical material, i
The objective is to connect high-level architecture with practical implementation. The objective is to connect high-level architecture with practical implementation.
Available now: [PlantUML diagrams](../diagrams/readme.md), a [C++ A/B trial boot simulation](../examples/simulations/ab-trial-boot/readme.md), and a [C++ interrupted installation model](../examples/simulations/interrupted-installation/readme.md). Chapters 03–05 have PNG previews; chapter 06 currently has sources only. Raspberry Pi and QEMU demonstrations are planned. Available now: [PlantUML diagrams](../diagrams/readme.md), a [C++ A/B trial boot simulation](../examples/simulations/ab-trial-boot/readme.md), a [C++ interrupted installation model](../examples/simulations/interrupted-installation/readme.md), and a [C++ rollback policy simulation](../examples/simulations/rollback-policy/readme.md). Chapters 03–07 have PNG previews alongside their PlantUML sources. Raspberry Pi and QEMU demonstrations are planned.
--- ---
+5 -1
View File
@@ -58,6 +58,8 @@ If the request returns an error or its result is lost during a restart, the OTA
[Preparation and activation sequence — PlantUML source](../diagrams/plantuml/ab-installation-sequence.puml) [Preparation and activation sequence — PlantUML source](../diagrams/plantuml/ab-installation-sequence.puml)
![Preparation and activation sequence](../diagrams/rendered/ab-installation-sequence.png)
There is a gap between steps 6 and 7: boot control may have saved the request while the OTA manager still says “prepared.” After a restart, boot control can start B. The OTA manager must accept that saved boot decision and repair its progress record. It must not overwrite B or reset the trial attempt count. There is a gap between steps 6 and 7: boot control may have saved the request while the OTA manager still says “prepared.” After a restart, boot control can start B. The OTA manager must accept that saved boot decision and repair its progress record. It must not overwrite B or reset the trial attempt count.
Two separate records do not form one transaction. Safe ordering and recovery rules handle the states left between them. Two separate records do not form one transaction. Safe ordering and recovery rules handle the states left between them.
@@ -103,6 +105,8 @@ Package scripts, database migrations, and external actions also need their own r
[Installation states and restart decisions — PlantUML source](../diagrams/plantuml/ab-installation-recovery.puml) [Installation states and restart decisions — PlantUML source](../diagrams/plantuml/ab-installation-recovery.puml)
![Installation states and restart decisions](../diagrams/rendered/ab-installation-recovery.png)
[ADR-019: Recover Interrupted Updates From Durable State](../adr/ADR-019-recover-interrupted-updates-from-durable-state.md) defines the ownership and recovery rules used here. [ADR-017](../adr/ADR-017-keep-recovery-state-outside-replaceable-software.md) defines where the records must live. [ADR-019: Recover Interrupted Updates From Durable State](../adr/ADR-019-recover-interrupted-updates-from-durable-state.md) defines the ownership and recovery rules used here. [ADR-017](../adr/ADR-017-keep-recovery-state-outside-replaceable-software.md) defines where the records must live.
## 6. An Independent Service Update ## 6. An Independent Service Update
@@ -151,4 +155,4 @@ Model tests check ordering. Platform tests must also interrupt real writes, rese
Preparation keeps incomplete software out of the boot path. Activation gives a complete version a chance to run. Confirmation still depends on checks after startup. Preparation keeps incomplete software out of the boot path. Activation gives a complete version a chance to run. Confirmation still depends on checks after startup.
Next, we will look more closely at rollback: when to return to a previous version, what recovery needs, and when that previous version is no longer a usable choice. Next, [chapter 07: Rollback and Recovery](07-rollback-and-recovery.md) examines when to return to a previous version, what recovery needs, and when that previous version is no longer a usable choice.
+149
View File
@@ -0,0 +1,149 @@
# 07. Rollback and Recovery: When the Previous Version Still Works
[Chapter 06](06-atomic-updates.md) kept incomplete software out of the boot path and defined how to continue an interrupted installation. A complete update can still fail when it runs.
Suppose B starts, changes a shared database, and then fails its checks. Boot control returns to A. A's files are intact, but A cannot read the new database. The slot switch worked; recovery did not.
The question for this chapter is: **what must remain usable for the device to recover, and what happens when returning to the previous version is no longer safe?**
## 1. Three Different Operations
| Operation | What it changes | What it does not establish |
|---|---|---|
| System rollback | Selects a retained system release, including its matching boot artifacts | That its data is compatible or its services work |
| Package downgrade | Reinstalls an older package on the current platform | That dependencies, scripts, and data can be restored |
| Recovery | Restores the required device functions through a defined procedure | That returning to an old release is always the right procedure |
Rollback and downgrade are possible parts of recovery. Recovery may instead require a separate environment, a repair procedure, or physical service.
We keep the A/B example from earlier chapters. A was confirmed before B's trial. Boot control owns slot selection and attempts. A separate supervisor coordinates local checks and recovery. Its records remain outside the replaceable slots. These roles need not be separate executables.
## 2. Decide Which Failure Starts Recovery
The trial rules from [ADR-014](../adr/ADR-014-trial-boot-and-confirmation.md) still apply:
- If B reaches local checks and fails them, request recovery without spending every remaining trial attempt.
- If B cannot reach the checks, saved attempt limits let boot control return to A on a later boot.
- If B hangs, a watchdog or another defined mechanism must cause that later boot.
Losing the server connection alone is not a failed local health check in this design. Recovery and its checks work offline; reporting can wait.
A failure after B has been confirmed is a different case. The trial may have ended, A may have been replaced, or shared data may have changed. Do not silently restart the old trial policy. Reassess the available recovery target under the product's current policy.
For an early boot failure, Linux cannot run a compatibility check before choosing A. Before allowing B's trial, the system must establish that the retained software and permitted trial data changes preserve this recovery path. A runtime supervisor can perform additional checks when it runs, but cannot replace that admission rule.
## 3. A Previous Release Must Still Be Eligible
Before relying on A, establish:
1. Its release identity and matching boot artifacts are known, retained, and valid under the platform's verification policy.
2. Its required shared components and device configuration remain compatible.
3. It can use the current data, or an approved restore procedure can prepare compatible data.
4. The rollback window is still open and the platform's security policy permits this release.
5. The local recovery mechanism and a finite attempt budget are available.
These conditions are a recovery contract. A record saying that A passed checks last week does not prove them today. Retain the artifacts, dependencies, and verification information that the contract requires, rather than relying on a later download.
The contract must also cover update-state formats shared by A and B. Old software that cannot interpret the recovery record is not a usable recovery target.
[ADR-020: Define Rollback Eligibility and Retention](../adr/ADR-020-define-rollback-eligibility-and-retention.md) defines this choice. The following decision diagram shows the ordinary rollback path and where it must stop.
![Rollback policy](../diagrams/rendered/rollback-policy.png)
[Rollback policy — PlantUML source](../diagrams/plantuml/rollback-policy.puml)
## 4. Recovering Software and Data Together
The [storage chapter](05-storage-layout.md) separated device data from system slots. That preserves data when an image is replaced, but both versions still need to understand it.
| Approach | Benefit | Cost or limit |
|---|---|---|
| Keep a format usable by A and B | Return to A can retain records written during B's trial | Both readers and writers must preserve compatibility |
| Keep a separate trial copy | B's changes can be isolated from A's copy | Extra space; new records need an explicit merge or loss policy |
| Restore a validated backup | Can recover a compatible earlier dataset | Restore needs interruption handling; newer records may be lost |
| Delay an incompatible migration | Preserves the old recovery path during trial | B must work before that migration; the rollback window must end explicitly |
Compatibility includes write behavior. B must not merely read A's format; its writes must leave data A can still use. Configuration changes need the same review.
For the shared-data reference design, keep data compatible during trial. Reject or defer a release that needs an incompatible change before confirmation and has no supported recovery path. [ADR-021](../adr/ADR-021-preserve-data-compatibility-during-trial.md) records this decision.
A backup is an alternative, not an automatic escape hatch. Suppose it contains measurements 1 and 2, while the live database contains 1 through 4. Restoring it discards measurements 3 and 4 unless another mechanism preserves them. Product policy must authorize that loss before the restore. A copy also needs a valid format, consistency checks, sufficient space, and a tested restore procedure.
An incompatible migration cannot normally be postponed until after a failed early boot and then undone by an unavailable Linux supervisor. If restoring data is necessary before A can start, a recovery environment capable of doing that work must run first.
## 5. Execute Recovery as a Saved Procedure
A possible procedure after a detected failure is:
1. Save the failed release identity, reason, recovery target, remaining budget, and a block against automatically retrying the failed release.
2. Exclude a competing installation while recovery owns the target and data.
3. Check target eligibility. Prepare compatible data through the approved procedure, if needed.
4. Ask boot control to select A under its durable-state contract. Account for each recovery boot attempt before handing control to A.
5. Start A and check the required local functions within a defined time limit.
6. Save either recovered or recovery failed, the running release, any data loss, and a pending report.
A selected slot is not a successful recovery. Only the checks establish that the required functions have returned.
![Recovery sequence](../diagrams/rendered/rollback-recovery-sequence.png)
[Recovery sequence — PlantUML source](../diagrams/plantuml/rollback-recovery-sequence.puml)
Boot control remains authoritative about boot selection and attempts. The supervisor's journal must not override it after a reset. If selection succeeded but saving local progress did not, read the saved boot state and reconcile the journal, following [ADR-019](../adr/ADR-019-recover-interrupted-updates-from-durable-state.md).
Give every restore, package operation, and state transition an interruption rule. After a reset, determine whether it completed before repeating it. In particular, do not restore an old backup again after normal operation has resumed and written new data. Do not replenish an attempt budget merely because a coordinator restarted.
For an independent service, use the package manager's repair and downgrade rules from [ADR-015](../adr/ADR-015-service-checks-and-package-downgrade.md). Keep its supervisor usable when the service fails. A package command completing still needs service checks afterward.
## 6. Stop Repeated Failures
After recovering A, a normal update check may discover B again. Without a saved block, the device can repeat the same failed update forever.
In this example, the block identifies the exact failed system release on this device, survives reset, and applies to every automatic installation path. Rediscovering B does not remove it. A different release can be considered through the normal admission checks. An authorized operator can clear the block after review, with the action recorded. Use an unambiguous release identity rather than a reusable display name.
If A fails its checks, cannot start within the recovery budget, or is ineligible, stop the ordinary A/B path. Do not keep alternating A and B. Record the available diagnosis and enter the platform's separate recovery policy.
That policy must identify what can actually run: for example, a recovery environment with independent boot support, or a physical service path. A third partition sharing the same broken boot code does not by itself solve the failure. Define whether the device stops affected functions, what diagnostics remain available, and how service is requested.
[ADR-022: Bound Recovery and Block Failed System Releases](../adr/ADR-022-bound-recovery-and-block-failed-system-releases.md) records these rules. This chapter defines the boundary; it does not implement a rescue image or hardware recovery mechanism.
## 7. Define When the Rollback Window Ends
**Confirmation** accepts B after its required checks. **Closing the rollback window** ends the promise that the device can return to A. **Reusing A's slot** removes the retained software. These are separate events, although a product may deliberately coordinate them.
If the product promises rollback after confirmation, it must continue preserving A and compatible data for that period. Otherwise it can explicitly close the window after confirmation, then perform an incompatible migration or reuse the slot under the next update's policy. Before either destructive action, save that A is no longer an ordinary recovery choice. If the transition is interrupted, a conservative closed window is safer than promising an unavailable target.
![Rollback window](../diagrams/rendered/rollback-window.png)
[Rollback window — PlantUML source](../diagrams/plantuml/rollback-window.puml)
Recovery selection must also respect the platform's security policy. A local failure block prevents repeating a failed deployment; a security anti-rollback rule restricts which older software may run. Their purposes differ. Signature verification and security version policy follow in the security chapter.
## 8. A C++ Model of Recovery Policy
The [rollback policy example](../examples/simulations/rollback-policy/readme.md) supplies software eligibility and health-check outcomes to a small C++17 state machine. It models shared data as a schema number and a list of record IDs, with an optional backup.
It covers successful rollback, incompatible data without restore permission, an approved restore with explicit data loss, reset during recovery, failure of A, rediscovery of blocked B, attempt exhaustion, a closed rollback window, and an unavailable target.
The existing [trial boot example](../examples/simulations/ab-trial-boot/readme.md) explains boot mechanics. The new model explains recovery decisions and data consequences. They are separate programs. The backup scenarios deliberately explore an alternative to ADR-021's trial policy; they do not authorize incompatible trial writes in the shared-data reference design.
Each modeled transition is assumed to finish durably. Reset points occur between transitions. The model does not implement real backup restoration, a bootloader, package operations, signatures, storage flushing, watchdogs, or interruption within a write. A boolean eligibility input is an assumption, not an implementation of release verification.
## 9. Check the Design Against Failures
| Failure | Required result |
|---|---|
| B fails checks; A and data remain usable | Return to A, check it, save recovery success, keep B blocked |
| B cannot reach the supervisor | Boot control and the reset path enforce the admitted trial recovery contract |
| A cannot read current data | Use an approved restore path or separate recovery; do not start A on incompatible data |
| Backup restore would lose unapproved data | Stop ordinary rollback without changing the live data |
| Reset during recovery | Resume from authoritative saved state without refreshing the budget |
| A also fails or exhausts attempts | Enter separate recovery; do not alternate slots indefinitely |
| B is offered again after reset | Preserve and enforce the failed-release block |
| Rollback window closed or A replaced | Reject A as an ordinary recovery target |
| Device is offline | Perform local checks and recovery; retain the report for later |
Model checks establish policy behavior under supplied assumptions. Platform tests must also interrupt actual restore and boot-state writes, force startup hangs, and exercise the separate recovery path.
## 10. What Comes Next?
Recovery needs both an available target and permission to use it. Next, we will examine update integrity, signatures, trust, and the security policy that determines which releases a device may accept.
@@ -0,0 +1,23 @@
cmake_minimum_required(VERSION 3.20)
project(rollback_policy LANGUAGES CXX)
set(CMAKE_CXX_EXTENSIONS OFF)
add_library(rollback-model simulation.cpp)
target_compile_features(rollback-model PUBLIC cxx_std_17)
if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
target_compile_options(rollback-model PRIVATE -Wall -Wextra -Wpedantic)
endif()
add_executable(rollback-simulate main.cpp)
target_link_libraries(rollback-simulate PRIVATE rollback-model)
include(CTest)
if(BUILD_TESTING)
add_executable(rollback-tests tests.cpp)
target_link_libraries(rollback-tests PRIVATE rollback-model)
add_test(NAME recovery-policy-invariants COMMAND rollback-tests)
add_test(NAME all-scenarios COMMAND rollback-simulate all)
add_test(NAME cli-help COMMAND rollback-simulate --help)
add_test(NAME cli-invalid COMMAND rollback-simulate invalid)
set_tests_properties(cli-invalid PROPERTIES WILL_FAIL TRUE)
endif()
@@ -0,0 +1,49 @@
#include "simulation.hpp"
#include <exception>
#include <iostream>
#include <string>
namespace model = ota::rollback;
namespace {
void usage(std::ostream &output) {
output << "Usage: rollback-simulate [all|SCENARIO|--help]\nScenarios:\n";
for (const auto &scenario: model::scenarios()) output << " " << scenario << '\n';
}
void run(const std::string &scenario) {
const auto result = model::run_scenario(scenario);
std::cout << scenario << ": " << model::name(result.phase)
<< "; running=" << (result.running == model::Slot::A ? "A" : "B")
<< "; boots=" << result.boots_requested
<< "; attempts-left=" << result.attempts_left
<< "; records-lost=" << result.records_lost
<< "; failed-release-blocked=" << (result.blocked_releases.count(result.failed_release) != 0)
<< "; " << result.recovery_reason << '\n';
}
}// namespace
int main(int argc, char **argv) {
if (argc == 2 && std::string(argv[1]) == "--help") {
usage(std::cout);
return 0;
}
if (argc > 2) {
usage(std::cerr);
return 1;
}
try {
const std::string requested = argc == 2 ? argv[1] : "all";
if (requested == "all") {
for (const auto &scenario: model::scenarios()) run(scenario);
} else
run(requested);
} catch (const std::exception &error) {
std::cerr << error.what() << '\n';
usage(std::cerr);
return 1;
}
}
@@ -0,0 +1,70 @@
# Rollback policy simulation
This C++17 standard-library example accompanies [chapter 07](../../../docs/07-rollback-and-recovery.md), [ADR-020](../../../adr/ADR-020-define-rollback-eligibility-and-retention.md), [ADR-021](../../../adr/ADR-021-preserve-data-compatibility-during-trial.md), and [ADR-022](../../../adr/ADR-022-bound-recovery-and-block-failed-system-releases.md). It models recovery decisions after B fails, including data compatibility, bounded attempts, and a persistent failed-release block.
Build with CMake 3.20 or later and a C++17 compiler from the repository root. No third-party libraries are needed:
```sh
cmake -S examples/simulations/rollback-policy -B /tmp/ota-rollback-build -DCMAKE_BUILD_TYPE=Debug
cmake --build /tmp/ota-rollback-build
/tmp/ota-rollback-build/rollback-simulate all
/tmp/ota-rollback-build/rollback-simulate backup-restore
ctest --test-dir /tmp/ota-rollback-build --output-on-failure
```
No argument also runs all scenarios. `--help` lists their names. Invalid arguments return a nonzero status.
## Scenarios
| Scenario | Expected result |
|---|---|
| `success` | Compatible data; A passes checks; recovery succeeds. |
| `incompatible-data` | A cannot read schema 2; no approved restore; stop without changing data or booting A. |
| `backup-restore` | Approved schema-1 backup replaces schema-2 data; A passes; two live record IDs are lost. |
| `restart-during-recovery` | Reset before A completes checks; another boot consumes the remaining attempt and passes. |
| `recovery-failure` | A explicitly fails local checks; enter separate recovery immediately. |
| `blocked-release` | After recovery and reset, rediscovering system-2 leaves it blocked. |
| `attempts-exhausted` | Two A boots fail to reach completed checks; no third boot is allowed. |
| `window-closed` | Retained A is outside the promised rollback window; do not select it. |
| `target-unavailable` | Supplied target eligibility fails; do not select A. |
The backup scenarios explore an alternative to ADR-021's compatible shared-data trial policy. They are not permission for incompatible writes during that reference design's trial. A real early-boot failure would need a recovery environment able to restore data before A uses it.
## State and Ownership
`Storage` represents saved boot-control and supervisor records plus data. It survives reconstruction of `Coordinator`; nothing is persisted to real files. The normal path is:
```text
Idle -> Assess -> [RestoreData] -> SelectPrevious -> CheckPrevious -> Recovered
| | | |
+-----------+----------------+-----------------+--> SeparateRecovery
```
`start()` saves the failure reason, exact release block, and recovery intent before taking action. Calling it again is rejected. Resume the saved phase instead. During recovery, automatic offers are excluded. `can_offer()` checks only that exclusion and the local failed-release block; it does not implement signature, hardware, or full admission policy.
The modeled boot-control transition saves an attempt before handoff to A. Creating another coordinator does not consume or refresh attempts. `after_reset()` represents an actual device reset: an A boot that had not completed checks must go through another accounted handoff. After the illustrative budget of two boots is exhausted, the result is separate recovery. An explicit failed check stops sooner.
Data consists of a schema number and unique record IDs. A accepts schema 1. Backup validity and restore permission are supplied inputs; the model counts live IDs missing from the restored copy. A completed restore is not repeated on restart. Terminal recovery does not restore the backup over later records.
`running` records the last modeled handoff. At separate recovery it is diagnostic history, not evidence that a system is healthy or that a rescue environment started. Pending results and blocks are retained; no network transport or block-removal API is implemented.
## Checks and Limits
[tests.cpp](tests.cpp) checks all scenario outcomes, reset boundaries before and after restore and boot, data-loss accounting, block retention, exclusion of competing offers, finite attempt consumption, check-before-success ordering, invalid or unauthorized backups, and changes in eligibility before an action. The checks use exceptions and remain enabled in Release builds.
Each transition is assumed to complete durably. Data replacement and its phase change form one assumed operation here. The example does not test a crash inside a restore or between separate real boot-control and journal writes. Real implementations need their own reconciliation rules, atomic data selection or restartable restore procedure, and platform failure injection.
The model supplies eligibility and health outcomes. It does not validate actual artifacts, implement signatures or security anti-rollback, interpret a real schema, run a watchdog, measure deadlines, invoke a package manager, or enter a rescue system. A closed window and an unavailable target are policy inputs rather than a retention implementation. There are no concurrent writers. The block covers one failed update on one device.
The [trial boot simulation](../ab-trial-boot/readme.md) covers boot mechanics; the [interrupted installation model](../interrupted-installation/readme.md) covers preparation and activation. The three programs are independent, not an integrated updater.
## Sources and Diagrams
- [simulation.hpp](simulation.hpp) and [simulation.cpp](simulation.cpp): saved state, policy transitions, and scenarios.
- [main.cpp](main.cpp): scenario runner.
- [tests.cpp](tests.cpp) and [CMakeLists.txt](CMakeLists.txt): invariants and CTest integration.
- Rollback policy: [PlantUML](../../../diagrams/plantuml/rollback-policy.puml), [PNG](../../../diagrams/rendered/rollback-policy.png).
- Recovery sequence: [PlantUML](../../../diagrams/plantuml/rollback-recovery-sequence.puml), [PNG](../../../diagrams/rendered/rollback-recovery-sequence.png).
- Rollback window: [PlantUML](../../../diagrams/plantuml/rollback-window.puml), [PNG](../../../diagrams/rendered/rollback-window.png).
Format C++ sources with the repository's root `.clang-format`. Diagrams include editable PlantUML sources and PNG previews.
@@ -0,0 +1,193 @@
#include "simulation.hpp"
#include <algorithm>
#include <stdexcept>
namespace ota::rollback {
namespace {
bool compatible(const Data &data) {
return data.schema == 1;
}
bool can_restore(const Storage &storage) {
return storage.restore_authorized && storage.backup_valid && storage.backup && compatible(*storage.backup);
}
bool eligible(const Storage &storage) {
return storage.window_open && storage.target_eligible;
}
void finish(Coordinator &coordinator, Storage &storage, bool healthy = true) {
for (int steps = 0; !terminal(storage.phase); ++steps) {
if (steps >= 16) throw std::runtime_error("Recovery did not terminate");
coordinator.advance(healthy);
}
}
}// namespace
Coordinator::Coordinator(Storage &storage) : storage_(storage) {}
void Coordinator::start(const std::string &reason) {
if (storage_.phase != Phase::Idle) throw std::logic_error("Recovery already started; resume saved state");
if (reason.empty() || storage_.failed_release.empty() || storage_.previous_release.empty() || storage_.failed_release == storage_.previous_release)
throw std::logic_error("Recovery requires distinct release identities and a failure reason");
storage_.failure_reason = reason;
storage_.blocked_releases.insert(storage_.failed_release);
storage_.report_pending = true;
storage_.phase = Phase::Assess;
}
void Coordinator::stop(const std::string &reason) {
storage_.recovery_reason = reason;
storage_.phase = Phase::SeparateRecovery;
}
void Coordinator::advance(bool previous_healthy) {
switch (storage_.phase) {
case Phase::Idle:
throw std::logic_error("Save recovery intent before taking action");
case Phase::Assess:
if (!eligible(storage_)) stop("previous release unavailable or rollback window closed");
else if (storage_.attempts_left <= 0)
stop("recovery boot budget exhausted");
else if (compatible(storage_.data))
storage_.phase = Phase::SelectPrevious;
else if (can_restore(storage_))
storage_.phase = Phase::RestoreData;
else
stop("incompatible data; no authorized valid restore");
break;
case Phase::RestoreData: {
if (!eligible(storage_) || storage_.attempts_left <= 0 || !can_restore(storage_)) {
stop("restore prerequisites no longer hold");
break;
}
// Record IDs are unique. Count live records absent from the backup.
const auto &saved = storage_.backup->records;
storage_.records_lost = static_cast<std::size_t>(std::count_if(
storage_.data.records.begin(), storage_.data.records.end(),
[&](int record) { return std::find(saved.begin(), saved.end(), record) == saved.end(); }));
storage_.data = *storage_.backup;
++storage_.restores;
// Data replacement and this progress transition are one assumed durable
// operation in this model. Real restores need their own recovery protocol.
storage_.phase = Phase::SelectPrevious;
break;
}
case Phase::SelectPrevious:
if (!eligible(storage_) || !compatible(storage_.data)) {
stop("previous release or data no longer eligible");
break;
}
if (storage_.attempts_left <= 0) {
stop("recovery boot budget exhausted");
break;
}
// Abstract boot control saves the consumed attempt before handoff.
--storage_.attempts_left;
++storage_.boots_requested;
storage_.running = Slot::A;
storage_.phase = Phase::CheckPrevious;
break;
case Phase::CheckPrevious:
if (!eligible(storage_) || !compatible(storage_.data) || storage_.running != Slot::A)
stop("recovery prerequisites changed before checks");
else if (!previous_healthy)
stop("previous release failed local checks");
else {
storage_.recovery_reason = "previous release passed local checks";
storage_.phase = Phase::Recovered;
}
break;
case Phase::Recovered:
case Phase::SeparateRecovery:
break;
}
}
void Coordinator::after_reset() {
// A reset before completed checks requires another accounted boot. Recreating
// the Linux coordinator alone does not imply a device reset or new attempt.
if (storage_.phase == Phase::CheckPrevious) storage_.phase = Phase::SelectPrevious;
}
bool terminal(Phase phase) {
return phase == Phase::Recovered || phase == Phase::SeparateRecovery;
}
bool can_offer(const Storage &storage, const std::string &release) {
// This models the local block and exclusion only, not full update admission.
return (storage.phase == Phase::Idle || storage.phase == Phase::Recovered) &&
storage.blocked_releases.count(release) == 0;
}
const char *name(Phase phase) {
switch (phase) {
case Phase::Idle:
return "idle";
case Phase::Assess:
return "assess";
case Phase::RestoreData:
return "restore-data";
case Phase::SelectPrevious:
return "select-previous";
case Phase::CheckPrevious:
return "check-previous";
case Phase::Recovered:
return "recovered";
case Phase::SeparateRecovery:
return "separate-recovery";
}
throw std::logic_error("Unknown recovery phase");
}
const std::vector<std::string> &scenarios() {
static const std::vector<std::string> values{
"success", "incompatible-data", "backup-restore", "restart-during-recovery",
"recovery-failure", "blocked-release", "attempts-exhausted", "window-closed", "target-unavailable"};
return values;
}
Storage run_scenario(const std::string &scenario) {
if (std::find(scenarios().begin(), scenarios().end(), scenario) == scenarios().end())
throw std::invalid_argument("Unknown scenario: " + scenario);
Storage storage;
if (scenario == "incompatible-data" || scenario == "backup-restore") storage.data.schema = 2;
if (scenario == "backup-restore") {
storage.backup = Data{1, {1, 2}};
storage.backup_valid = true;
storage.restore_authorized = true;
}
if (scenario == "window-closed") storage.window_open = false;
if (scenario == "target-unavailable") storage.target_eligible = false;
{
Coordinator coordinator(storage);
coordinator.start("B failed required local checks");
if (scenario == "restart-during-recovery" || scenario == "attempts-exhausted") {
coordinator.advance();// Assess.
coordinator.advance();// First A boot; checks not completed.
} else {
finish(coordinator, storage, scenario != "recovery-failure");
}
}
if (scenario == "restart-during-recovery" || scenario == "attempts-exhausted") {
Coordinator restarted(storage);
restarted.after_reset();
if (scenario == "attempts-exhausted") {
restarted.advance();// Second A boot also fails to reach completed checks.
restarted.after_reset();
}
finish(restarted, storage);
}
if (scenario == "blocked-release") {
Coordinator restarted(storage);
restarted.after_reset();
if (can_offer(storage, storage.failed_release)) throw std::runtime_error("Failed release was offered again");
}
return storage;
}
}// namespace ota::rollback
@@ -0,0 +1,67 @@
#pragma once
#include <cstddef>
#include <optional>
#include <set>
#include <string>
#include <vector>
namespace ota::rollback {
enum class Phase { Idle,
Assess,
RestoreData,
SelectPrevious,
CheckPrevious,
Recovered,
SeparateRecovery };
enum class Slot { A,
B };
struct Data {
int schema = 1;
std::vector<int> records{1, 2, 3, 4};
};
// These are assumed complete durable records, not a storage implementation.
struct Storage {
Phase phase = Phase::Idle;
Slot running = Slot::B;
std::string failed_release = "system-2";
std::string previous_release = "system-1";
std::set<std::string> blocked_releases;
std::string failure_reason;
std::string recovery_reason;
bool report_pending = false;
bool window_open = true;
bool target_eligible = true;
int attempts_left = 2;
int boots_requested = 0;
Data data;
std::optional<Data> backup;
bool backup_valid = false;
bool restore_authorized = false;
std::size_t records_lost = 0;
int restores = 0;
};
class Coordinator {
public:
explicit Coordinator(Storage &storage);
void start(const std::string &reason);
void advance(bool previous_healthy = true);
// Call once for an actual reset, not merely a coordinator restart.
void after_reset();
private:
Storage &storage_;
void stop(const std::string &reason);
};
bool terminal(Phase phase);
bool can_offer(const Storage &storage, const std::string &release);
const char *name(Phase phase);
const std::vector<std::string> &scenarios();
Storage run_scenario(const std::string &scenario);
}// namespace ota::rollback
@@ -0,0 +1,161 @@
#include "simulation.hpp"
#include <iostream>
#include <stdexcept>
namespace model = ota::rollback;
namespace {
void expect(bool condition, const char *message) {
if (!condition) throw std::runtime_error(message);
}
template<class Operation>
void rejected(Operation operation) {
try {
operation();
} catch (const std::logic_error &) {
return;
}
throw std::runtime_error("Invalid operation accepted");
}
void finish(model::Storage &storage) {
model::Coordinator coordinator(storage);
for (int steps = 0; !model::terminal(storage.phase); ++steps) {
expect(steps < 16, "Recovery did not terminate");
coordinator.advance();
}
}
void scenario_outcomes() {
for (const auto &scenario: model::scenarios()) {
const auto storage = model::run_scenario(scenario);
const bool recovered = scenario == "success" || scenario == "backup-restore" ||
scenario == "restart-during-recovery" || scenario == "blocked-release";
expect(storage.phase == (recovered ? model::Phase::Recovered : model::Phase::SeparateRecovery), "Wrong scenario outcome");
expect(storage.blocked_releases.count("system-2") == 1, "Failed release block lost");
expect(storage.report_pending && !storage.failure_reason.empty() && !storage.recovery_reason.empty(), "Recovery diagnosis lost");
expect(storage.attempts_left >= 0 && storage.boots_requested <= 2, "Boot budget violated");
if (recovered) {
expect(storage.running == model::Slot::A && storage.data.schema == 1, "Recovered without usable A and data");
expect(model::can_offer(storage, "system-3"), "Different release cannot reach normal admission");
}
expect(!model::can_offer(storage, "system-2"), "Failed release automatically retried");
}
const auto restored = model::run_scenario("backup-restore");
expect(restored.data.records == std::vector<int>({1, 2}) && restored.records_lost == 2 && restored.restores == 1, "Restore data loss not accounted");
const auto incompatible = model::run_scenario("incompatible-data");
expect(incompatible.data.schema == 2 && incompatible.data.records.size() == 4 && incompatible.boots_requested == 0, "Unsafe rollback changed data or booted A");
const auto exhausted = model::run_scenario("attempts-exhausted");
expect(exhausted.attempts_left == 0 && exhausted.boots_requested == 2, "Reset refreshed recovery attempts");
}
void reset_boundaries() {
// Cut after intent, assessment, restore, boot handoff, and completed checks.
for (int cut = 0; cut <= 4; ++cut) {
model::Storage storage;
storage.data.schema = 2;
storage.backup = model::Data{1, {1, 2}};
storage.backup_valid = true;
storage.restore_authorized = true;
{
model::Coordinator coordinator(storage);
coordinator.start("trial health failure");
for (int step = 0; step < cut; ++step) coordinator.advance();
}
const int consumed = storage.boots_requested;
model::Coordinator restarted(storage);
restarted.after_reset();
expect(storage.boots_requested == consumed && storage.attempts_left == 2 - consumed, "Reset changed saved budget");
rejected([&] { restarted.start("rediscovered failure"); });
finish(storage);
expect(storage.phase == model::Phase::Recovered, "Reset prevented recovery");
expect(storage.restores == 1 && storage.records_lost == 2, "Reset repeated backup restore");
// Data acquired after recovery must not be replaced by the old backup.
storage.data.records.push_back(5);
model::Coordinator next(storage);
next.after_reset();
next.advance();
expect(storage.data.records == std::vector<int>({1, 2, 5}) && storage.restores == 1, "Completed recovery restored old backup again");
}
}
void admission_and_guards() {
model::Storage storage;
model::Coordinator coordinator(storage);
rejected([&] { coordinator.advance(); });
rejected([&] { coordinator.start(""); });
coordinator.start("local health failure");
expect(storage.blocked_releases.count(storage.failed_release) == 1 && storage.boots_requested == 0, "Recovery acted before saving block");
expect(!model::can_offer(storage, "system-3"), "Competing update admitted during recovery");
coordinator.advance();
coordinator.advance();
expect(storage.phase == model::Phase::CheckPrevious, "Boot handoff reported success before checks");
model::Coordinator process_restart(storage);
process_restart.advance(false);
expect(storage.phase == model::Phase::SeparateRecovery && storage.boots_requested == 1, "Failed recovery did not stop");
process_restart.after_reset();
process_restart.advance();
expect(storage.phase == model::Phase::SeparateRecovery && storage.boots_requested == 1, "Terminal failure started another boot");
}
void restore_guards() {
for (int fault = 0; fault < 4; ++fault) {
model::Storage storage;
storage.data.schema = 2;
storage.backup = model::Data{1, {1, 2}};
storage.restore_authorized = true;
storage.backup_valid = true;
if (fault == 0) storage.restore_authorized = false;
if (fault == 1) storage.backup_valid = false;
if (fault == 2) storage.backup->schema = 2;
if (fault == 3) storage.backup.reset();
model::Coordinator coordinator(storage);
coordinator.start("trial failed after data change");
finish(storage);
expect(storage.phase == model::Phase::SeparateRecovery && storage.restores == 0 && storage.boots_requested == 0, "Invalid restore accepted");
expect(storage.data.schema == 2 && storage.data.records.size() == 4, "Rejected restore changed live data");
}
model::Storage storage;
storage.data.schema = 2;
storage.backup = model::Data{1, {1, 2}};
storage.backup_valid = storage.restore_authorized = true;
model::Coordinator coordinator(storage);
coordinator.start("trial failed");
coordinator.advance();
storage.restore_authorized = false;
coordinator.advance();
expect(storage.phase == model::Phase::SeparateRecovery && storage.restores == 0, "Restore ignored revoked permission");
}
void target_changes() {
for (bool change_data: {false, true}) {
model::Storage storage;
model::Coordinator coordinator(storage);
coordinator.start("trial failed");
coordinator.advance();
if (change_data) storage.data.schema = 2;
else
storage.window_open = false;
coordinator.advance();
expect(storage.phase == model::Phase::SeparateRecovery && storage.boots_requested == 0, "Changed prerequisites allowed boot handoff");
}
}
}// namespace
int main() {
try {
scenario_outcomes();
reset_boundaries();
admission_and_guards();
restore_guards();
target_changes();
std::cout << "All rollback policy checks passed\n";
} catch (const std::exception &error) {
std::cerr << "Test failed: " << error.what() << '\n';
return 1;
}
}
+3 -1
View File
@@ -22,7 +22,9 @@ This repository is not intended to be a production-ready framework. Instead, it
The repository is being developed incrementally as a structured series of articles, diagrams, architecture decisions, and practical examples. The repository is being developed incrementally as a structured series of articles, diagrams, architecture decisions, and practical examples.
Start with the [series overview](docs/00-series-overview.md). The current practical material includes [PlantUML diagrams](diagrams/readme.md), an [A/B trial boot simulation](examples/simulations/ab-trial-boot/readme.md), and an [interrupted installation model](examples/simulations/interrupted-installation/readme.md). [Chapter 06: Atomic Updates](docs/06-atomic-updates.md) connects preparation, activation, and recovery after interruption. Start with the [series overview](docs/00-series-overview.md). The current practical material includes [PlantUML diagrams](diagrams/readme.md), an [A/B trial boot simulation](examples/simulations/ab-trial-boot/readme.md), an [interrupted installation model](examples/simulations/interrupted-installation/readme.md), and a [rollback policy simulation](examples/simulations/rollback-policy/readme.md).
[Chapter 06: Atomic Updates](docs/06-atomic-updates.md) connects preparation, activation, and recovery after interruption. [Chapter 07: Rollback and Recovery](docs/07-rollback-and-recovery.md) explains target eligibility, data compatibility, bounded recovery, and failed-release blocks.
See the [changelog](CHANGELOG.md) for dated updates to the chapters, architecture decisions, diagrams, and simulations, with links to the material and related history. See the [changelog](CHANGELOG.md) for dated updates to the chapters, architecture decisions, diagrams, and simulations, with links to the material and related history.