150 lines
13 KiB
Markdown
150 lines
13 KiB
Markdown
# 07. Rollback and Recovery: When the Previous Version Still Works
|
|
|
|
[Chapter 06](06-atomic-updates.md) kept incomplete software out of the boot path and defined how to continue an interrupted installation. A complete update can still fail when it runs.
|
|
|
|
Suppose B starts, changes a shared database, and then fails its checks. Boot control returns to A. A's files are intact, but A cannot read the new database. The slot switch worked; recovery did not.
|
|
|
|
The question for this chapter is: **what must remain usable for the device to recover, and what happens when returning to the previous version is no longer safe?**
|
|
|
|
## 1. Three Different Operations
|
|
|
|
| Operation | What it changes | What it does not establish |
|
|
|---|---|---|
|
|
| System rollback | Selects a retained system release, including its matching boot artifacts | That its data is compatible or its services work |
|
|
| Package downgrade | Reinstalls an older package on the current platform | That dependencies, scripts, and data can be restored |
|
|
| Recovery | Restores the required device functions through a defined procedure | That returning to an old release is always the right procedure |
|
|
|
|
Rollback and downgrade are possible parts of recovery. Recovery may instead require a separate environment, a repair procedure, or physical service.
|
|
|
|
We keep the A/B example from earlier chapters. A was confirmed before B's trial. Boot control owns slot selection and attempts. A separate supervisor coordinates local checks and recovery. Its records remain outside the replaceable slots. These roles need not be separate executables.
|
|
|
|
## 2. Decide Which Failure Starts Recovery
|
|
|
|
The trial rules from [ADR-014](../adr/ADR-014-trial-boot-and-confirmation.md) still apply:
|
|
|
|
- If B reaches local checks and fails them, request recovery without spending every remaining trial attempt.
|
|
- If B cannot reach the checks, saved attempt limits let boot control return to A on a later boot.
|
|
- If B hangs, a watchdog or another defined mechanism must cause that later boot.
|
|
|
|
Losing the server connection alone is not a failed local health check in this design. Recovery and its checks work offline; reporting can wait.
|
|
|
|
A failure after B has been confirmed is a different case. The trial may have ended, A may have been replaced, or shared data may have changed. Do not silently restart the old trial policy. Reassess the available recovery target under the product's current policy.
|
|
|
|
For an early boot failure, Linux cannot run a compatibility check before choosing A. Before allowing B's trial, the system must establish that the retained software and permitted trial data changes preserve this recovery path. A runtime supervisor can perform additional checks when it runs, but cannot replace that admission rule.
|
|
|
|
## 3. A Previous Release Must Still Be Eligible
|
|
|
|
Before relying on A, establish:
|
|
|
|
1. Its release identity and matching boot artifacts are known, retained, and valid under the platform's verification policy.
|
|
2. Its required shared components and device configuration remain compatible.
|
|
3. It can use the current data, or an approved restore procedure can prepare compatible data.
|
|
4. The rollback window is still open and the platform's security policy permits this release.
|
|
5. The local recovery mechanism and a finite attempt budget are available.
|
|
|
|
These conditions are a recovery contract. A record saying that A passed checks last week does not prove them today. Retain the artifacts, dependencies, and verification information that the contract requires, rather than relying on a later download.
|
|
|
|
The contract must also cover update-state formats shared by A and B. Old software that cannot interpret the recovery record is not a usable recovery target.
|
|
|
|
[ADR-020: Define Rollback Eligibility and Retention](../adr/ADR-020-define-rollback-eligibility-and-retention.md) defines this choice. The following decision diagram shows the ordinary rollback path and where it must stop.
|
|
|
|
|
|
|
|
[Rollback policy — PlantUML source](../diagrams/plantuml/rollback-policy.puml)
|
|
|
|
## 4. Recovering Software and Data Together
|
|
|
|
The [storage chapter](05-storage-layout.md) separated device data from system slots. That preserves data when an image is replaced, but both versions still need to understand it.
|
|
|
|
| Approach | Benefit | Cost or limit |
|
|
|---|---|---|
|
|
| Keep a format usable by A and B | Return to A can retain records written during B's trial | Both readers and writers must preserve compatibility |
|
|
| Keep a separate trial copy | B's changes can be isolated from A's copy | Extra space; new records need an explicit merge or loss policy |
|
|
| Restore a validated backup | Can recover a compatible earlier dataset | Restore needs interruption handling; newer records may be lost |
|
|
| Delay an incompatible migration | Preserves the old recovery path during trial | B must work before that migration; the rollback window must end explicitly |
|
|
|
|
Compatibility includes write behavior. B must not merely read A's format; its writes must leave data A can still use. Configuration changes need the same review.
|
|
|
|
For the shared-data reference design, keep data compatible during trial. Reject or defer a release that needs an incompatible change before confirmation and has no supported recovery path. [ADR-021](../adr/ADR-021-preserve-data-compatibility-during-trial.md) records this decision.
|
|
|
|
A backup is an alternative, not an automatic escape hatch. Suppose it contains measurements 1 and 2, while the live database contains 1 through 4. Restoring it discards measurements 3 and 4 unless another mechanism preserves them. Product policy must authorize that loss before the restore. A copy also needs a valid format, consistency checks, sufficient space, and a tested restore procedure.
|
|
|
|
An incompatible migration cannot normally be postponed until after a failed early boot and then undone by an unavailable Linux supervisor. If restoring data is necessary before A can start, a recovery environment capable of doing that work must run first.
|
|
|
|
## 5. Execute Recovery as a Saved Procedure
|
|
|
|
A possible procedure after a detected failure is:
|
|
|
|
1. Save the failed release identity, reason, recovery target, remaining budget, and a block against automatically retrying the failed release.
|
|
2. Exclude a competing installation while recovery owns the target and data.
|
|
3. Check target eligibility. Prepare compatible data through the approved procedure, if needed.
|
|
4. Ask boot control to select A under its durable-state contract. Account for each recovery boot attempt before handing control to A.
|
|
5. Start A and check the required local functions within a defined time limit.
|
|
6. Save either recovered or recovery failed, the running release, any data loss, and a pending report.
|
|
|
|
A selected slot is not a successful recovery. Only the checks establish that the required functions have returned.
|
|
|
|
|
|
|
|
[Recovery sequence — PlantUML source](../diagrams/plantuml/rollback-recovery-sequence.puml)
|
|
|
|
Boot control remains authoritative about boot selection and attempts. The supervisor's journal must not override it after a reset. If selection succeeded but saving local progress did not, read the saved boot state and reconcile the journal, following [ADR-019](../adr/ADR-019-recover-interrupted-updates-from-durable-state.md).
|
|
|
|
Give every restore, package operation, and state transition an interruption rule. After a reset, determine whether it completed before repeating it. In particular, do not restore an old backup again after normal operation has resumed and written new data. Do not replenish an attempt budget merely because a coordinator restarted.
|
|
|
|
For an independent service, use the package manager's repair and downgrade rules from [ADR-015](../adr/ADR-015-service-checks-and-package-downgrade.md). Keep its supervisor usable when the service fails. A package command completing still needs service checks afterward.
|
|
|
|
## 6. Stop Repeated Failures
|
|
|
|
After recovering A, a normal update check may discover B again. Without a saved block, the device can repeat the same failed update forever.
|
|
|
|
In this example, the block identifies the exact failed system release on this device, survives reset, and applies to every automatic installation path. Rediscovering B does not remove it. A different release can be considered through the normal admission checks. An authorized operator can clear the block after review, with the action recorded. Use an unambiguous release identity rather than a reusable display name.
|
|
|
|
If A fails its checks, cannot start within the recovery budget, or is ineligible, stop the ordinary A/B path. Do not keep alternating A and B. Record the available diagnosis and enter the platform's separate recovery policy.
|
|
|
|
That policy must identify what can actually run: for example, a recovery environment with independent boot support, or a physical service path. A third partition sharing the same broken boot code does not by itself solve the failure. Define whether the device stops affected functions, what diagnostics remain available, and how service is requested.
|
|
|
|
[ADR-022: Bound Recovery and Block Failed System Releases](../adr/ADR-022-bound-recovery-and-block-failed-system-releases.md) records these rules. This chapter defines the boundary; it does not implement a rescue image or hardware recovery mechanism.
|
|
|
|
## 7. Define When the Rollback Window Ends
|
|
|
|
**Confirmation** accepts B after its required checks. **Closing the rollback window** ends the promise that the device can return to A. **Reusing A's slot** removes the retained software. These are separate events, although a product may deliberately coordinate them.
|
|
|
|
If the product promises rollback after confirmation, it must continue preserving A and compatible data for that period. Otherwise it can explicitly close the window after confirmation, then perform an incompatible migration or reuse the slot under the next update's policy. Before either destructive action, save that A is no longer an ordinary recovery choice. If the transition is interrupted, a conservative closed window is safer than promising an unavailable target.
|
|
|
|
|
|
|
|
[Rollback window — PlantUML source](../diagrams/plantuml/rollback-window.puml)
|
|
|
|
Recovery selection must also respect the platform's security policy. A local failure block prevents repeating a failed deployment; a security anti-rollback rule restricts which older software may run. Their purposes differ. Signature verification and security version policy follow in the security chapter.
|
|
|
|
## 8. A C++ Model of Recovery Policy
|
|
|
|
The [rollback policy example](../examples/simulations/rollback-policy/readme.md) supplies software eligibility and health-check outcomes to a small C++17 state machine. It models shared data as a schema number and a list of record IDs, with an optional backup.
|
|
|
|
It covers successful rollback, incompatible data without restore permission, an approved restore with explicit data loss, reset during recovery, failure of A, rediscovery of blocked B, attempt exhaustion, a closed rollback window, and an unavailable target.
|
|
|
|
The existing [trial boot example](../examples/simulations/ab-trial-boot/readme.md) explains boot mechanics. The new model explains recovery decisions and data consequences. They are separate programs. The backup scenarios deliberately explore an alternative to ADR-021's trial policy; they do not authorize incompatible trial writes in the shared-data reference design.
|
|
|
|
Each modeled transition is assumed to finish durably. Reset points occur between transitions. The model does not implement real backup restoration, a bootloader, package operations, signatures, storage flushing, watchdogs, or interruption within a write. A boolean eligibility input is an assumption, not an implementation of release verification.
|
|
|
|
## 9. Check the Design Against Failures
|
|
|
|
| Failure | Required result |
|
|
|---|---|
|
|
| B fails checks; A and data remain usable | Return to A, check it, save recovery success, keep B blocked |
|
|
| B cannot reach the supervisor | Boot control and the reset path enforce the admitted trial recovery contract |
|
|
| A cannot read current data | Use an approved restore path or separate recovery; do not start A on incompatible data |
|
|
| Backup restore would lose unapproved data | Stop ordinary rollback without changing the live data |
|
|
| Reset during recovery | Resume from authoritative saved state without refreshing the budget |
|
|
| A also fails or exhausts attempts | Enter separate recovery; do not alternate slots indefinitely |
|
|
| B is offered again after reset | Preserve and enforce the failed-release block |
|
|
| Rollback window closed or A replaced | Reject A as an ordinary recovery target |
|
|
| Device is offline | Perform local checks and recovery; retain the report for later |
|
|
|
|
Model checks establish policy behavior under supplied assumptions. Platform tests must also interrupt actual restore and boot-state writes, force startup hangs, and exercise the separate recovery path.
|
|
|
|
## 10. What Comes Next?
|
|
|
|
Recovery needs both an available target and permission to use it. Next, we will examine update integrity, signatures, trust, and the security policy that determines which releases a device may accept.
|