ota-reference-design/docs/07-rollback-and-recovery.md

07. Rollback and Recovery: When the Previous Version Still Works

Chapter 06 kept incomplete software out of the boot path and defined how to continue an interrupted installation. A complete update can still fail when it runs.

Suppose B starts, changes a shared database, and then fails its checks. Boot control returns to A. A's files are intact, but A cannot read the new database. The slot switch worked; recovery did not.

The question for this chapter is: what must remain usable for the device to recover, and what happens when returning to the previous version is no longer safe?

1. Three Different Operations

Operation	What it changes	What it does not establish
System rollback	Selects a retained system release, including its matching boot artifacts	That its data is compatible or its services work
Package downgrade	Reinstalls an older package on the current platform	That dependencies, scripts, and data can be restored
Recovery	Restores the required device functions through a defined procedure	That returning to an old release is always the right procedure

Rollback and downgrade are possible parts of recovery. Recovery may instead require a separate environment, a repair procedure, or physical service.

We keep the A/B example from earlier chapters. A was confirmed before B's trial. Boot control owns slot selection and attempts. A separate supervisor coordinates local checks and recovery. Its records remain outside the replaceable slots. These roles need not be separate executables.

2. Decide Which Failure Starts Recovery

The trial rules from ADR-014 still apply:

• If B reaches local checks and fails them, request recovery without spending every remaining trial attempt.
• If B cannot reach the checks, saved attempt limits let boot control return to A on a later boot.
• If B hangs, a watchdog or another defined mechanism must cause that later boot.

Losing the server connection alone is not a failed local health check in this design. Recovery and its checks work offline; reporting can wait.

A failure after B has been confirmed is a different case. The trial may have ended, A may have been replaced, or shared data may have changed. Do not silently restart the old trial policy. Reassess the available recovery target under the product's current policy.

For an early boot failure, Linux cannot run a compatibility check before choosing A. Before allowing B's trial, the system must establish that the retained software and permitted trial data changes preserve this recovery path. A runtime supervisor can perform additional checks when it runs, but cannot replace that admission rule.

3. A Previous Release Must Still Be Eligible

Before relying on A, establish:

1. Its release identity and matching boot artifacts are known, retained, and valid under the platform's verification policy.
2. Its required shared components and device configuration remain compatible.
3. It can use the current data, or an approved restore procedure can prepare compatible data.
4. The rollback window is still open and the platform's security policy permits this release.
5. The local recovery mechanism and a finite attempt budget are available.

These conditions are a recovery contract. A record saying that A passed checks last week does not prove them today. Retain the artifacts, dependencies, and verification information that the contract requires, rather than relying on a later download.

The contract must also cover update-state formats shared by A and B. Old software that cannot interpret the recovery record is not a usable recovery target.

ADR-020: Define Rollback Eligibility and Retention defines this choice. The following decision diagram shows the ordinary rollback path and where it must stop.

Rollback policy — PlantUML source

4. Recovering Software and Data Together

The storage chapter separated device data from system slots. That preserves data when an image is replaced, but both versions still need to understand it.

Approach	Benefit	Cost or limit
Keep a format usable by A and B	Return to A can retain records written during B's trial	Both readers and writers must preserve compatibility
Keep a separate trial copy	B's changes can be isolated from A's copy	Extra space; new records need an explicit merge or loss policy
Restore a validated backup	Can recover a compatible earlier dataset	Restore needs interruption handling; newer records may be lost
Delay an incompatible migration	Preserves the old recovery path during trial	B must work before that migration; the rollback window must end explicitly

Compatibility includes write behavior. B must not merely read A's format; its writes must leave data A can still use. Configuration changes need the same review.

For the shared-data reference design, keep data compatible during trial. Reject or defer a release that needs an incompatible change before confirmation and has no supported recovery path. ADR-021 records this decision.

A backup is an alternative, not an automatic escape hatch. Suppose it contains measurements 1 and 2, while the live database contains 1 through 4. Restoring it discards measurements 3 and 4 unless another mechanism preserves them. Product policy must authorize that loss before the restore. A copy also needs a valid format, consistency checks, sufficient space, and a tested restore procedure.

An incompatible migration cannot normally be postponed until after a failed early boot and then undone by an unavailable Linux supervisor. If restoring data is necessary before A can start, a recovery environment capable of doing that work must run first.

5. Execute Recovery as a Saved Procedure

A possible procedure after a detected failure is:

1. Save the failed release identity, reason, recovery target, remaining budget, and a block against automatically retrying the failed release.
2. Exclude a competing installation while recovery owns the target and data.
3. Check target eligibility. Prepare compatible data through the approved procedure, if needed.
4. Ask boot control to select A under its durable-state contract. Account for each recovery boot attempt before handing control to A.
5. Start A and check the required local functions within a defined time limit.
6. Save either recovered or recovery failed, the running release, any data loss, and a pending report.

A selected slot is not a successful recovery. Only the checks establish that the required functions have returned.

Recovery sequence — PlantUML source

Boot control remains authoritative about boot selection and attempts. The supervisor's journal must not override it after a reset. If selection succeeded but saving local progress did not, read the saved boot state and reconcile the journal, following ADR-019.

Give every restore, package operation, and state transition an interruption rule. After a reset, determine whether it completed before repeating it. In particular, do not restore an old backup again after normal operation has resumed and written new data. Do not replenish an attempt budget merely because a coordinator restarted.

For an independent service, use the package manager's repair and downgrade rules from ADR-015. Keep its supervisor usable when the service fails. A package command completing still needs service checks afterward.

6. Stop Repeated Failures

After recovering A, a normal update check may discover B again. Without a saved block, the device can repeat the same failed update forever.

In this example, the block identifies the exact failed system release on this device, survives reset, and applies to every automatic installation path. Rediscovering B does not remove it. A different release can be considered through the normal admission checks. An authorized operator can clear the block after review, with the action recorded. Use an unambiguous release identity rather than a reusable display name.

If A fails its checks, cannot start within the recovery budget, or is ineligible, stop the ordinary A/B path. Do not keep alternating A and B. Record the available diagnosis and enter the platform's separate recovery policy.

That policy must identify what can actually run: for example, a recovery environment with independent boot support, or a physical service path. A third partition sharing the same broken boot code does not by itself solve the failure. Define whether the device stops affected functions, what diagnostics remain available, and how service is requested.

ADR-022: Bound Recovery and Block Failed System Releases records these rules. This chapter defines the boundary; it does not implement a rescue image or hardware recovery mechanism.

7. Define When the Rollback Window Ends

Confirmation accepts B after its required checks. Closing the rollback window ends the promise that the device can return to A. Reusing A's slot removes the retained software. These are separate events, although a product may deliberately coordinate them.

If the product promises rollback after confirmation, it must continue preserving A and compatible data for that period. Otherwise it can explicitly close the window after confirmation, then perform an incompatible migration or reuse the slot under the next update's policy. Before either destructive action, save that A is no longer an ordinary recovery choice. If the transition is interrupted, a conservative closed window is safer than promising an unavailable target.

Rollback window — PlantUML source

Recovery selection must also respect the platform's security policy. A local failure block prevents repeating a failed deployment; a security anti-rollback rule restricts which older software may run. Their purposes differ. Signature verification and security version policy follow in the security chapter.

8. A C++ Model of Recovery Policy

The rollback policy example supplies software eligibility and health-check outcomes to a small C++17 state machine. It models shared data as a schema number and a list of record IDs, with an optional backup.

It covers successful rollback, incompatible data without restore permission, an approved restore with explicit data loss, reset during recovery, failure of A, rediscovery of blocked B, attempt exhaustion, a closed rollback window, and an unavailable target.

The existing trial boot example explains boot mechanics. The new model explains recovery decisions and data consequences. They are separate programs. The backup scenarios deliberately explore an alternative to ADR-021's trial policy; they do not authorize incompatible trial writes in the shared-data reference design.

Each modeled transition is assumed to finish durably. Reset points occur between transitions. The model does not implement real backup restoration, a bootloader, package operations, signatures, storage flushing, watchdogs, or interruption within a write. A boolean eligibility input is an assumption, not an implementation of release verification.

9. Check the Design Against Failures

Failure	Required result
B fails checks; A and data remain usable	Return to A, check it, save recovery success, keep B blocked
B cannot reach the supervisor	Boot control and the reset path enforce the admitted trial recovery contract
A cannot read current data	Use an approved restore path or separate recovery; do not start A on incompatible data
Backup restore would lose unapproved data	Stop ordinary rollback without changing the live data
Reset during recovery	Resume from authoritative saved state without refreshing the budget
A also fails or exhausts attempts	Enter separate recovery; do not alternate slots indefinitely
B is offered again after reset	Preserve and enforce the failed-release block
Rollback window closed or A replaced	Reject A as an ordinary recovery target
Device is offline	Perform local checks and recovery; retain the report for later

Model checks establish policy behavior under supplied assumptions. Platform tests must also interrupt actual restore and boot-state writes, force startup hangs, and exercise the separate recovery path.

10. What Comes Next?

Recovery needs both an available target and permission to use it. Next, we will examine update integrity, signatures, trust, and the security policy that determines which releases a device may accept.