60 lines
3.7 KiB
Markdown
60 lines
3.7 KiB
Markdown
# ADR-022: Bound Recovery and Block Failed System Releases
|
|
|
|
## Context
|
|
|
|
Returning to A can fail, and a server may offer failed B again after a successful return. Restarting a coordinator must not give either path unlimited new attempts. The device must recover locally and retain results while offline.
|
|
|
|
Boot control owns boot selection and attempt accounting. A supervisor coordinates recovery checks and keeps policy records outside the slots.
|
|
|
|
## Decision
|
|
|
|
Save the failed system release's unambiguous identity and reason, its automatic retry block, and recovery intent before recovery actions. Apply the block on every automatic installation path. It survives reset and rediscovery of the same release. A different release is subject to normal admission checks. An authorized operator may clear the block after review, with the action recorded.
|
|
|
|
Use a finite recovery budget and a defined check deadline. Boot control accounts for each recovery boot attempt durably before handing control to the target. A restart does not refresh this budget. The reset mechanism must cover hangs that would otherwise prevent progress.
|
|
|
|
Reconcile the supervisor's progress with authoritative boot state after interruption. Exclude competing installs while recovery owns the target and data. Report recovery success only after the restored system passes its essential local checks.
|
|
|
|
If the target is ineligible, checks fail, or the recovery budget expires, stop the ordinary A/B path. Enter a separately defined recovery policy rather than alternating releases. The platform must specify the available recovery environment or physical service path, behavior of affected functions, and diagnostics that remain possible.
|
|
|
|
Retain the failed release, failure reason, recovery result, running release, active block, and any approved data loss for later reporting. Losing connectivity does not prevent local recovery.
|
|
|
|
## Alternatives Considered
|
|
|
|
### Retry Until One Slot Works
|
|
|
|
Can leave a device switching indefinitely without providing its required functions or reaching a diagnosable terminal state.
|
|
|
|
### Let Rediscovery Clear the Failure
|
|
|
|
Normal polling could immediately repeat a known failing installation. A new offer alone is not evidence that the failure was corrected.
|
|
|
|
### Block All Future Releases
|
|
|
|
May fit stricter products, but prevents an otherwise acceptable corrective release from being considered. This example blocks the exact failed release on this device.
|
|
|
|
## Consequences
|
|
|
|
### Positive
|
|
|
|
- Failed updates cannot automatically repeat without policy intervention.
|
|
- Recovery has a finite outcome and continues offline.
|
|
- Reports distinguish an update failure from a failed recovery.
|
|
|
|
### Negative
|
|
|
|
- Boot control and supervisor records need coordination and compatible formats.
|
|
- Retry limits, check deadlines, and watchdog coverage need platform validation.
|
|
- A separate recovery path and a block-removal process must be maintained.
|
|
|
|
## Applicability and Limits
|
|
|
|
Use for automatic system rollback with locally retained state. This extends the package retry-block pattern from ADR-015 to system release identities. It does not implement a rescue environment or a fleet rollout pause. The block is operational policy; it does not replace signature verification or security anti-rollback rules.
|
|
|
|
## Related Documents
|
|
|
|
- [Chapter 07: Rollback and Recovery](../docs/07-rollback-and-recovery.md)
|
|
- [ADR-014: Trial Boot and Confirmation](ADR-014-trial-boot-and-confirmation.md)
|
|
- [ADR-015: Service Checks and Package Downgrade](ADR-015-service-checks-and-package-downgrade.md)
|
|
- [ADR-019: Recover Interrupted Updates From Durable State](ADR-019-recover-interrupted-updates-from-durable-state.md)
|
|
- [ADR-020: Define Rollback Eligibility and Retention](ADR-020-define-rollback-eligibility-and-retention.md)
|