Ref #1. 04. Starting an Updated System or Service

2026-10-05 18:01:02 -04:00
1 parent 1ebebb86e2
commit eb363ec42d
@@ -90,7 +90,7 @@ The package manager is responsible for:
 - verifying signatures and integrity;
 - executing installation scripts;
 - replacing files;
- retaining previously installed versions.
+- retaining previous packages when configured to do so, or working with a separate local package store.

 The OTA manager is responsible for update policy, including when updates are allowed, network restrictions, user approval, and operational conditions.

@@ -98,9 +98,11 @@ Using a package manager alone does not constitute a complete OTA solution.

 ### Component Rollback

-Each component can be upgraded and downgraded independently.
+An independent component can be restored without reverting the entire system when the package manager supports downgrade and the previous version remains compatible with its dependencies, configuration, and data.

-Most package managers retain previously installed versions, allowing an individual component to be rolled back without reverting the entire system.
+The update design must keep the previous package available. Automatic retention must be checked for the selected package manager; it cannot be assumed.
+
+[ADR-015: Service Checks and Package Downgrade](ADR-015-service-checks-and-package-downgrade.md) describes a recovery policy for a service that fails after an update. It covers startup checks, an explicit downgrade, a saved block against retrying the failed version, and reporting. That example requires local retention of the previous package and verified downgrade support.

 ## Limitations

@@ -0,0 +1,77 @@
+# ADR-014: Trial Boot and Confirmation for System Updates
+
+* **Status:** Proposed
+* **Date:** 2026-10-05
+
+## Context
+
+A new system image can install correctly and still fail to start or run the main application. If the new system cannot start the OTA manager, that manager cannot request recovery.
+
+This example assumes that the device can keep a previous working system while preparing a new one. Its boot control can select either system and read saved update state. An A/B layout, with two system slots, is one way to meet these conditions.
+
+The device can check its essential functions locally. Installation, startup checks, and recovery must not require a continuous server connection.
+
+## Decision
+
+Start a new system as a trial. Keep the previous working system available until the new system passes its required checks.
+
+Boot control records each trial attempt before handing control to the new system. Allow a limited number of unconfirmed attempts. The platform design must set this limit and explain how an interrupted state write is handled.
+
+Once the new system starts, local health checks test its essential functions within a defined time limit. Linux starting alone is not enough. If the checks pass, save confirmation and make the new system the normal boot choice.
+
+If the checks fail, request a return to the previous system. If the new system never reaches the checks, boot control must select the previous system after the trial attempts are used up.
+
+A hang must lead to a reset through a watchdog or another defined mechanism. A watchdog is a timer that can reset the device when software stops showing that it is alive. Its coverage must include the startup stages where the design requires automatic recovery.
+
+After returning to the previous system, check its essential functions too. Save the outcome and report it when a connection is available. If the previous system also fails, follow a separately defined recovery policy rather than endlessly switching between versions.
+
+## Decision Drivers
+
+- The device must recover even when the new system cannot start the OTA manager.
+- A successful installation does not prove that the system works.
+- Trial attempts must end within a defined limit.
+- Local recovery must remain possible while the device is offline.
+
+## Alternatives Considered
+
+### Select the New System Permanently After Installation
+
+This needs less boot state, but installation checks cannot detect all startup and application failures. It does not provide the required automatic return to a working system.
+
+### Let the OTA Manager Handle Every Failure
+
+The manager can handle failures after it starts. It cannot handle a failure that prevents its own startup.
+
+### Wait for Server Confirmation
+
+Server approval may be useful for some products. It is not used here because these devices can check their essential functions locally and must recover without a server connection.
+
+## Consequences
+
+### Positive
+
+- Early boot failures can trigger recovery without the OTA manager.
+- The device keeps a previous system until the new one passes its checks.
+- Offline devices can finish the trial and report later.
+
+### Negative
+
+- Boot control, health checks, and reset behavior must work together.
+- Saved attempt and confirmation state must survive power loss.
+- Storage is needed for the previous system.
+- Poor checks can accept a broken system or reject a working one.
+- The previous system may fail if the new version changed shared data.
+
+## Applicability and Limits
+
+Use this approach when the platform supports selection of a retained system, reliable boot state, and a reset path for the failures being covered.
+
+Retaining an old image does not guarantee recovery from damaged storage, failure of shared boot components, or incompatible data changes. Those cases need their own recovery design.
+
+The platform must define the attempt limit, check time limit, confirmation owner, watchdog coverage, and behavior when recovery also fails. The exact storage format and bootloader commands belong in the platform example.
+
+## Related Documents
+
+- [Chapter 04: Starting an Updated System or Service](../docs/04-starting-an-updated-system-or-service.md)
+- [ADR-004: Full-System Image Updates](ADR-004-full-system-image-updates.md)
+- [ADR-015: Service Checks and Package Downgrade](ADR-015-service-checks-and-package-downgrade.md)
@@ -0,0 +1,94 @@
+# ADR-015: Service Checks and Package Downgrade
+
+* **Status:** Proposed
+* **Date:** 2026-10-05
+
+## Context
+
+An independently updated service can fail after its new package is installed. It may crash, fail to become ready, or keep running without doing its job. Restarting it repeatedly may leave the device without an essential function.
+
+This example uses a stable system platform and a service that can be updated independently. A separate supervisor can control installation and check the service. The supervisor remains able to run when the updated service fails.
+
+Before an update starts, the device has the exact previous package available locally, with the information needed to verify it. The package manager supports installing that older version. Dependencies, installation scripts, configuration, and service data have been checked for downgrade compatibility.
+
+## Decision
+
+After installing the new package, start the service and check its required functions within a defined time limit. Keep the new version only if these checks pass.
+
+If the service fails, save a block against another automatic installation of that failed version. Then stop the service, explicitly reinstall the previous package, start it, and check it again.
+
+This explicit request to install an older version is a **forced downgrade**. It overrides the normal preference for a newer version. It does not skip verification or compatibility checks.
+
+Save recovery progress so the supervisor can decide what to do after a restart. The supervisor must handle an interrupted installation or downgrade according to the package manager's recovery rules. Automatic updates must not start a competing installation while recovery is in progress.
+
+If the previous service passes its checks, record that recovery succeeded. If installation, downgrade, or the checks fail, record the failed recovery and follow the device's recovery policy. Do not treat a completed package command as proof that the service works.
+
+### Update Lock
+
+Here, a **lock** means a saved rule that prevents another automatic attempt at the failed package version. It is separate from a temporary package-manager lock used during installation.
+
+The rule applies to the failed version of this package on this device. It survives a restart and is checked on every automatic update path. Discovering the same release again does not remove it. It does not block unrelated packages or all future versions of this package.
+
+A different version may be tried after the usual checks. An authorized operator may remove the rule to retry the same version after reviewing the failure. Removal must be recorded. A server may use device reports to pause a wider deployment, but that is a separate deployment decision.
+
+### Reporting
+
+Save the failed package version, the reason for failure, the recovery result, the version now running, and the active update lock. Send the report when a connection is available.
+
+If the failed service provides network access, local recovery still runs. Reporting may have to wait until communication is restored.
+
+## Decision Drivers
+
+- A package installation can succeed while the updated service fails.
+- The device needs a local way to restore an essential service.
+- The same failed version must not be installed again automatically.
+- Operators need to distinguish update failure from recovery failure.
+
+## Alternatives Considered
+
+### Keep Restarting the New Service
+
+Restarts can help with a temporary fault. They do not fix a repeatable error in the new version. This design limits startup checks and restores the previous version when they fail.
+
+### Restore the Package Without Blocking the Failed Version
+
+The next automatic check could offer the same version and repeat the failure. A saved block prevents that cycle.
+
+### Hold the Package at the Previous Version
+
+Blocking all later versions until review can fit products with stricter approval rules. This example blocks only the failed version so a later fix can be considered through the normal update process.
+
+### Restore the Entire System
+
+A system rollback may fit tightly coupled software. It is not selected here because this service is an independent update unit on a stable platform.
+
+## Consequences
+
+### Positive
+
+- The device can restore the service without replacing the whole system.
+- Recovery can run without a server connection.
+- A restart does not remove the failure block.
+- Reports show whether the previous service was actually restored.
+
+### Negative
+
+- The previous package and its verification information need local storage.
+- The supervisor, checks, saved state, and package manager must work together.
+- Downgrade paths need testing, including scripts and data changes.
+- Locks need explicit removal rules and operator visibility.
+- Service downtime includes the failed attempt and recovery.
+
+## Applicability and Limits
+
+Use this approach for independently updated services when the platform and package manager support a tested downgrade path.
+
+It does not make package replacement atomic or undo every action of an installation script. Restoring software does not automatically restore data. If the old version cannot use the resulting data, this recovery path is insufficient.
+
+The product must define the service checks, their time limit, the supervisor's dependencies, and what happens if recovery fails. Package retention and downgrade support must be verified for the selected tools.
+
+## Related Documents
+
+- [Chapter 04: Starting an Updated System or Service](../docs/04-starting-an-updated-system-or-service.md)
+- [ADR-005: Package-Based Updates](ADR-005-package-based-updates.md)
+- [ADR-014: Trial Boot and Confirmation for System Updates](ADR-014-trial-boot-and-confirmation.md)