5.6 KiB
ADR-015: Service Checks and Package Downgrade
- Status: Proposed
- Date: 2026-10-05
Context
An independently updated service can fail after its new package is installed. It may crash, fail to become ready, or keep running without doing its job. Restarting it repeatedly may leave the device without an essential function.
This example uses a stable system platform and a service that can be updated independently. A separate supervisor can control installation and check the service. The supervisor remains able to run when the updated service fails.
Before an update starts, the device has the exact previous package available locally, with the information needed to verify it. The package manager supports installing that older version. Dependencies, installation scripts, configuration, and service data have been checked for downgrade compatibility.
Decision
After installing the new package, start the service and check its required functions within a defined time limit. Keep the new version only if these checks pass.
If the service fails, save a block against another automatic installation of that failed version. Then stop the service, explicitly reinstall the previous package, start it, and check it again.
This explicit request to install an older version is a forced downgrade. It overrides the normal preference for a newer version. It does not skip verification or compatibility checks.
Save recovery progress so the supervisor can decide what to do after a restart. The supervisor must handle an interrupted installation or downgrade according to the package manager's recovery rules. Automatic updates must not start a competing installation while recovery is in progress.
If the previous service passes its checks, record that recovery succeeded. If installation, downgrade, or the checks fail, record the failed recovery and follow the device's recovery policy. Do not treat a completed package command as proof that the service works.
Update Lock
Here, a lock means a saved rule that prevents another automatic attempt at the failed package version. It is separate from a temporary package-manager lock used during installation.
The rule applies to the failed version of this package on this device. It survives a restart and is checked on every automatic update path. Discovering the same release again does not remove it. It does not block unrelated packages or all future versions of this package.
A different version may be tried after the usual checks. An authorized operator may remove the rule to retry the same version after reviewing the failure. Removal must be recorded. A server may use device reports to pause a wider deployment, but that is a separate deployment decision.
Reporting
Save the failed package version, the reason for failure, the recovery result, the version now running, and the active update lock. Send the report when a connection is available.
If the failed service provides network access, local recovery still runs. Reporting may have to wait until communication is restored.
Decision Drivers
- A package installation can succeed while the updated service fails.
- The device needs a local way to restore an essential service.
- The same failed version must not be installed again automatically.
- Operators need to distinguish update failure from recovery failure.
Alternatives Considered
Keep Restarting the New Service
Restarts can help with a temporary fault. They do not fix a repeatable error in the new version. This design limits startup checks and restores the previous version when they fail.
Restore the Package Without Blocking the Failed Version
The next automatic check could offer the same version and repeat the failure. A saved block prevents that cycle.
Hold the Package at the Previous Version
Blocking all later versions until review can fit products with stricter approval rules. This example blocks only the failed version so a later fix can be considered through the normal update process.
Restore the Entire System
A system rollback may fit tightly coupled software. It is not selected here because this service is an independent update unit on a stable platform.
Consequences
Positive
- The device can restore the service without replacing the whole system.
- Recovery can run without a server connection.
- A restart does not remove the failure block.
- Reports show whether the previous service was actually restored.
Negative
- The previous package and its verification information need local storage.
- The supervisor, checks, saved state, and package manager must work together.
- Downgrade paths need testing, including scripts and data changes.
- Locks need explicit removal rules and operator visibility.
- Service downtime includes the failed attempt and recovery.
Applicability and Limits
Use this approach for independently updated services when the platform and package manager support a tested downgrade path.
It does not make package replacement atomic or undo every action of an installation script. Restoring software does not automatically restore data. If the old version cannot use the resulting data, this recovery path is insufficient.
The product must define the service checks, their time limit, the supervisor's dependencies, and what happens if recovery fails. Package retention and downgrade support must be verified for the selected tools.