From eb363ec42d2f5637a7b5c0f1e5ecf9830ff2c5f9 Mon Sep 17 00:00:00 2001 From: deeaitch Date: Mon, 5 Oct 2026 18:01:02 -0400 Subject: [PATCH] Ref #1. 04. Starting an Updated System or Service --- adr/ADR-005-package-based-updates.md | 8 +- adr/ADR-014-trial-boot-and-confirmation.md | 77 ++++++++ ...15-service-checks-and-package-downgrade.md | 94 +++++++++ docs/00-series-overview.md | 21 +- docs/03-system-architecture.md | 2 +- ...4-starting-an-updated-system-or-service.md | 186 ++++++++++++++++++ 6 files changed, 379 insertions(+), 9 deletions(-) create mode 100644 adr/ADR-014-trial-boot-and-confirmation.md create mode 100644 adr/ADR-015-service-checks-and-package-downgrade.md create mode 100644 docs/04-starting-an-updated-system-or-service.md diff --git a/adr/ADR-005-package-based-updates.md b/adr/ADR-005-package-based-updates.md index dd8ff3b..93c1c4b 100644 --- a/adr/ADR-005-package-based-updates.md +++ b/adr/ADR-005-package-based-updates.md @@ -90,7 +90,7 @@ The package manager is responsible for: - verifying signatures and integrity; - executing installation scripts; - replacing files; -- retaining previously installed versions. +- retaining previous packages when configured to do so, or working with a separate local package store. The OTA manager is responsible for update policy, including when updates are allowed, network restrictions, user approval, and operational conditions. @@ -98,9 +98,11 @@ Using a package manager alone does not constitute a complete OTA solution. ### Component Rollback -Each component can be upgraded and downgraded independently. +An independent component can be restored without reverting the entire system when the package manager supports downgrade and the previous version remains compatible with its dependencies, configuration, and data. -Most package managers retain previously installed versions, allowing an individual component to be rolled back without reverting the entire system. +The update design must keep the previous package available. Automatic retention must be checked for the selected package manager; it cannot be assumed. + +[ADR-015: Service Checks and Package Downgrade](ADR-015-service-checks-and-package-downgrade.md) describes a recovery policy for a service that fails after an update. It covers startup checks, an explicit downgrade, a saved block against retrying the failed version, and reporting. That example requires local retention of the previous package and verified downgrade support. ## Limitations diff --git a/adr/ADR-014-trial-boot-and-confirmation.md b/adr/ADR-014-trial-boot-and-confirmation.md new file mode 100644 index 0000000..8951da1 --- /dev/null +++ b/adr/ADR-014-trial-boot-and-confirmation.md @@ -0,0 +1,77 @@ +# ADR-014: Trial Boot and Confirmation for System Updates + +* **Status:** Proposed +* **Date:** 2026-10-05 + +## Context + +A new system image can install correctly and still fail to start or run the main application. If the new system cannot start the OTA manager, that manager cannot request recovery. + +This example assumes that the device can keep a previous working system while preparing a new one. Its boot control can select either system and read saved update state. An A/B layout, with two system slots, is one way to meet these conditions. + +The device can check its essential functions locally. Installation, startup checks, and recovery must not require a continuous server connection. + +## Decision + +Start a new system as a trial. Keep the previous working system available until the new system passes its required checks. + +Boot control records each trial attempt before handing control to the new system. Allow a limited number of unconfirmed attempts. The platform design must set this limit and explain how an interrupted state write is handled. + +Once the new system starts, local health checks test its essential functions within a defined time limit. Linux starting alone is not enough. If the checks pass, save confirmation and make the new system the normal boot choice. + +If the checks fail, request a return to the previous system. If the new system never reaches the checks, boot control must select the previous system after the trial attempts are used up. + +A hang must lead to a reset through a watchdog or another defined mechanism. A watchdog is a timer that can reset the device when software stops showing that it is alive. Its coverage must include the startup stages where the design requires automatic recovery. + +After returning to the previous system, check its essential functions too. Save the outcome and report it when a connection is available. If the previous system also fails, follow a separately defined recovery policy rather than endlessly switching between versions. + +## Decision Drivers + +- The device must recover even when the new system cannot start the OTA manager. +- A successful installation does not prove that the system works. +- Trial attempts must end within a defined limit. +- Local recovery must remain possible while the device is offline. + +## Alternatives Considered + +### Select the New System Permanently After Installation + +This needs less boot state, but installation checks cannot detect all startup and application failures. It does not provide the required automatic return to a working system. + +### Let the OTA Manager Handle Every Failure + +The manager can handle failures after it starts. It cannot handle a failure that prevents its own startup. + +### Wait for Server Confirmation + +Server approval may be useful for some products. It is not used here because these devices can check their essential functions locally and must recover without a server connection. + +## Consequences + +### Positive + +- Early boot failures can trigger recovery without the OTA manager. +- The device keeps a previous system until the new one passes its checks. +- Offline devices can finish the trial and report later. + +### Negative + +- Boot control, health checks, and reset behavior must work together. +- Saved attempt and confirmation state must survive power loss. +- Storage is needed for the previous system. +- Poor checks can accept a broken system or reject a working one. +- The previous system may fail if the new version changed shared data. + +## Applicability and Limits + +Use this approach when the platform supports selection of a retained system, reliable boot state, and a reset path for the failures being covered. + +Retaining an old image does not guarantee recovery from damaged storage, failure of shared boot components, or incompatible data changes. Those cases need their own recovery design. + +The platform must define the attempt limit, check time limit, confirmation owner, watchdog coverage, and behavior when recovery also fails. The exact storage format and bootloader commands belong in the platform example. + +## Related Documents + +- [Chapter 04: Starting an Updated System or Service](../docs/04-starting-an-updated-system-or-service.md) +- [ADR-004: Full-System Image Updates](ADR-004-full-system-image-updates.md) +- [ADR-015: Service Checks and Package Downgrade](ADR-015-service-checks-and-package-downgrade.md) diff --git a/adr/ADR-015-service-checks-and-package-downgrade.md b/adr/ADR-015-service-checks-and-package-downgrade.md new file mode 100644 index 0000000..16199ef --- /dev/null +++ b/adr/ADR-015-service-checks-and-package-downgrade.md @@ -0,0 +1,94 @@ +# ADR-015: Service Checks and Package Downgrade + +* **Status:** Proposed +* **Date:** 2026-10-05 + +## Context + +An independently updated service can fail after its new package is installed. It may crash, fail to become ready, or keep running without doing its job. Restarting it repeatedly may leave the device without an essential function. + +This example uses a stable system platform and a service that can be updated independently. A separate supervisor can control installation and check the service. The supervisor remains able to run when the updated service fails. + +Before an update starts, the device has the exact previous package available locally, with the information needed to verify it. The package manager supports installing that older version. Dependencies, installation scripts, configuration, and service data have been checked for downgrade compatibility. + +## Decision + +After installing the new package, start the service and check its required functions within a defined time limit. Keep the new version only if these checks pass. + +If the service fails, save a block against another automatic installation of that failed version. Then stop the service, explicitly reinstall the previous package, start it, and check it again. + +This explicit request to install an older version is a **forced downgrade**. It overrides the normal preference for a newer version. It does not skip verification or compatibility checks. + +Save recovery progress so the supervisor can decide what to do after a restart. The supervisor must handle an interrupted installation or downgrade according to the package manager's recovery rules. Automatic updates must not start a competing installation while recovery is in progress. + +If the previous service passes its checks, record that recovery succeeded. If installation, downgrade, or the checks fail, record the failed recovery and follow the device's recovery policy. Do not treat a completed package command as proof that the service works. + +### Update Lock + +Here, a **lock** means a saved rule that prevents another automatic attempt at the failed package version. It is separate from a temporary package-manager lock used during installation. + +The rule applies to the failed version of this package on this device. It survives a restart and is checked on every automatic update path. Discovering the same release again does not remove it. It does not block unrelated packages or all future versions of this package. + +A different version may be tried after the usual checks. An authorized operator may remove the rule to retry the same version after reviewing the failure. Removal must be recorded. A server may use device reports to pause a wider deployment, but that is a separate deployment decision. + +### Reporting + +Save the failed package version, the reason for failure, the recovery result, the version now running, and the active update lock. Send the report when a connection is available. + +If the failed service provides network access, local recovery still runs. Reporting may have to wait until communication is restored. + +## Decision Drivers + +- A package installation can succeed while the updated service fails. +- The device needs a local way to restore an essential service. +- The same failed version must not be installed again automatically. +- Operators need to distinguish update failure from recovery failure. + +## Alternatives Considered + +### Keep Restarting the New Service + +Restarts can help with a temporary fault. They do not fix a repeatable error in the new version. This design limits startup checks and restores the previous version when they fail. + +### Restore the Package Without Blocking the Failed Version + +The next automatic check could offer the same version and repeat the failure. A saved block prevents that cycle. + +### Hold the Package at the Previous Version + +Blocking all later versions until review can fit products with stricter approval rules. This example blocks only the failed version so a later fix can be considered through the normal update process. + +### Restore the Entire System + +A system rollback may fit tightly coupled software. It is not selected here because this service is an independent update unit on a stable platform. + +## Consequences + +### Positive + +- The device can restore the service without replacing the whole system. +- Recovery can run without a server connection. +- A restart does not remove the failure block. +- Reports show whether the previous service was actually restored. + +### Negative + +- The previous package and its verification information need local storage. +- The supervisor, checks, saved state, and package manager must work together. +- Downgrade paths need testing, including scripts and data changes. +- Locks need explicit removal rules and operator visibility. +- Service downtime includes the failed attempt and recovery. + +## Applicability and Limits + +Use this approach for independently updated services when the platform and package manager support a tested downgrade path. + +It does not make package replacement atomic or undo every action of an installation script. Restoring software does not automatically restore data. If the old version cannot use the resulting data, this recovery path is insufficient. + +The product must define the service checks, their time limit, the supervisor's dependencies, and what happens if recovery fails. Package retention and downgrade support must be verified for the selected tools. + +## Related Documents + +- [Chapter 04: Starting an Updated System or Service](../docs/04-starting-an-updated-system-or-service.md) +- [ADR-005: Package-Based Updates](ADR-005-package-based-updates.md) +- [ADR-014: Trial Boot and Confirmation for System Updates](ADR-014-trial-boot-and-confirmation.md) diff --git a/docs/00-series-overview.md b/docs/00-series-overview.md index 0b26080..4754de9 100644 --- a/docs/00-series-overview.md +++ b/docs/00-series-overview.md @@ -71,13 +71,17 @@ The emphasis is always on understanding the underlying design rather than memori ## Repository Structure +The structure below shows the planned repository. The `diagrams/` and `examples/` directories will be added as the practical material develops. + ```text ota-reference-design/ │ ├── docs/ │ ├── 00-series-overview.md -│ ├── 01-ota-introduction.md -│ ├── 02-update-strategies.md +│ ├── 01-what-is-an-ota-update.md +│ ├── 02-update_strategies.md +│ ├── 03-system-architecture.md +│ ├── 04-starting-an-updated-system-or-service.md │ ├── ... │ ├── diagrams/ @@ -93,14 +97,21 @@ ota-reference-design/ │ ├── ADR-002-license.md │ └── ADR-003-reference-design.md │ -└── README.md +└── readme.md ``` --- ## Learning Path -The chapters are designed to build on each other. +The chapters are designed to build on each other. The current chapters are: + +- [01. What Is an OTA Update?](01-what-is-an-ota-update.md) +- [02. Choosing an OTA Update Strategy](02-update_strategies.md) +- [03. OTA System Architecture](03-system-architecture.md) +- [04. Starting an Updated System or Service](04-starting-an-updated-system-or-service.md) + +The learning path below also includes planned chapters. ```text Introduction @@ -112,7 +123,7 @@ Update Strategies System Architecture │ ▼ -Boot Process +System and Service Startup │ ▼ Storage Layout diff --git a/docs/03-system-architecture.md b/docs/03-system-architecture.md index c94e4cd..d388999 100644 --- a/docs/03-system-architecture.md +++ b/docs/03-system-architecture.md @@ -62,4 +62,4 @@ If the update itself is interrupted, the device needs enough saved information t [ADR-011: Device-Initiated Update Checks](../adr/ADR-011-device-initiated-update-checks.md) explains how a device checks for updates at power-on, on a schedule, or at the operator’s request. -Next, we will follow the boot process and see what happens before Linux and the OTA manager start. +Next, we will follow [the startup of an updated system or service](04-starting-an-updated-system-or-service.md). We will look at recovery before the OTA manager starts, and at restoring an older package when an updated service fails. diff --git a/docs/04-starting-an-updated-system-or-service.md b/docs/04-starting-an-updated-system-or-service.md new file mode 100644 index 0000000..380c619 --- /dev/null +++ b/docs/04-starting-an-updated-system-or-service.md @@ -0,0 +1,186 @@ +# 04. Starting an Updated System or Service + +The previous chapter described who prepares, installs, and checks an update. Now we will follow what happens when the new software starts. + +Installing files is only part of the work. The new software must start and do its job before the device decides to keep it. + +We will use two examples: a system update with two system slots, and a package update for one service. These are examples of possible designs. Each needs a recovery mechanism suited to its platform. + +## 1. From Power-On to the Application + +A Linux device starts in several steps. A simple view is: + +```text +Power on or reset + | + v +Early platform code + | + v +Bootloader selects and loads a system + | + v +Linux kernel starts + | + v +Root filesystem becomes available + | + v +Init starts services + | + v +Main application starts +``` + +The exact steps depend on the hardware. Early platform code prepares enough hardware to continue loading software. The **bootloader** loads the operating system and passes control to it. + +The **kernel** is the core of Linux. It manages hardware and provides the basic functions that programs use. The **root filesystem** contains the files and directories needed by the system. **Init** is the first program Linux starts; it arranges the startup of services. A **service** is a program that runs to provide a function, such as communication or diagnostics. + +The OTA manager starts later in this chain. If Linux cannot reach that point, the manager cannot repair the device. Recovery for an early failure must therefore work before the manager starts. + +## 2. Which Software Will Start? + +A system update may change the kernel, the root filesystem, or both. Some platforms also use a **device tree**, a description of hardware that helps the kernel use the device. + +These parts must work together. Selecting a new root filesystem while loading an incompatible kernel can prevent the system from working. + +The design must say which parts belong to a system version and how they are selected together. Later chapters will describe where these parts are stored. + +A package update has a different starting point. Linux is already running. The device replaces an application package and starts the updated service on the existing system platform. It may need a service restart rather than a device reboot. + +## 3. Installed Does Not Mean Working + +Consider four different events: + +| Event | What we know | +|---|---| +| Installed | The installation step has finished. | +| Started | The new system or service has started. | +| Checked | The required checks have passed. | +| Confirmed | The device has decided to keep the new version. | + +A process can start and then crash. It can also keep running while failing to answer requests or read a required sensor. + +A **health check** tests whether the software can do the work the device needs. For example, it might check that the main service answers a local request and can read a required device. + +The checks need a time limit. Some services need time to become ready, but the device must not wait forever. The design must define what to check, how long to wait, and what counts as failure. + +In these examples, the checks can run locally. Losing the server connection alone does not make the new software unhealthy. Reports can wait until the connection returns. + +## 4. Example: Starting a New System in an A/B Layout + +An **A/B layout** provides two places, called slots, for system software. In this example, slot A contains the working version. The installer prepares the new version in slot B while A keeps running. + +The new version first gets a **trial boot**: a chance to start and pass the checks before it becomes the normal choice. + +```text +A is running + | +Install new system in B + | +Select B for a trial boot + | +Restart into B + | +Check essential functions + | + +-- Pass --> Confirm B + | + +-- Fail --> Return to A and check it +``` + +The boot control mechanism limits the number of unconfirmed attempts. If B never reaches the program that confirms it, a later boot must be able to select A again. + +This requires more than a choice between two slots. If B hangs, something must cause another reset. A **watchdog** is a timer that can reset the device when software stops showing that it is alive. Its setup must cover the failures the design intends to recover from. + +The boot attempt state must be saved so a reset cannot erase the fact that an attempt was made. The device also needs a clear record of whether B has been confirmed. + +Returning to a previous version is called **rollback**. Keeping A available makes rollback possible, but does not prove that A will still work. Shared data changed by B may also affect A. + +[ADR-014: Trial Boot and Confirmation for System Updates](../adr/ADR-014-trial-boot-and-confirmation.md) explains when to choose this policy and what the platform must support. + +## 5. Example: Starting an Updated Service + +Now imagine a device with a stable system platform and an independently updated communication service. Version 1 works. Version 2 arrives as a package. + +A separate update supervisor controls this change. It can be part of the OTA manager. It must stay able to run when the communication service fails. + +Before installation, the device keeps the exact version 1 package available locally, with the information needed to verify it. This example assumes that both versions work on the current platform and that the service's data remains usable by version 1. + +```text +Service version 1 is running + | +Install version 2 and restart the service + | +Check that version 2 can do its job + | + +-- Pass --> Keep version 2 and report success + | + +-- Fail --> Block another automatic attempt at version 2 + | + Stop the failed service + | + Reinstall version 1 + | + Start and check version 1 + | + Save and report the result +``` + +Reinstalling an older package is called a **downgrade**. Here, a **forced downgrade** means that the supervisor explicitly requests the previous version, even though the normal update policy prefers newer versions. It still verifies the package and checks compatibility. + +The package manager must support this operation. The supervisor must also account for installation scripts and dependencies. Keeping an old package does not by itself make downgrade safe. + +After recovery, the supervisor applies a **lock** that prevents another automatic installation of the failed version. Here, lock means a saved update rule, not a temporary lock used while writing files. The failure block is saved before recovery starts, so a restart cannot immediately trigger the same failed update again. + +For this example, the rule blocks version 2 of this package on this device. Other designs may hold the package at version 1 until an operator reviews the failure. The scope of the lock and the conditions for removing it must be explicit. + +If version 1 also fails its checks, recovery has failed. The supervisor records that outcome and follows the device's recovery policy. It must not report success just because the downgrade command finished. + +[ADR-005: Package-Based Updates](../adr/ADR-005-package-based-updates.md) describes the conditions under which independent packages fit a stable system platform. + +[ADR-015: Service Checks and Package Downgrade](../adr/ADR-015-service-checks-and-package-downgrade.md) describes the recovery policy used here, including the scope of the update lock and the conditions for removing it. + +## 6. Who Handles Each Failure? + +| Failure | Who handles it in these examples? | +|---|---| +| New system cannot reach Linux services | Boot control uses the saved trial state to select the previous system on a later boot. | +| New system hangs | A watchdog causes a reset; boot control then uses the trial state. | +| Linux starts, but an essential function fails | Health checks request system recovery. | +| Updated service crashes or fails its checks | The separate supervisor downgrades it and blocks the failed version. | +| Previous system or service also fails | The device follows its recovery policy and records that recovery failed. | +| Server connection is lost | Local checks and recovery continue; reporting waits. | + +These mechanisms must be connected deliberately. Two system slots alone do not detect a hang. A service manager that restarts a crashed process does not, by itself, restore the previous package. + +## 7. Remembering What Happened + +Both examples need information that survives a restart. This is called **persistent state**. + +The device needs enough information to answer questions such as: + +- Which version was working before the update? +- Which version is being tried now? +- Has it passed the required checks? +- Has recovery started or finished? +- Is this version blocked from another automatic attempt? +- Which result still needs to be reported? + +Saving this information is part of the design. Power can fail while it is being written. After a restart, the device must be able to read a valid state and decide what to do next. + +A useful report includes the failed version, the reason for failure, the recovery result, the version now running, and any update lock. If the device is offline, it saves the report and sends it later. + +## 8. Recovering Software and Data + +Returning to old software does not automatically restore old data. + +For example, version 2 may change a database into a format that version 1 cannot read. The old system slot or package can still be present, yet recovery can fail. + +The update design must keep data compatible, provide a tested way to restore it, or define another recovery path. The same question applies to configuration and changes made by package installation scripts. + +## 9. What Comes Next? + +Both examples follow the same pattern: start the new version, check its required functions, then keep it or recover. They use different mechanisms because a system boot and a service restart fail in different ways. + +Next, we will look at storage layout: where software versions, device data, and update state live, and what must remain available when an update fails.