From 572d1f7acc776d023b8d4c214b4bad52e062e918 Mon Sep 17 00:00:00 2001 From: deeaitch Date: Mon, 21 Sep 2026 13:31:00 -0400 Subject: [PATCH] 03-system-architecture. Initial commit --- ...-011-device-controlled-update-execution.md | 80 ++++++++ ...-012-update-management-and-installation.md | 88 +++++++++ docs/03-system-architecture.md | 177 ++++++++++++++++++ 3 files changed, 345 insertions(+) create mode 100644 adr/ADR-011-device-controlled-update-execution.md create mode 100644 adr/ADR-012-update-management-and-installation.md create mode 100644 docs/03-system-architecture.md diff --git a/adr/ADR-011-device-controlled-update-execution.md b/adr/ADR-011-device-controlled-update-execution.md new file mode 100644 index 0000000..d5369e3 --- /dev/null +++ b/adr/ADR-011-device-controlled-update-execution.md @@ -0,0 +1,80 @@ +# ADR-011: Device-Controlled Update Execution + +* **Status:** Proposed +* **Date:** 2026-09-21 + +## Context + +This educational example considers remotely deployed embedded Linux devices with intermittent connectivity. A backend assigns authorized releases, but a device may lose contact after receiving an assignment or preparing an update. + +The device can evaluate local operating conditions and has a locally available activation and recovery mechanism. Its essential post-update health checks can be evaluated without contacting the deployment service. + +The architectural question is which participant controls execution once the device has accepted an update assignment. Requiring a remote command for every step would make progress and recovery depend on continued connectivity. + +## Decision + +Let the backend manage release assignment and fleet deployment policy. Let the device coordinate the local execution of an accepted update within that authorization and its local operating policy. + +After the required content and verification information are available, local activation, health acceptance, and recovery must not require a live connection to the deployment service. + +Keep enough durable local state to reconcile an interrupted operation with installation and boot state. Report progress and the resulting software state when connectivity permits. Treat backend knowledge as a potentially delayed observation of the device. + +This decision assigns responsibility to the device as a whole. It does not require the userspace update manager to handle failures that occur before it can run. + +## Decision Drivers + +- Connectivity may disappear during an update. +- Local conditions determine when disruptive operations are permissible. +- A candidate that fails to boot cannot rely on its own update agent for recovery. +- Loss of reporting connectivity should not alone invalidate a locally healthy release. +- Operators need to distinguish intended deployment state from observed device state. + +## Alternatives Considered + +### Server-Controlled Execution of Every Step + +The server authorizes each transition separately, including activation and confirmation. + +This provides tighter online coordination, but a communication outage can leave an operation waiting for a command. It does not remove the need for local handling of boot failures or interrupted writes. It is less suitable for the assumed connectivity model. + +### Local Execution with Mandatory Remote Acceptance + +The device installs and boots independently but requires the backend to accept the candidate before confirmation. + +This can be appropriate when external service behavior is part of release acceptance. In this example, it would couple an otherwise healthy local update to network availability. Such a design also needs an explicit timeout and recovery policy for unavailable remote acceptance. + +### Fully Autonomous Release Selection + +The device selects any compatible authorized release without a backend assignment. + +This can fit simpler products, but does not meet the example's need for centrally controlled release assignment. Local execution autonomy does not imply authority to select arbitrary releases. + +## Consequences + +### Positive + +- Prepared updates can complete or recover during a backend outage. +- Local operational constraints can delay activation without a remote command round trip. +- Boot failure handling remains available independently of the deployment service. +- Requested and observed software states have explicit meanings. + +### Negative + +- The device needs durable execution state and interruption handling. +- Backend status may remain uncertain until the device reconnects. +- Remote cancellation cannot take effect immediately on an offline device. +- Backend and device policies need clear precedence and reconciliation rules. + +## Notes + +The implementation must define when an assignment is accepted, any expiry or cancellation rules, and the points at which an operation may safely be stopped. A backend cancellation cannot undo an activation that already occurred while the device was offline. + +Repeated assignments and repeated reports must be reconciled with the operation identity and actual device state. They must not blindly repeat destructive installation steps. + +This decision does not choose polling, push notifications, storage layout, or the boot failure detection mechanism. It assumes those mechanisms can support the stated local responsibilities; later chapters must establish how. + +## Applicability + +This choice fits devices whose local health can be assessed offline and whose recovery must remain available during communication failures. + +Consider external acceptance when successful operation fundamentally depends on a remote system. Specify how to distinguish failure of the candidate from failure of that system or the network. diff --git a/adr/ADR-012-update-management-and-installation.md b/adr/ADR-012-update-management-and-installation.md new file mode 100644 index 0000000..7c25715 --- /dev/null +++ b/adr/ADR-012-update-management-and-installation.md @@ -0,0 +1,88 @@ +# ADR-012: Separate Update Management from Installation + +* **Status:** Proposed +* **Date:** 2026-09-21 + +## Context + +This educational example considers a device with product-specific update conditions and an installation mechanism that already understands its artifact format. The device may later need more than one mechanism, such as a system image installer and an application package manager. + +Product policy includes when updates are allowed, which local conditions must hold, and how installation relates to activation and health assessment. Installation requires different knowledge: how to validate and prepare a particular artifact and what state remains after interruption. + +[ADR-005](ADR-005-package-based-updates.md) introduces this distinction for package-based updates. This decision applies the same reasoning to the wider device architecture. + +## Decision + +Separate update coordination and product policy from artifact-specific installation responsibilities. + +The update manager evaluates assignments and local conditions, coordinates execution, and tracks the overall operation. The installer owns the artifact-specific operations it implements and reports their results and recoverable state. + +Assign downloading, verification, activation, and recovery integration explicitly according to the selected mechanism. Reuse capabilities already provided by the installer or framework. Do not introduce a second coordinator where an existing framework already fulfills the required management role. + +This is a logical separation. The roles may be modules in one program, existing framework components, or separate processes. + +## Decision Drivers + +- Product operating policy should be understandable independently of artifact-writing code. +- Existing installers should retain responsibility for the operations they support. +- An installer completing its work does not necessarily mean that the new software is active or healthy. +- Interruption handling requires accurate information from the component that performs the operation. +- Multiple update channels need coordinated access to resources and activation. + +## Alternatives Considered + +### Put All Product Policy in the Installer + +One component evaluates operating conditions, installs artifacts, activates software, and reports the outcome. + +This can be sufficient for a small product with one mechanism. In the assumed system, embedding policy into installation code makes it harder to reuse that policy across update channels or change the mechanism independently. + +### Let the Backend Coordinate Installer Commands Directly + +The backend issues low-level commands while the device exposes installation operations. + +This shifts coordination away from local operating state and makes communication failures part of the execution path. It also leaves the device needing local protection against unsafe commands and interrupted operations. [ADR-011](ADR-011-device-controlled-update-execution.md) discusses the execution boundary. + +### Introduce a Universal Installer Interface Immediately + +Every mechanism is hidden behind an identical install-and-rollback interface. + +This appears to simplify coordination but can obscure meaningful differences: a package operation may modify a live filesystem, while an image installer may prepare an inactive target. Recovery guarantees are not interchangeable. Define interfaces around actual requirements and capabilities rather than assuming all mechanisms behave alike. + +## Consequences + +### Positive + +- Product policy has an explicit owner. +- Existing installation and verification capabilities can be reused. +- Different update channels can participate in a coordinated operation. +- Installation completion remains distinct from activation and health acceptance. + +### Negative + +- The boundary needs a defined contract for progress, errors, and interrupted operations. +- Manager and installer state can diverge after a crash and must be reconciled. +- A generic interface may need mechanism-specific capabilities or restrictions. +- Separate processes, if used, introduce communication and permission management costs. + +## Notes + +For each integration, establish: + +- how an operation is identified and its current state queried; +- which component retrieves content and performs each required verification; +- whether installation prepares an inactive candidate or changes running software; +- what “completed” means and what remains before activation; +- whether retry, cancellation, and recovery are supported at each stage; +- which component owns activation and confirmation; +- how concurrent operations are prevented from conflicting. + +The manager must not infer that installation failed solely because its connection to the installer was lost. It must inspect or reconcile the actual state before retrying. + +Likewise, the manager cannot promise rollback merely because an installer exposes a downgrade command. Software, persistent data, and compatibility requirements determine whether recovery is possible. + +## Applicability + +This separation is useful when product policy is substantial, when existing installation tools are reused, or when several update channels need coordination. + +A small single-purpose updater may implement both responsibilities together. Preserve clear ownership without adding processes or abstraction layers solely to match the diagram. diff --git a/docs/03-system-architecture.md b/docs/03-system-architecture.md new file mode 100644 index 0000000..caa23bf --- /dev/null +++ b/docs/03-system-architecture.md @@ -0,0 +1,177 @@ +# 03. OTA System Architecture: Responsibilities and Cooperation + +The previous chapter explored what a device updates and which mechanisms can deliver those changes. The next question is: + +> **Which parts of the system cooperate to complete an update, and what is each part responsible for?** + +An image format or a package manager answers only part of that question. Someone must prepare a release, decide where it should go, determine whether installation is allowed, activate it, and establish whether it works. + +This chapter introduces those responsibilities through a simple system update. The example uses a prepared system release activated by rebooting. Application and data updates can follow a similar structure, with different installation and activation mechanisms. + +The roles below are architectural responsibilities, not a requirement to build a separate service for every box. + +--- + +## 1. From Release Preparation to a Working Device + +Consider a device running release N. A new release, N + 1, has been built and tested. An operator wants an eligible device to install it when local operating conditions permit. + +```text +Build and test + | + v +Release preparation -----> Artifact storage + | | + v | update content +Deployment management | + | | + | update assignment | + v v +Device: update manager <----> Installation mechanism + | | + | coordinate | prepare software + v v + Activation / boot control -> New software runs + ^ | + | v + +-------------- Health assessment + +Device state and results ------> Deployment management +``` + +Release preparation establishes what is being published. Deployment management determines which devices should receive it. The device carries out the update and reports what actually happened. + +The arrows describe logical exchanges. They do not specify a network protocol, process boundary, or required direction for opening a connection. + +## 2. Responsibilities Outside the Device + +### Release Preparation + +The release process builds and tests the software, identifies its intended targets, and prepares the artifacts and metadata needed for delivery and verification. + +A release identity must refer to specific content. Otherwise, a report that a device installed “release N + 1” would not reliably identify the software it received. + +This responsibility also includes arranging authorization of the published release, such as signing artifacts or release metadata. Key handling and verification mechanisms belong to the security chapter. + +### Artifact Storage + +Artifact storage makes update content available for retrieval. It may be a simple file server, an object store, or part of a larger update service. + +Serving a file does not by itself decide which device should install it. Likewise, successful download does not establish that installation or activation succeeded. + +### Deployment Management + +Deployment management associates releases with devices or groups of devices. It may use hardware type, installed versions, or an operator's selection to determine eligibility. + +It also receives progress and results. Its view can be delayed or incomplete when devices are offline. A requested version and a reported running version therefore need to remain distinguishable. + +These responsibilities can share one implementation in a small system. Separating them conceptually helps explain how the design can later accommodate different storage and deployment arrangements. + +## 3. Responsibilities on the Device + +### Update Management + +The update manager coordinates the operation. It evaluates the update assignment against local policy and state: whether the device is eligible, whether another update is in progress, and whether current conditions allow installation or activation. + +For example, a device may download while in use but defer a restart until its current work is complete. + +The manager tracks progress across the steps it coordinates. An interrupted operation must be reconciled with the actual installation and boot state before execution continues. + +### Installation Mechanism + +The installer understands how to prepare a particular update unit. It might write an image, install a package, prepare a filesystem tree, or replace a versioned data artifact. + +Downloading and verification may be implemented here or in supporting components. The architecture must assign those duties explicitly; it does not need to duplicate them in the update manager. + +This extends the distinction between OTA manager and package manager introduced in [ADR-005](../adr/ADR-005-package-based-updates.md). [ADR-012](../adr/ADR-012-update-management-and-installation.md) examines that separation as an architectural choice. + +### Activation and Boot Control + +Activation makes prepared software operational. For a system update, this may require selecting a boot target and restarting the device. For an application, it may require switching versions and restarting a service. + +Boot control matters because the ordinary update manager may not run if the new system cannot boot. The design needs a defined owner for decisions made before that manager becomes available. + +The next chapters will explain boot behavior and storage arrangements. At this level, the important distinction is between preparing software and making it active. + +### Health Assessment + +Health assessment supplies evidence that the activated version can perform its intended work. Reaching the Linux login prompt or starting the update agent may be insufficient for the product. + +An application or dedicated health service might check essential functionality. The architecture must identify who provides the evidence and who uses it to confirm the update or initiate recovery. + +A health check is a defined acceptance criterion, not proof that the release contains no defects. + +## 4. What Participants Exchange + +The update artifact is only one part of the exchange: + +| Information | Purpose | +|---|---| +| Release metadata | Identifies content, intended targets, compatibility requirements, and verification information | +| Update assignment | Expresses which release a device is requested or permitted to install | +| Local execution state | Records the operation and the steps needed to continue or recover after interruption | +| Installation result | Describes what the installer prepared or changed and whether the operation completed | +| Health evidence | Supports the decision to accept an activated version or recover | +| Device report | Communicates observed versions, progress, and failures to deployment management | + +These are logical categories, not six required files or API objects. One implementation may combine several of them. + +The distinction between **requested**, **prepared**, **running**, and **confirmed** versions is especially useful. A device can finish installing N + 1 while still running N. It can also boot N + 1 without having confirmed its health. + +In a hybrid design, the reported state may include separate platform, application, and content versions, as discussed in [ADR-010](../adr/ADR-010-hybrid-update-strategies.md). + +## 5. Walking Through One Update + +For this example, assume the device can prepare a candidate system while retaining a usable previous system. The exact storage mechanism is left to a later chapter. + +1. **Publish the release.** The release process publishes tested content and the metadata needed to identify and verify it. +2. **Assign the release.** Deployment management makes N + 1 available to an eligible device. +3. **Evaluate local conditions.** The update manager checks applicability and policy. It can defer execution if the device is busy or lacks required resources. +4. **Retrieve, verify, and prepare.** The installation path obtains the content, verifies it as required, and prepares the candidate. Unverified content must not become the accepted active release. Verification and writing may be arranged differently by different installers. +5. **Request activation.** Once preparation succeeds and restart is allowed, the device selects the candidate for activation and reboots. +6. **Assess the running version.** The configured health checks determine whether the candidate meets the acceptance criteria. +7. **Confirm or recover.** The responsible local components confirm the candidate or follow the recovery path. A failure before userspace starts must be handled by components available at that stage. +8. **Report the outcome.** The device reports its observed state. If communication is unavailable, reporting is deferred according to the design's retry policy. + +This sequence explains responsibilities rather than defining a universal state machine. The implementation must specify durable state transitions, retry behavior, and recovery rules for its chosen mechanisms. + +## 6. Where Failures Cross Component Boundaries + +Failures reveal responsibilities that a successful sequence can hide. + +| Interruption | Architectural question | +|---|---| +| Network disappears during retrieval | Who retries or resumes, and how is incomplete content prevented from being activated? | +| The update manager restarts during installation | How does it determine whether installation is still running, failed, or completed? | +| Power is lost around activation | Which durable state tells the boot path what to attempt next? | +| The candidate cannot start the update manager | Which earlier component detects failure and selects a recovery path? | +| The candidate runs but fails an essential function | Who supplies health evidence and triggers the appropriate response? | +| The device confirms the release but its report is lost | How does the server learn the actual state without causing an unnecessary reinstall? | + +The chapter's example chooses local execution and recovery after an update has been authorized and prepared. A server connection is not required to confirm the local boot result. [ADR-011](../adr/ADR-011-device-controlled-update-execution.md) explains the assumptions and trade-offs behind that choice. + +Other systems may require external acceptance. They must define what happens when the external service is unreachable and whether loss of connectivity is itself a reason to reject the new version. + +## 7. Authority and Trust Boundaries + +Several decisions can look like a single “permission to update,” but serve different purposes: + +- Release authorization establishes which software comes from an authorized publisher. +- Deployment policy determines which eligible device should receive a release. +- Local policy determines whether the device can perform the operation now. +- Health acceptance determines whether the activated software satisfies the configured operational checks. + +A valid signature alone does not establish that a release is compatible with this device or safe to activate during its current work. A successful download does not establish an authorized source. A server assignment does not establish that the device actually installed the release. + +The architecture needs to identify the component responsible for each check and the information it trusts. The security chapter will develop the mechanisms and threat model in detail. + +## 8. Decisions to Explore Further + +Two reference decisions develop the central boundaries introduced here: + +- [ADR-011: Device-Controlled Update Execution](../adr/ADR-011-device-controlled-update-execution.md) considers devices that must complete local activation and recovery despite intermittent connectivity. +- [ADR-012: Separate Update Management from Installation](../adr/ADR-012-update-management-and-installation.md) considers how product policy can coordinate installation mechanisms without reimplementing them. + +Neither decision prescribes a framework, transport protocol, or number of processes. Choices such as polling versus notifications and combined versus separate backend services can be evaluated once the operating constraints are clear. + +The next chapter examines the boot process: which parts of this architecture can act before the operating system is available, and how responsibility passes from boot components to the running system.