Closes #2

adr/ADR-016-separate-system-software-and-persistent-device-data.md

@@ -0,0 +1,74 @@
+# ADR-016: Separate System Software and Persistent Device Data
+
+## Context
+
+This example uses system images and an A/B layout. The device keeps the current system while preparing a new version in the other slot.
+
+An image installer can replace all contents of its target area. Device identity, calibration, settings, and application data must survive that operation. Both retained system versions may need these files.
+
+The device has enough storage for two system versions and a separate persistent data area. The product can define which data formats the retained versions support.
+
+## Decision
+
+Store system software in the system slots. Store device identity, calibration, device settings, and application data outside the areas replaced by ordinary system image updates.
+
+Keep the running system files read-only in this example. Programs write changing files to the persistent data area. The platform defines the locations and access rules for those files.
+
+An image may include default configuration. Device settings are stored separately, with a defined rule for how the application combines them with the defaults. A normal system release must not overwrite device identity or calibration.
+
+Share persistent data between the slots only when the supported recovery versions can use it. Before changing a data format, define and test compatibility or a restore path. If recovery uses a saved copy, define what happens to data written after that copy was made.
+
+The product must state when returning to an older system stops being supported. Confirmation alone does not prove that later data changes remain compatible with that system.
+
+## Decision Drivers
+
+- Installing a system image must not erase information that belongs to the device.
+- Returning to the previous system must preserve a usable data path.
+- The boundary between release files and changing device files must be clear.
+- Normal application writes should not change the system software under trial.
+
+## Alternatives Considered
+
+### Keep Device Data Inside Each System Image
+
+This is a simple layout, but replacing an image would erase the data unless the installer copies it elsewhere first. The copying and restore steps would need their own failure handling.
+
+### Keep a Separate Data Copy for Each Slot
+
+This can help isolate data changes during a trial. It needs more storage and rules for copying data, choosing the active copy, and handling new records after a return to the old slot. It may be appropriate when shared data cannot support both versions.
+
+### Keep One Writable System With All Data
+
+This can suit package-based updates. It does not meet this example's requirement to prepare and replace a complete system image while retaining the current system.
+
+## Consequences
+
+### Positive
+
+- System image replacement preserves device-specific files.
+- New application records can remain available after a return to the previous system.
+- The areas that the image installer may change are easier to define.
+
+### Negative
+
+- The platform must arrange paths and access to persistent files.
+- Shared data needs compatibility rules for retained software versions.
+- A shared data failure can affect both slots.
+- Copies or backups used for data recovery need extra storage and cleanup rules.
+
+## Applicability and Limits
+
+Use this approach when system software is replaced as an image and device data must survive that replacement.
+
+Separation may use partitions or other storage areas outside the installer's target. A directory inside a partition being overwritten is not sufficient.
+
+This choice does not protect against failure of the whole storage device. It also does not make data writes safe under power loss or make migrations reversible. Those need separate mechanisms and tests.
+
+Key storage and access protection need a security design. Placing identity files outside a system slot is only a rule about their lifetime.
+
+## Related Documents
+
+- [Chapter 05: Storage Layout](../docs/05-storage-layout.md)
+- [ADR-004: Full-System Image Updates](ADR-004-full-system-image-updates.md)
+- [ADR-014: Trial Boot and Confirmation for System Updates](ADR-014-trial-boot-and-confirmation.md)
+- [ADR-017: Keep Recovery State Outside Replaceable Software](ADR-017-keep-recovery-state-outside-replaceable-software.md)

adr/ADR-017-keep-recovery-state-outside-replaceable-software.md

@@ -0,0 +1,74 @@
+# ADR-017: Keep Recovery State Outside Replaceable Software
+
+## Context
+
+Trial boot and service downgrade both depend on saved state. After a restart, the device needs to know which version is under trial, whether it was confirmed, and whether recovery is in progress.
+
+If that state exists only inside the software being replaced, installation or recovery may erase it. If boot control cannot read it before Linux starts, it cannot use it to handle an early startup failure.
+
+This example assumes that the platform provides storage accessible to boot control and persistent storage accessible to the OTA manager or service supervisor. These may be different areas.
+
+## Decision
+
+Keep state required for recovery outside the software areas that the relevant update or recovery operation replaces.
+
+Keep boot selection, trial attempts, and confirmation in a location and format supported by boot control. The Linux component that requests a trial or confirms the system must use the platform's defined boot control interface.
+
+Keep installation progress, service recovery progress, failed version blocks, and pending reports in persistent storage accessible to the OTA manager or supervisor. Package installation, removal scripts, and cleanup must preserve these records while they are needed.
+
+Do not require one common file for every reader. Define who writes each record and how the readers agree on its meaning. Both retained software versions must understand any state they need after recovery.
+
+Use a storage method that handles interrupted writes. Readers must detect invalid or incomplete records. The platform must define a recovery action when no valid record remains; unreadable state must not be interpreted as successful confirmation.
+
+Define when each record may be removed. A report may be removed after its delivery is acknowledged. A failed version block follows the explicit removal policy in ADR-015. Recovery progress must remain available until the recorded operation has reached a known outcome.
+
+## Decision Drivers
+
+- Restarting or replacing software must not erase the information needed to recover.
+- Boot control must make decisions before the OTA manager starts.
+- Offline recovery must retain progress and results locally.
+- Interrupted state writes must have a defined outcome.
+
+## Alternatives Considered
+
+### Store State Only Inside the Target System Slot
+
+Replacing that slot may erase the state. Boot control may also be unable to read it when the new system fails. This does not meet the recovery requirements.
+
+### Store All State in the Application's Data Files
+
+This may be convenient for reporting. It is unsuitable when boot control cannot read those files or when package operations can remove them. Separate records are used according to their readers and lifetime.
+
+### Keep the Authoritative State Only on the Server
+
+The server can track deployment progress, but it may be unreachable during boot or recovery. The device needs local state to complete those steps while offline.
+
+## Consequences
+
+### Positive
+
+- Boot and service recovery retain the information they need across replacement and restart.
+- Each reader has state it can access at the required time.
+- Reports can wait locally for the server connection to return.
+
+### Negative
+
+- The platform needs tested methods for saving and reading state under power loss.
+- Boot control and Linux must agree on state formats and ownership.
+- Multiple records may need coordination if an operation is interrupted between writes.
+- Persistent storage needs space limits and retention rules.
+
+## Applicability and Limits
+
+Use this approach when recovery decisions depend on state that must survive software replacement or an interrupted operation.
+
+This ADR defines where state belongs and who must read it. It does not select a filesystem, record format, or bootloader command. Those choices belong in the platform example and must be tested under power loss.
+
+Separating state from software does not guarantee that the state is valid. It also does not make several writes one atomic operation. The update design must handle the intermediate states those writes can leave behind.
+
+## Related Documents
+
+- [Chapter 05: Storage Layout](../docs/05-storage-layout.md)
+- [ADR-014: Trial Boot and Confirmation for System Updates](ADR-014-trial-boot-and-confirmation.md)
+- [ADR-015: Service Checks and Package Downgrade](ADR-015-service-checks-and-package-downgrade.md)
+- [ADR-016: Separate System Software and Persistent Device Data](ADR-016-separate-system-software-and-persistent-device-data.md)

docs/00-series-overview.md

@@ -82,6 +82,7 @@ ota-reference-design/
 │   ├── 02-update_strategies.md
 │   ├── 03-system-architecture.md
 │   ├── 04-starting-an-updated-system-or-service.md
+│   ├── 05-storage-layout.md
 │   ├── ...
 │
 ├── diagrams/
@@ -110,6 +111,7 @@ The chapters are designed to build on each other. The current chapters are:
 - [02. Choosing an OTA Update Strategy](02-update_strategies.md)
 - [03. OTA System Architecture](03-system-architecture.md)
 - [04. Starting an Updated System or Service](04-starting-an-updated-system-or-service.md)
+- [05. Storage Layout](05-storage-layout.md)
 
 The learning path below also includes planned chapters.
 

docs/04-starting-an-updated-system-or-service.md

@@ -194,4 +194,4 @@ These Architecture Decision Records (ADRs) explain when to choose the approaches
 
 Both examples follow the same pattern: start the new version, check its required functions, then keep it or recover. They use different mechanisms because a system boot and a service restart fail in different ways.
 
-Next, we will look at storage layout: where software versions, device data, and update state live, and what must remain available when an update fails.
+Next, we will look at [storage layout](05-storage-layout.md): where software versions, device data, and update state live, and what must remain available when an update fails.

docs/05-storage-layout.md

@@ -0,0 +1,214 @@
+# 05. Storage Layout: Software Versions, Persistent Data, and Update State
+
+The previous chapter followed the startup of a new system or service. It also showed why the device may need to return to an older version.
+
+That return is only possible if the needed software, data, and update state are still available.
+
+A **storage layout** describes where these things live. It also defines which parts an update may change. We will use the same two examples: an A/B system update and an independent service package update.
+
+## 1. Different Files Have Different Jobs
+
+Before choosing partitions or directories, separate the contents by purpose.
+
+| Contents | Example | What an update must preserve |
+|---|---|---|
+| System software | Kernel, system libraries, services | A usable previous version during a trial |
+| Device identity and calibration | Device keys, sensor calibration | The information that belongs to this device |
+| Configuration | Network settings, application settings | Settings needed by the versions that may run |
+| Application data | Measurements, a local database | Data needed for operation and recovery |
+| Update state | Trial target, confirmation, failure block | Enough information to choose the next safe action |
+| Recovery files | Previous service package and verification information | The files needed to restore the previous service |
+| Temporary files | Partial downloads, work files | Nothing essential; incomplete work may be discarded |
+
+These groups do not all need separate partitions. A **partition** is a defined region of a storage device. Several groups may use different directories in one partition.
+
+The important question is what an operation can replace or erase. If the installer writes a complete partition image, a separate directory inside that partition will not protect the files in it.
+
+## 2. An Example A/B Layout
+
+In this example, each system slot holds a matching kernel, device tree, and root filesystem. The boot control mechanism selects them together.
+
+The device tree describes hardware for the kernel. The root filesystem holds the system files Linux needs. Some platforms store these parts differently. The example shows their roles, rather than a required partition table.
+
+```text
+Device storage: logical areas
+|
++-- Shared boot components
+|   `-- Code needed to select and load a system
+|
++-- Boot state
+|   `-- Trial target, attempts left, confirmation
+|
++-- System slot A
+|   `-- Kernel A + device tree A + root filesystem A
+|
++-- System slot B
+|   `-- Kernel B + device tree B + root filesystem B
+|
++-- Persistent device data
+|   +-- Device identity and calibration
+|   +-- Configuration and application data
+|   `-- OTA progress, failure blocks, pending reports
+|
+`-- Download and work area
+    `-- Temporary files used to prepare an update
+```
+
+**Persistent** means that information remains after a restart. Here, persistent device data also remains when either system slot is replaced.
+
+Suppose A is running and B will receive the next version. The installer may replace B. It must preserve A and the data needed to start and use A again.
+
+The running system in this example keeps its system files read-only. Files that change during normal operation go into the persistent data area. A read-only filesystem prevents ordinary writes to those system files. It does not protect them from every failure or from an installer that writes directly to the storage device.
+
+The layout still has shared parts. If an update damages the shared boot components, neither slot may start. Updating those components needs its own safe installation and recovery method.
+
+Two slots on one storage device also do not protect against failure of that whole device.
+
+## 3. What Lives Outside the System Slots?
+
+A system release provides common software. Device identity and calibration belong to an individual device. Installing a system image must not replace them with values from the release build.
+
+Configuration has two possible sources. The system image can provide default settings. Settings chosen for this device live outside the image. The design must explain how the application uses both.
+
+Application data also lives outside the replaceable system slots in this example. Updating the system should not erase stored measurements or a local database.
+
+Separation does not mean that every program may write every file. The platform must define access rules. For example, an ordinary application should not be able to replace the device's identity keys.
+
+[ADR-016: Separate System Software and Persistent Device Data](../adr/ADR-016-separate-system-software-and-persistent-device-data.md) explains the conditions for this choice.
+
+## 4. Shared Data Must Work With Both Versions
+
+Both A and B can use the same persistent data. This saves space and keeps new measurements available after a return to A. It also creates a compatibility requirement.
+
+Imagine this sequence:
+
+```text
+A uses database format 1
+          |
+B starts and changes the database to format 2
+          |
+B fails its health checks
+          |
+The device returns to A
+          |
+Can A still read the database?
+```
+
+If the answer is no, keeping A was not enough.
+
+A **data migration** changes stored data into a new format. Before allowing one, define how recovery will work. Possible approaches include:
+
+- Keep a format that both versions can use during the trial.
+- Prepare a separate copy for the new version and keep the old copy for recovery.
+- Save a backup and use a tested restore procedure.
+- Delay a change that the old version cannot read until the device no longer promises a return to it.
+
+Each approach has a cost. Copies and backups need space. Restoring old data may lose measurements or settings written after the copy was made. Delaying a migration only helps if the new version can work before the change.
+
+Configuration needs the same care. A new setting may be harmless to an old application, or it may stop that application from starting.
+
+The product must define how long the previous version remains a supported recovery choice. Passing health checks does not make every later data change safe for that version.
+
+## 5. Update State Must Be Available to Its Readers
+
+Chapter 04 introduced saved trial attempts, confirmation, recovery progress, and failure blocks. These records have different readers.
+
+| State | Who needs it? | When must it be available? |
+|---|---|---|
+| Selected system and trial attempts | Boot control | Before the selected Linux system starts |
+| System confirmation | Boot control and the program that confirms the system | During confirmation and on later boots |
+| Installation and recovery progress | OTA manager or service supervisor | After a restart or interrupted operation |
+| Failed package version block | Update policy owner | Before another automatic installation |
+| Pending result report | OTA manager | When reporting can resume |
+
+Putting all these records in one file is not always possible. Boot control may not be able to read the filesystem used by the OTA manager. The platform must choose a location and format each reader supports.
+
+State needed to recover from replacing B must not exist only inside B. State needed by both systems must remain available when either slot is replaced.
+
+Saving state also needs a method that handles power loss. After an interrupted write, the reader must recognize a valid record or follow a defined recovery path. The platform must define what happens when no valid record remains. It must not treat unreadable state as proof that the new version was confirmed.
+
+Boot control and Linux must also agree on what the records mean. Both retained system versions need to understand the state they may read during recovery.
+
+These requirements are described in [ADR-017: Keep Recovery State Outside Replaceable Software](../adr/ADR-017-keep-recovery-state-outside-replaceable-software.md). The next chapter will look more closely at safe writes and activation.
+
+## 6. Storage for an Independent Service Update
+
+The package example uses a stable system platform. The service files may be replaced in the running filesystem. It does not gain A/B protection just because its data is stored separately.
+
+A possible layout is:
+
+```text
+Stable system platform
+|
++-- Installed service files
+|
++-- Service configuration and data
+|
++-- Local recovery files
+|   `-- Exact previous package + verification information
+|
++-- Supervisor state
+|   `-- Update progress, failed version block, pending report
+|
+`-- Download and work area
+    `-- New package and temporary files
+```
+
+The supervisor must remain able to run when the service fails. Its recovery files and state must survive the package operations it controls. They must not be deleted by the service's removal scripts or by ordinary cache cleanup.
+
+Keeping the previous package locally allows recovery while offline. The device must also retain any dependencies and installation inputs needed for the supported downgrade. A single package file may not be enough.
+
+Before installing version 2, check that version 1 can still use the resulting configuration and data. Package installation scripts may change these files even before the service starts.
+
+[ADR-015: Service Checks and Package Downgrade](../adr/ADR-015-service-checks-and-package-downgrade.md) defines the recovery policy for this example.
+
+## 7. Plan for Space and Cleanup
+
+The update needs more space than the final installed software alone. Depending on the method, it may need:
+
+- The running version and the prepared new version.
+- A downloaded artifact and files used during installation.
+- Previous packages and their required dependencies.
+- A data copy or backup used for recovery.
+- Saved state, reports, and normal application data.
+
+Some installers write verified download data directly into an inactive slot. Others keep a complete download first. The storage plan must match the chosen method.
+
+Check the available space before starting. Normal services may keep writing data during an update, so the design also needs limits or reserved space for essential state. Logs and downloads must not consume all space needed for recovery records.
+
+Cleanup needs clear rules. A partial download can usually be removed or downloaded again. A previous package needed for recovery cannot be treated as an ordinary disposable cache file.
+
+For A/B, the inactive slot may become the target of the next update after the current system is confirmed. Replacing it removes that older recovery version. The update policy must make this change deliberately.
+
+Never delete the only usable recovery files to make room for an unconfirmed update.
+
+## 8. Check the Layout Against Failures
+
+For each operation, ask which areas it can change and what remains usable if it stops halfway.
+
+| Failure | What this design needs |
+|---|---|
+| Power is lost while writing B | A and the information needed to select it remain usable. B is not selected as a ready system. |
+| Power is lost while saving trial state | Boot control detects a valid record or follows its defined recovery path. |
+| B changes shared data and then fails | A can use the data, or a tested data recovery path is available. |
+| The download area fills up | The update stops safely; essential state can still be saved. |
+| Service installation is interrupted | The supervisor can read its progress and follow the package manager's recovery rules. |
+| Cleanup runs during service recovery | Required packages, verification information, and recovery state are retained. |
+| Shared boot code or the storage device fails | A separate recovery method is needed; two system slots alone are insufficient. |
+
+This table describes requirements. A storage diagram does not prove they are met. Practical examples will need to test interrupted writes, installation, startup, and recovery.
+
+## 9. Related Architecture Decisions
+
+| Question | Further reading |
+|---|---|
+| When should a system get a trial boot? | [ADR-014: Trial Boot and Confirmation for System Updates](../adr/ADR-014-trial-boot-and-confirmation.md) |
+| How is a failed service package restored? | [ADR-015: Service Checks and Package Downgrade](../adr/ADR-015-service-checks-and-package-downgrade.md) |
+| When should device data be separate from system software? | [ADR-016: Separate System Software and Persistent Device Data](../adr/ADR-016-separate-system-software-and-persistent-device-data.md) |
+| Where should recovery state live? | [ADR-017: Keep Recovery State Outside Replaceable Software](../adr/ADR-017-keep-recovery-state-outside-replaceable-software.md) |
+
+## 10. What Comes Next?
+
+A useful storage layout makes the boundaries clear: what the update replaces, what it preserves, and what recovery can still read.
+
+Next, we will look at atomic updates. We will ask how to prepare and activate a new version without exposing an incomplete one, and how the device makes a safe decision after an interrupted operation.