Hermetic /usr (otherwise known as hermetic-usr) is a state in which an entire operating system is distributed through a read-only /usr directory (preferably partition). This allows easier atomic image-based updates using tools such as systemd-sysupdate and also limits configuration drift for users of regular package-based updates.
Hermetic /usr (otherwise known as hermetic-usr) is a state in which an entire
operating system is distributed through a read-only /usr directory (preferably
partition). This allows easier atomic image-based updates using tools such as
systemd-sysupdate and also limits configuration drift for users of regular
package-based updates.
This is implemented by only installing files under `/usr/`. This would mean creating directories such as `/usr/etc/` and potentially `/usr/var/` and more.
This is implemented by only installing files under `/usr/`. This would mean
creating directories such as `/usr/etc/` and potentially `/usr/var/` and more.
## Motivation
Currently, the plan for an immutable version of postmarketOS entails using systemd-sysupdate. This is a very powerful tool that allows fine-grained control over the update process of an operating system. The issue is that this fine-grained control can lead to the update process becoming overly complex and unmaintainable.
Currently, the plan for an immutable version of postmarketOS entails using
systemd-sysupdate. This is a very powerful tool that allows fine-grained control
over the update process of an operating system. The issue is that this
fine-grained control can lead to the update process becoming overly complex and
unmaintainable.
This issue can be resolved by distributing the entire operating system as a disk image. This process is made much easier by updating a single top-level directory: `/usr/`. This way, while all other directories will differ between machines, `/usr/` will remain stateless and consistent.
This issue can be resolved by distributing the entire operating system as a disk
image. This process is made much easier by updating a single top-level
directory: `/usr/`. This way, while all other directories will differ between
machines, `/usr/` will remain stateless and consistent.
## Consequences
Implementing a hermetic-usr will have a variety of benefits and annoyances:
* Benefits
* Easier image-based update by only relying on one directory instead of multiple.
* Atomicity of updates by defining all system configuration in a read-only partition that is swaped in its entirety on update.
* Allows for roll-backs to a clean system state without leaving behind update artifacts.
* Simplify base by separating host and distro files.
* Allow for "Factory Resets" similar to Android by wiping all directories except `/usr/` and automatically repopulating the rest of the system with systemd-tmpfiles.
* Allows A/B(/C/D/E/...) updates by swaping the `/usr/` partition.
* This also allows for cleaner dual-booting of postmarketOS versions or potentially other distros by only *requiring* one directory change.
* Ties configuration version to package version.
* Better compatability with distros like [openSUSE](https://kubic.opensuse.org/blog/2019-12-05-usr-etc/), [Fedora](https://lwn.net/Articles/946526/), and [Solus](https://getsol.us/2024/09/04/usr-merge/).
* Alignment with the [UAPI Configuration Files Specification](https://uapi-group.org/specifications/specs/configuration_files_specification/).
* Annoyances / Issues
* Many packages read directly from /etc and pass over /usr/etc.
* Basic list of these in this [UAPI issue](https://github.com/uapi-group/specifications/issues/76).
* This is an age-old problem that probably cannot be fixed distro-side. There is [libeconf](https://github.com/openSUSE/libeconf) and [update-conf](https://gitlab.alpinelinux.org/alpine/alpine-conf/-/blob/master/update-conf.in) but for full support we rely on program developers.
* Hard to support properly on OpenRC as there is no polyfill for systemd-tmpfiles other than Chimera Linux's [sd-tools](https://github.com/chimera-linux/sd-tools).
* Would make us differ from upstream Alpine even more if they do not switch to hermetic-usr as well.
- Benefits
- Easier image-based update by only relying on one directory instead of
multiple.
- Atomicity of updates by defining all system configuration in a read-only
partition that is swaped in its entirety on update.
- Allows for roll-backs to a clean system state without leaving behind
update artifacts.
- Simplify base by separating host and distro files.
- Allow for "Factory Resets" similar to Android by wiping all directories
except `/usr/` and automatically repopulating the rest of the system with
systemd-tmpfiles.
- Allows A/B(/C/D/E/...) updates by swaping the `/usr/` partition.
- This also allows for cleaner dual-booting of postmarketOS versions or
potentially other distros by only _requiring_ one directory change.
- Would make us differ from upstream Alpine even more if they do not switch to
hermetic-usr as well.
## Draft implementation plan
The order of these doesn't necessarily matter (except for #2, which *has* to come before everything after it) but this is what I recommend.
The order of these doesn't necessarily matter (except for #2, which _has_ to
come before everything after it) but this is what I recommend.
1. Create a separate /usr partition for new installs.
2. Move users created in pmaports and while using pmbootstrap to use systemd-sysusers.
- Potentially this could also be applied to packages from aports by creating a shim that turns `useradd` and `groupadd` into commands that create the respective systemd-sysusers configs.
- This can be done on OpenRC as well using Chimera Linux's [sd-tools](https://github.com/chimera-linux/sd-tools) if need-be.
2. Move users created in pmaports and while using pmbootstrap to use
systemd-sysusers.
3. Install all packages under /usr. There are two options for doing this.
- Add hermetic-usr to the filesystem hierarchy standard and convince Alpine to switch to it. Compatability can be maintained without systemd-tmpfiles by installing symlinks to directories that were moved. Downstream we can use systemd-tmpfiles to allow for factory resetting.
- The worse option would be to patch apk-tools to install all files under their respective directories within /usr and autogenerate systemd-tmpfiles configs for where they would usually go.
- Potentially this could also be applied to packages from aports by creating
a shim that turns `useradd` and `groupadd` into commands that create the
respective systemd-sysusers configs.
- This can be done on OpenRC as well using Chimera Linux's
[sd-tools](https://github.com/chimera-linux/sd-tools) if need-be.
4. Update our custom tools (e.g. mkinitfs and boot-deploy) to use configs under `/usr/` unless configs in `/etc/` are found.
-[libeconf](https://github.com/openSUSE/libeconf) is a library that already does this automatically and is recommended by UAPI, so the best method would be to switch our tools' configuration management to it.
3. Install all packages under /usr. There are two options for doing this.
5. Modify all packages in pmaports to install under `/usr/` with the respective systemd-tmpfiles config.
- Add hermetic-usr to the filesystem hierarchy standard and convince Alpine
to switch to it. Compatability can be maintained without systemd-tmpfiles
by installing symlinks to directories that were moved. Downstream we can
use systemd-tmpfiles to allow for factory resetting.
- The worse option would be to patch apk-tools to install all files under
their respective directories within /usr and autogenerate systemd-tmpfiles
configs for where they would usually go.
Why systemd-tmpfiles instead of symlinks?
4. Update our custom tools (e.g. mkinitfs and boot-deploy) to use configs under
`/usr/` unless configs in `/etc/` are found.
- While symlinks might work fine for moving a directory, `/usr/etc/` and family wouldn't be the top-level directory directly moved, as with `/usr/bin/`, but instead just be vendor (us) supplied configs that can be overridden by files in `/etc/`. A lot of programs don't actually use `/usr/etc/` though, so systemd-tmpfiles would populate the files correctly.
-[libeconf](https://github.com/openSUSE/libeconf) is a library that already
does this automatically and is recommended by UAPI, so the best method
would be to switch our tools' configuration management to it.
Why not just copy the files from `/usr/etc/` and family to their respective TLDs?
5. Modify all packages in pmaports to install under `/usr/` with the respective
systemd-tmpfiles config.
1. This would make it impossible to correctly handle config updates through packages. Say a package is installed with a config in `/usr/etc/` and copied to `/etc/`. The user modifies this config and goes about their day. A new update comes in that modifies the config in `/usr/etc/`. How do we handle this? Do we replace the user's config in `/etc/` with the new config, removing the customization that the user did, or do we not update the config in `/etc/`, leaving the user with potentially outdated configs that drift from our repos? Better to not directly copy at all.
Why systemd-tmpfiles instead of symlinks?
2. This would invalidate the ability to factory reset, as every directory except `/usr/` would need to be deleted. We would need to have some kind of logic to determine when a directory is removed and automatically regenerate it. The easiest way to do this is the use systemd-tmpfiles.
- While symlinks might work fine for moving a directory, `/usr/etc/` and family
wouldn't be the top-level directory directly moved, as with `/usr/bin/`, but
instead just be vendor (us) supplied configs that can be overridden by files
in `/etc/`. A lot of programs don't actually use `/usr/etc/` though, so
systemd-tmpfiles would populate the files correctly.
Why not just copy the files from `/usr/etc/` and family to their respective
TLDs?
1. This would make it impossible to correctly handle config updates through
packages. Say a package is installed with a config in `/usr/etc/` and copied
to `/etc/`. The user modifies this config and goes about their day. A new
update comes in that modifies the config in `/usr/etc/`. How do we handle
this? Do we replace the user's config in `/etc/` with the new config,
removing the customization that the user did, or do we not update the config
in `/etc/`, leaving the user with potentially outdated configs that drift
from our repos? Better to not directly copy at all.
2. This would invalidate the ability to factory reset, as every directory except
`/usr/` would need to be deleted. We would need to have some kind of logic to
determine when a directory is removed and automatically regenerate it. The
easiest way to do this is the use systemd-tmpfiles.
### The proposer
This proposal is a collaboration between Aster Boese (@JustSoup321) and Caleb Connolly (@caleb). Aster is a Trusted Contributor while Caleb is a Core Contributor.
This proposal is a collaboration between Aster Boese (@JustSoup321) and Caleb
Connolly (@caleb). Aster is a Trusted Contributor while Caleb is a Core
Contributor.
This was first proposed by Aster [here](https://gitlab.postmarketos.org/postmarketOS/postmarketos/-/issues/62#note_300370) with help from Clayton Craft (@craftyguy) and Caleb.
