immutable pmOS modular approach

One good use for systemd-sysext is to have the system be made up on a bunch of layers containing different functionality. This has many potential advantages, the most obvious being the potential to avoid generating full OS images for all the devices we support, we could instead split the device specific parts out and have a single generic OS layer (kinda like the Android GSI thing, but with the restriction that we generate both the device parts and the generic parts and they must match exactly, no messy abstraction layers).

For example we might have a rootfs made up of the following layers:

1. device-boot (base device package, firmware, and other deps that are needed for hardware to work)
2. system-core (basically what you get when you install postmarketos-ui-none)
3. system-ui (e.g. plasma-mobile and all deps including middleware like modemmanager)
4. device-overrides (any necessary overrides to system layers, should ideally be empty)

This would make building images for more devices much much more efficient, as the system parts only have to be built once per arch. As a result, it also drastically lowers the barrier of entry for folks to build their own customised system images as they wouldn't have to build for all devices they want to support.

Outcomes of this issue

• Determine if and when we want to investigate this approach
• Determine impacts on overall design (see %Immutable postmarketOS design and PoC)

changed milestone to %Immutable postmarketOS design and PoC

So, device-overrides would be used for device-specific quirks?

By Newbyte on 2024-06-07T06:58:00

@craftyguy proposed for me to put this here instead of making a new discussion so it doesn't get scattered.

I propose that the following should be considered when it comes to immutability:

• Updates:
  • systemd-sysupdate
    • Enable easy delta updates and better security through the trust chain. This would also enable a/b (or even a/b/c!) updates, allowing rollbacks and atomic updates which would greatly increase update stability.
  • flatpak
    • User applications will be installed and updated through flatpak as it is the dominant Linux software repository at the moment. Along with this, flatpak would make it easier for a user to install applications within an immutable system using a familiar interface. It is also much more secure than normal applications as flatpak applications are sandboxed and require specific permissions from the user to break out of the sandbox.
• Image Building:
  • mkosi
    • Makes it much easier to create an image with or without pmbootstrap and allows for more consistent builds.
  • pmbootstrap
    • pmbootstrap should be able to be dropped entirely in the image build process. While it is a very useful development tool, it shouldn't be pretty much required for the generation of an image. This is where I believe mkosi comes into play.
  • Follow the Discoverable Partitions Specification
    • Currently partitions are named distro-specifically in pmOS. I propose shifting to using partition uuids specified in https://uapi-group.org/specifications/specs/discoverable_partitions_specification/. This would not only make it easier for other distros to be installed beside pmOS (eg through a separate root partition), but it would also make postmarketOS follow another standard, which is always a good thing in my book.
• User and Group Handling:
  • systemd-sysusers
    • This would make it easier to handle updates as we wouldn't have to take into account user or group management, as we could just offload this onto systemd.
• Boot Process:
  • Unified Kernel Image
    • A UKI should be used to boot as it makes secure-boot much easier to support and lowers the attack surface of the boot process.
      • (A note on Secure Boot) While secure-boot may not be truely secure, it would make it more convenient for users to have this supported as computers can be annoying when it is disabled. For instance, my Surface Pro 3 makes the entire UEFI display bright red if secure-boot is disabled.
  • dm-verity
    • Partitions should be able to pass dm-verity as we can ensure they can through an immutable root.
  • Visual Boot Menu
    • On boot, the user should be able to hit a key combination and enter a menu where they are able to rollback (or forwards) to either the a or b partition. This is already implemented using desktop bootloaders but I am unsure of the state of mobile.
  • Boot Counter
    • We could implement something similar to Android and ChromeOS where if a partition fails to boot a certain number of times the device switches to booting from the other root partition and marks the failing partition of broken.
  • Hermetic /usr
    • This is just wishful thinking, but being able to generate a full system (minus /var because that needs to be persistent) on boot from just a /usr partition would be wonderful for preventing partitions from failing dm-verity due to a random update messing something up in /etc or such. This would also enable easy factory resets as all top-level directories bar /usr could be deleted and the system would return to a clean, known-good state.
  • systemd-tmpfiles
    • This is more of an extension of the previous point than a new one entirely. systemd-tmpfiles could be used to handle all non-/usr directories and clear them when the users requests so (eg. factory reset)
• Modularity:
  • Developer Mode
    • While the root would usually be read-only, the user can enable developer mode which would (optionally) enable read-write root and enable various things defined further down this piece.
  • systemd-sysext
    • As @calebccff proposed, systemd-sysext would be perfect for this. Although, instead of only four layers, I propose a fifth that contains development tools. This fifth layer would only be activated when the "Developer Mode" option is enabled. This goes into my next few points
• Security:
  • Use polkit instead of sudo/doas
    • To limit attack surfaces and confine what damage a (naive or malicious) user can do to the system, sudo or doas must be removed from the base image and replaced with various polkit rules. sudo or doas can instead be moved to the development layer which would on
  • Remove apk
    • This might be the most contentious of my proposals, but I do think removing apk and moving it to the development layer would make sense as we do not want users using apk in the first place. If a user does want to use apk, then they can enable "Developer Mode" and take off any consistency guarantees we give.

By Aster Boese on 2024-09-09T02:31:12

This is just a quick and dirty proposal, so feel free to tear it apart (kindly) and poke holes.

By Aster Boese on 2024-09-09T02:31:12

This is really nice! Some comments below, but to be fair, anything I didn't touch on I believe is greatly aligned with our vision and goals. On the risk of making some mistakes, some thoughts:

On boot, the user should be able to hit a key combination and enter a menu

We already have some sort of support in the initramfs to do this. Would probably make sense to extend it. And the right place to implement it seems to me to be https://gitlab.com/postmarketOS/buffybox

Follow the Discoverable Partitions Specification

If pmbootstrap is not needed during the image-building process, then it might make this quite easier. Following standards is good though. Just wondering how would this interact with having and flashing a development build.

Unified Kernel Image

Makes sense on paper, specially with the Secure Boot in place. I wonder if we should set right expectations here though, as this might be orders of magnitude harder to get right in Android devices than on everything else. I know Caleb has some thoughts though.

Boot Counter

For Android devices we don't need to re-invent the wheel. We use qbootctl already to avoid them swapping partitions. For other devices, I would be surprised if systemd does not already have something for this.

Developer Mode

Yes, this is a hard-requirement. In that regard, I think @craftyguy referenced how this is extraordinarily well-done in the Steam Deck. Where after any development is done, you can just "disable" it, and then you're back to the regular immutable system. Would make it very convenient, and should be possible with sysext, IIUC

Use polkit instead of sudo/doas

I wonder if there is something that is needed for this. Removing sudo/doas makes sense. But would we need any further polkit rules in an immutable system than in a non-immutable one? Why? I'm sure I'm missing something here.

Remove apk

Sounds crazy, but yes, I think there's agreement with that. If an user wants to use APK, either they are a dev and can use it in developer mode, or they are better off with a non-immutable version.

If you think this makes sense, I'll try to ping Caleb to have his approval. And then if you want to go ahead, you could start opening issues linked to the milestone, and working on them if you feel like it. It's really good you went ahead and wrote this!!

By Pablo Correa Gomez on 2024-09-11T14:28:56

Thanks for the response! It seems to make sense to me. A lot of this is meant to target the "perfect device", so of course some of this might not be possible in production except on generic computers, but I would like to try.

I wonder if there is something that is needed for this. Removing sudo/doas makes sense. But would we need any further polkit rules in an immutable system than in a non-immutable one? Why? I'm sure I'm missing something here.

There will probably be a few things still requiring root, so polkit would be needed, but stripping those to a minimum would be the goal.

By Aster Boese on 2024-09-15T20:02:41

Thanks for writing this all up! I think I'm in favour with pretty much all of these goals, just a few comments (also sorry for taking so long to reply to this!).

Boot Counter

This is a fairly high priority for me!

Developer Mode

There's two different approaches to this. One is like the steam deck where you just make the rootfs R/W, this might be tricky to implement because it breaks dm_verity, and it also means you need to re-flash that slot to get back to a stock system. The steam does not have any secure boot features and updates are shipped as full partition images (not deltas), this is the only way to support the R/W usecase.

What I would propose instead is to just use an overlayfs. /etc and /var would already be R/W, but without sudo it's impossible to mess with them, so developer mode would enable sudo support and mount an overlayfs on /usr. The main downside to overlayfs is the copy-up on-edit implementation, but I don't think this is a huge deal for this implementation (since you're more likely to be adding new files than editing existing ones in /usr).

As an added benefit, if we're also able to support to hermetic /usr model correctly (such that no packages install files outside of /usr), we could basically use developer mode as a way to install arbitrary additional packages on your device and have them maintained across OS updates by hooking the update process so that it makes a note of the packages you added manually and re-installs them on top of the updated image (re-creating the overlayfs safely). But this is maybe delving too far into technical details...

Remove apk

I guess this is fine, and more of an implementation detail. We will for sure need to be careful about the UX here to not alienate folks (e.g. we should have some stub that tells you what's up and how to enable developer mode or whatever we decide to call it).

---

I think in general you've painted a perfect picture of (what I would agree) is the best possible immutable postmarketOS. My big questions are:

1. Do you think it makes sense to make an MVP with a subset of these features (those that are more feasible to achieve in a short amount of time), and
2. How would you prioritise this?

Personally I'm starting to think it might make sense to ship an "immutable v1" which delivers on the basic promise of a more stable system with delta image updates, and a "developer mode" of some kind. Then folks can get most of the important benefits, we can get hardware testing up and we can start ensuring that the software on peoples devices is bit-for-bit identical with what has been boot tested in CI, and start to get test coverage of fiddly annoying things.

In the mean time, we can work towards the harder goals of systemd-tmpfiles, sysusers, polkit, "factory reset by deleting the root partition", etc.

By Caleb Connolly on 2024-09-18T21:33:59

Do you think it makes sense to make an MVP with a subset of these features?

I think creating an MVP would be the correct approach here. My approach to it would be getting the basics down (systemd-sysupdate, systemd-sysext) and going from there in an "alpha" state.

In this alpha state we make no guarantees that the system will be stable or even boot after an update. We can completely change the tools used in the system with no need to care for a migration path between updates.

After alpha we can switch to beta, which would allow people (such as the testing team) to test out the OS and give feedback. By this point the system software should be set-in-stone and users should expect their system to be relatively stable. This should still come with a disclaimer that the OS is not ready for general users and is only for testing.

Next, we can switch to pre-release, which would be where everything is "stable" and immutability can be simply selected and installed for users. This doesn't mean that all features are available, just that we would support migration and updates without breaking the system. This would only be available on edge.

Finally, we can have a release version, which would release with a point release of pmOS. At this point most of, if not all, major features should be available and working. Updates should be stable and users should be able to run this version as a daily-driver and not worry about technical details.

How would you prioritise this?

Prioritization of features for the MVP would be based on:

1. systemd support
2. pmbootstrap v3 support
3. Required for boot?
4. Doesn't break anything

Past this features can be added in later updates of the alpha or, to a lesser extent, beta version.

By Aster Boese on 2024-09-19T00:25:26

What I would propose instead is to just use an overlayfs. /etc and /var would already be R/W, but without sudo it's impossible to mess with them, so developer mode would enable sudo support and mount an overlayfs on /usr. The main downside to overlayfs is the copy-up on-edit implementation, but I don't think this is a huge deal for this implementation (since you're more likely to be adding new files than editing existing ones in /usr).

I was thinking that there could be a developer mode menu that unlocks certain features. Similar to Android, enabling developer mode wouldn't do much as the user has to manually enable developer features.

So, for example, unlocking r/w and disabling dm-verity would be an option, otherwise overlayfs would be used.

By Aster Boese on 2024-09-19T00:26:53

I wonder if there is something that is needed for this. Removing sudo/doas makes sense. But would we need any further polkit rules in an immutable system than in a non-immutable one? Why? I'm sure I'm missing something here.

There will probably be a few things still requiring root, so polkit would be needed, but stripping those to a minimum would be the goal.

I think especially the flathub remote is installed on a system-scope.

Past this features can be added in later updates of the alpha or, to a lesser extent, beta version.

I'm thinking weather we should maybe also wait for the usr-bin-merge.

Cool that there is a plan now. I'm excited!

By fossdd on 2024-09-19T08:16:23

Well it's a few large points we have to (or at least were planning to) do in order:

1. Do usr-merge, preferably upstream in Alpine
2. Merge systemd to pmaports master
3. Get immutable up and running

Step 1 is currently very actively happening (thanks @pabloyoyoista!) and we're close to getting 2 done (thanks @craftyguy!).

By Bart Ribbers on 2024-09-19T08:25:04

Once the first two of those steps are done we can actually get to work on making an MVP. Before that there isn't much we can do other than talk.

By Aster Boese on 2024-09-19T15:14:46

I guess we can then close this issue, given the design is already in-place, and we have the milestone? I can link to this discussion from the milestone description so it's easier to find

By Pablo Correa Gomez on 2024-09-24T18:47:22

Developer Mode

• updatectl makes this a really fancy workflow, on GNOME OS you can just do updatectl enable devel and it will enable those development components for you, that is what we want to use ref: https://www.freedesktop.org/software/systemd/man/latest/sysupdate.features.html#

Currently partitions are named distro-specifically in pmOS. I propose shifting to using partition uuids specified in https://uapi-group.org/specifications/specs/discoverable_partitions_specification/. This would not only make it easier for other distros to be installed beside pmOS (eg through a separate root partition), but it would also make postmarketOS follow another standard, which is always a good thing in my book.

@JustSoup321 you can specifify those uuid to pmos and pmos would boot via those uuid btw.

By ΞЖKƆ/QVH on 2024-09-23T19:26:43

regarding apk-sysext, since the apk database is just a single file, it would be interesting to explore an alternative way of storing it that might be enough to make apk "sysext aware": what if we split the database into a file per package? apparently dpkg does this. it means the sysext would only contains the parts of the database for the packages it touches (including "whiteouts" for removed packages)

By Caleb Connolly on 2024-09-26T14:05:16

Why would we need apk to be sysext aware though? I may be missing something but once a package is installed into a sysext it acts the same as if it was installed normally.

By Aster Boese on 2024-09-26T14:05:15

we should also investigate further on delta uodates with chunks, apparently erofs is well optimised for this with standard chunk sizes.

By Caleb Connolly on 2024-09-26T14:05:42

We could also use zchunk or just generic zstd:chunked.

By Aster Boese on 2024-09-26T14:05:42

I read a bit more on erofs and I think it is a great idea to use that as the default filesystem for immutable partitions. Really excited to get this implementation rolling!!!

By Aster Boese on 2024-09-26T14:44:25

Spoke with @adrianvovk, sysupdate is planning to eventually have some very nice implementation around this where updates are delivered as DDI's mounted as a network block device over HTTP from the server (so block accesses are converted to HTTP range requests).

By Caleb Connolly on 2024-09-27T19:53:38

Some phones barely have enough storage for a single copy of postmarketOS and thus wouldn't be able to support A/B partitions due to lack of space. How will this be handled? Will a mutable version of pmOS without A/B partitions remain supported forever?

By bjorn31 on 2024-09-27T19:54:36

non-immutable pmOS is definitely not going away any time soon. For space constrained device we could investigate a few things, but i think the focus here is mostly on newer well supported devices.

By Caleb Connolly on 2024-09-27T19:55:49

I wanted to suggest something that may be a bit more out of scope and more into the future, but I think would fit really really well into the direction of this proposal.

Multi compositor and wayland compositor handoff:

This was primarily demonstrated and explored by David Edmundson in this blog post (read the blog and watch the tech demos, really cool!). There are several direct advantages of course, such as if the compositor crashes, applications can remain open, and the compositor can restart and reconnect. Of course, maybe especially in a mobile space crashing isn't that bad (unless it's regular of course), it crashed oh well, can just restart, but there are other several major cool things that could be done with this tech.

In the blog post it's shown how it's possible switch compositors at runtime. I think this could be really cool and beneficial, if certain apps don't work well in certain compositors, or perhaps you prefer certain compositors depending on what you're doing.

I think where it really becomes relevant to the mobile space is the Crio demo where you can do checkpoint restore in user-space. Essentially, you can save the state of an application to disk, and then restore it. This could be used to help save battery/performance by hibernating applications for example. You could do even more insane stuff, and go beyond Apple's ecosystem by transferring app states between different machines, so open an app on one device, and then transfer it to another seamlessly. With flatpaks, could even transfer the application if it wasn't already installed.

I've also thought this could be a nice solution to battery saving modes. Instead of running a whole plasma session, why not have the lock screen be a really really simple compositor like cage that just manages the pin, maybe camera app, notifications, and calling, and then launch a new plasma session on unlock.

Either way even if this doesn't get implemented, I think with modularity in mind, it would be nice to design pmOS to be more agnostic of which desktop environment you choose. Perhaps Desktop Environment should be seperated into a seperate layer (dunno)? There should be an option to use a different display manager then the desktop environment equivalent, and it should be possible to install more then one desktop environment without packaging conflicts, and switch between sessions through the display manager.

Sorry if this is a bit out of scope, but it would be absolutely amazing if compositor switching and app restore/moving-across-machines would work, I really really hope one day it gets implemented somewhere!

extra-resources: see wl-reconnect of an example of compositor restore already implemented

This is really cool! While it is definitely outside of the scope of immutability, I would love to see something like this tested. I do think that the battery saving issue is better solved with unified push, but I do like the idea of "suspending" apps.

Definitely out of scope for sure! (I won't pretend I entirely understand immutability to be perfectly honest) but saw the words "modularity" and "system-ui layer" in the issue description, and wanted to suggest future proofing for this as immutability gets designed. It would be nice to at least support multiple Desktop Environments on the same install.

I know David Edmundson also recently mentioned in his talk at Akademy about flatpaks, that he wanted to make Plasma more and more immutable through flatpaks, and make more of components of Plasma through flatpaks. I guess that would definetly help in the future, but with any DE, I guess it won't 100% be in a flatpak.