On-device installer

added discussion help wanted + 1 deleted label

Agreed. Calamares looks good to me from a quick glance, thanks for adding apk support already - and I like how short the patch was!

I'm not sure if it's mobile-friendly at the moment (read: responsive UI and touch input capable), but I think it's less to work to make it so rather than creating our own thing.

It's probably a good idea to ask the Manjaro folks if they are already working on making calamares mobile friendly, just do avoid duplicate efforts.

As discussed in the meeting, some ideas on how the installer could operate:

for the pinephone, we could run it from the sd card and then install to internal storage
but that does not work with all phones of course; for other phones (and maybe as generic solution for the pinephone too?), we could try:
flash it to the internal storage, with a small partition
create a new partition in the installer right after the installer partition
install postmarketOS to that new partition
delete the installer partition
auto-resize the postmarketOS partition to use that space after reboot (when implementing, make sure that it also works with FDE)

The second approach is a lot more effort, so it would probably be nice to have the installer working from sd card first, and implement the more complicated thing later on.

By Oliver Smith on 2020-04-24T07:15:15

It's probably a good idea to ask the Manjaro folks if they are already working on making calamares mobile friendly

They aren't, but milohr from the Maui project is, in coordination with Adriaan de Groot (the lead developer of Calamares). Adriaan mentions it's a lot of work before it'll be finished though.

By Bart Ribbers on 2020-04-24T07:15:15

I wonder if we could fit the installer in a (Android) boot image. So on other devices you could simply fastboot boot it, without flashing anything.

(Note: The limit for booting Android boot image is usually much higher than the partition for flashing them, since they are normally only loaded into RAM...)

By Minecrell on 2020-02-29T21:47:12

how about something like "GNOME Initial Setup" instead of being an actual installer it just lets you configure basic end user settings at first boot

By Raatty on 2020-03-01T04:52:46

Another alternative would be to write that graphical pmbootstrap frontend. I think it would fit our purpose better than e.g. Calamares as it would provide a common interface for installation on desktop-to-phone and phone-to-phone. It would also avoid corner case bugs that I fear would arise when multiplying ways one could install pmOS.

By Antoine Fontaine on 2020-03-01T09:24:14

remember to think of the future when phones might ship with postmarketOS pre installed so there will have to be something more than just the bootstrap?

By Raatty on 2020-03-01T10:24:39

I still think the graphical installer should be simple and user-friendly. It should ask you your device, whether you want non-free drivers, your UI, your password and that's probably all. It would even know your device already in the case of prebuilt images!

I don't think it is possible to have the same amount of complexity in the graphical installer anyway: if you would like the full-featured pmbootstrap command-line, you probably should use the command line anyway.

By Antoine Fontaine on 2020-03-01T10:24:39

mentioned in issue #525 (closed)

By Anjandev Momi on 2020-04-23T07:17:49

changed milestone to %Other daily driver relevant

mentioned in issue pmbootstrap#63 (closed)

By Oliver Smith on 2020-05-10T14:51:58

Related new WIP wiki page: https://wiki.postmarketos.org/wiki/Installation_settings (from discussion in https://gitlab.com/postmarketOS/pmbootstrap/-/merge_requests/1933#note_335542213)

By Oliver Smith on 2020-05-10T15:50:35

mentioned in merge request pmbootstrap!1944 (merged)

By Oliver Smith on 2020-06-03T16:04:02

So I've been working on this since Monday, as part of the grant (see last blog post). Here's a status update.

postmarketos-oem-install-auto, postmarketos-mkinitfs changes

I've created a proof of concept postmarketos-oem-install-auto pmaport (0001-postmarketos-oem-install-auto-new-aport.patch):
- runs with openrc on boot
- creates a new encrypted ext4 partition after the install partition (parted, cryptsetup, mkfs.ext4)
  - finding the correct start sector needs some calculations
- prepares the installation (copies apk keys, fstab, ...)
- installs postmarketos-base, device-qemu-amd64 into that partition (downloading packages from the network for the proof of concept)
- renames install partition from pmOS_install to pmOS_deleteme_install
postmarketos-mkinitfs patch, so it deletes the pmOS_deleteme_install partition
the proof of concept is working. I can use it to test the mkinifs changes quickly (verify that it chooses the right partition before and after the installer runs) and maybe it's useful for that case in the future too

Calamares packaging

@PureTryOut's WIP calamares packaging was very helpful, I've started APKBUILDs based on that
postmarketOS branding is in a separate package, as recommended by the calamares wiki. I'll put it up in its own git repository.

Calamares UI

calamares installation screens are divided in modules. Most modules are not qml/kirigami based, as written above, but there are a few existing ones (welcomeq, keyboardq, localeq).
the existing luks support encrypts each partition with a keyfile, embeds the key into the initramfs and trusts grub2 to encrypt that initramfs. This won't work with postmarketOS, for almost all devices we don't use grub and the initramfs is not encrypted. However, the partitioning is much simpler with postmarketOS too, there is just one relevant partition that needs to be encrypted (dismissing that we could do advanced stuff like lvm over "system" and "userdata" partition at some point, that's out of scope for this installer for now).
I'm creating a new qml/kirigami based "partitionq" module, which will ask for a password and create the encrypted ext4 partition. My intend is to upstream it to calamares, and I'm giving it this generic name, so it can be extended to have more features of the regular "partition" module later (e.g. choosing a partition to install instead of having one hardcoded in the config, setting the encryption algorithm, setting the filesystem type, ...).
Here is a WIP screenshot of partitionq: WIP-calamares-partitionq-fde-pass (yes, we need to increase the fonts and make it look a lot nicer, but that's easy with QML)
Tomorrow I'll make the button functional; right now it just goes to the next screen without creating the partition in the background.
To develop this UI, I'm currently running Calamares from pmbootstrap with pmbootstrap chroot --xauth calamares (pmbootstrap!1944 (merged))

Planned pmbootstrap changes

Generating the OEM-installer with pmbootstrap will look something like this (thanks to @MartijnBraam for brainstorming!):

pmbootstrap install \
    --root-label="pmOS_install" \
    --add="postmarketos-oem-installer" \
    --add-cache="postmarketos-ui-weston,device-qemu-amd64" \
    --reserve-space="10G"

Most options are not implemented yet, I'll do that soon.

--root-label: don't use pmOS_root as label, but pmOS_install instead. postmarketos-mkinitfs will be adjusted to boot into pmOS_install if it exists, preferred to pmOS_root.
--add="postmarketos-oem-installer": (--add exists already) add that new package, which will pull in calamares, the postmarketOS branding, and start it automatically. I have not yet decided on the UI, something lightweight like i3 probably (not sure if there is a benefit if we go for something wayland based; i3 will probably work with more downstream devices?)
--add-cache: fill /var/apk/cache with these packages and its dependencies (so the installation works offline)
--reserve-space="10G": have 10 GB of empty space between the boot and root partitions, so it can be used for the installation. This will only be necessary if the final partition should overwrite the install partition in the end. (I've experimented with making the final partition after the install partition, but we can't grow it backwards.)

By Oliver Smith on 2020-06-04T10:41:02

i3 will probably work with more downstream devices

From reports over the years that this project has existed I believe Wayland actually works on more devices out-of-the-box, X had some problems on several devices in the past. Besides, when we use Wayland for this, we can use Cage which seems ideal for this.

Awesome work so far though!

By Bart Ribbers on 2020-06-04T10:41:02

cage requires drm though. a plain xorg session probably still has the most support, or weston.

By Martijn Braam on 2020-06-04T11:01:06

Generating the OEM-installer with pmbootstrap will look something like this

It seems that I'm stuck on the --add-cache option. The apk filenames in the cache have a hash attached to the file name, just like the APKINDEX:

$ cd ~/.local/var/pmbootstrap/cache_apk_x86_64
$ ls -1 | head -n10
APKINDEX.c25db003.tar.gz
APKINDEX.2c4ac24e.tar.gz
APKINDEX.40a3604f.tar.gz
abuild-3.6.0-r0.7d337637.apk
accountsservice-0.6.55-r1.61f12239.apk
adwaita-gtk2-theme-3.28-r1.2972eacf.apk
adwaita-icon-theme-3.36.1-r0.005c4ca1.apk
alpine-baselayout-3.2.0-r7.586a54e5.apk
alpine-base-3.12.0-r0.68487cea.apk
alpine-conf-3.9.0-r1.22024e76.apk

For the APKINDEX, pmbootstrap is able to compute the hash properly. However, for the apk files it does not work the same - from my understanding it is not derived from the URL, but instead from another hash of the actual file, that can be found in the APKINDEX. So we would need to correctly parse that other hash from the APKINDEX first, which seems like a big effort to re-implement properly in python. I don't think this is a good use of time.

Instead, I will go forward with another approach. I imagine:

$ pmbootstrap install --on-device [--sdcard]

do a regular installation, that yields the rootfs as image (like pmbootstrap install --split)
create a new chroot installer_$device (not the usual rootfs_$device)
copy the generated image into that new chroot
install the device package and postmarketos-on-device-installer into that new chroot
create a resulting new installation image or flash it to the SD card and apply the right "--reserve-space" automatically (we know the image size!)

Besides being easier to implement, this will also make the command-line interface much easier to use: only one new --on-device parameter vs 3 new parameters that only make sense when carefully used together.

Furthermore this should make the installation more stable, we can just extract an image in calamares, meaning there are no post-install scripts getting executed by apk, that can fail. These are all executed beforehand by pmbootstrap, we will catch such errors before building, flashing, testing the image on the target device.

By Oliver Smith on 2020-06-07T09:07:45

we can just extract an image in Calamares

That is how Calamares usually does it anyway so it's probably preferred over doing something custom.

By Bart Ribbers on 2020-06-07T09:07:45

pmbootstrap install --on-device-installer is probably less confusing than pmbootstrap install --on-device, I'll go with that.

By Oliver Smith on 2020-06-07T05:57:24

apk-tools sources look complicated because it basically has to reinvent the whhel by implementing all low-level stuff itself (because it wants to be as basic and low-level software, that does not depend on anything but libc and openssl/zlib?). Looks lime that blob.c code contains implementation of base64 encoding/decoding and murmur hashing (?) algorithms. There might be way to implement the same in python using higher-level constructs. Just saying..

2nd approach looks more logical anyway, especially if it is easier.

By Alexey Min on 2020-06-07T06:38:47

mentioned in merge request pmbootstrap!1946 (merged)

By Oliver Smith on 2020-06-09T02:36:10

Basic first version is ready:

By Oliver Smith on 2020-06-09T02:37:11

assigned to @ollieparanoid

By Oliver Smith on 2020-06-09T02:37:41

closed via merge request pmbootstrap!1946 (merged)

By Bart Ribbers on 2020-06-19T07:47:18

On-device installer

Child items ...

Activity

postmarketos-oem-install-auto, postmarketos-mkinitfs changes

Calamares packaging

Calamares UI

Planned pmbootstrap changes