32-bit packages missing on edge

As announced, we've deleted our 32-bit packages on edge to rebuild them against Alpine.

Currently we have ~50 failed armv7 packages in the queue, which mostly failed due to missing depends from Alpine. Today Alpine is building all packages on build-edge-armv7, with the builder configured to continue with the next package even if a build fails (usually it would wait forever at the failing package). When it went over all aports in community once, it should upload all successfully built community packages to the mirror (probably until tomorrow?) - and then these failing packages from our end should mostly build. We need to reset their status once to let them start building again though.

However, the armhf packages were not added to the queue for some reason. I'll analyze why this is the case. Once we add them to the queue, they should get pretty far with building - since Alpine just did the same thing with build-edge-armhf, but it finished already and therefore has uploaded all the passing community packages.

Overview

• armv7
• armhf
• x86

added type::bug label

Status update: packages are mostly available in Alpine again. Exceptions:

• evtest
  • I've prepared a patch to fix the build for armv7, armhf: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/10754
  • I'll put it in pmaports.git temp until it's merged and published in Alpine, so this is not a blocker
• plasma, plasma-pa, plasma-workspace, powerdevil
  • doesn't build because of a circular dependency: https://gitlab.alpinelinux.org/alpine/aports/-/issues/11785
  • I'll disable packages depending on those for armv7 for now (already disabled for armhf). -> https://gitlab.com/postmarketOS/pmaports/-/commit/35015139f36f091229f715ef821964b3911e0b6c
  • This means, plasma mobile will not be available on armv7 until it's resolved
• qt5-qtwebengine
  • Doesn't build: https://gitlab.alpinelinux.org/alpine/aports/-/issues/11791
  • I'll disable packages depending on that for armv7 as well. -> https://gitlab.com/postmarketOS/pmaports/-/commit/08ce273d6f136c3f75f88638734c1b4b8fdb908f

With these resolved / worked around, the armv7 pmOS repository should build again without errors, and once that is done, it will be published and available again.

For armhf, I'm still investigating why the packages are not properly queued. Building everything for armv7 will take some time anyway though.

EDIT: there's also x86, which almost nobody is using (we only have one booting device with x86). The situation is similar to armhf, the packages are not queued yet. I'll probably resolve both at the same time.

By Oliver Smith on 2020-07-29T10:23:17

mentioned in commit pmaports@35015139

By Oliver Smith on 2020-07-29T10:20:04

mentioned in commit pmaports@08ce273d

By Oliver Smith on 2020-07-29T10:22:38

mentioned in commit postmarketos.org@4d6d0498

By Oliver Smith on 2020-07-29T13:51:18

armv7 should be up in a few hours again. The only failing package is mesa-git, which should be fixed with https://gitlab.com/postmarketOS/pmaports/-/commit/983bc8318bf2fb696d8e0845764b55f5adba9a78. Besides that, three few device packages depending on mesa-git are in the queue, I expect them to build quickly and without problems. Afterwards, the repo will be published and work again as usually.

The circular dependency problem in Alpine's KDE packaging has been fixed; once the armv7 repository is published, we'll revert https://gitlab.com/postmarketOS/pmaports/-/commit/35015139f36f091229f715ef821964b3911e0b6c to enable plasma related packages on armv7 again. (So initially after the repository publish, these plasma packages - especially plasma mobile UI - will be missing, but it will be up soon again as well.)

After that, armhf and x86 repository will be rebuilt and then we are back to normal. Most phones are actually armv7 or aarch64, so having armv7 up again should resolve the problem for most users already.

By Oliver Smith on 2020-07-30T11:03:31

I will test on grouper as soon as it's in line.

By gullradriel on 2020-07-30T12:24:37

Looks like things are up at http://postmarketos1.brixit.nl/postmarketos/master/armv7 The build started successfully...

By gullradriel on 2020-07-30T12:54:04

I'm still getting errors when building. :(

By Newbyte on 2020-07-30T12:54:17

Missing packages here too.

By gullradriel on 2020-07-30T12:54:34

Failing on these 3 at the moment:

ERROR: sdl2-2.0.12-r2: BAD signature
ERROR: sdl2_ttf-2.0.15-r0: BAD signature
ERROR: sdl2_ttf-dev-2.0.15-r0: BAD signature

By gullradriel on 2020-07-30T12:58:25

Here's what I'm seeing when building:

ERROR: sdl2_ttf-2.0.15-r0: BAD signature
ERROR: libndp-1.7-r0: BAD signature
ERROR: libgudev-233-r1: BAD signature
ERROR: libcanberra-gtk3-0.30-r5: BAD signature
ERROR: libwpe-1.6.0-r1: BAD signature
ERROR: libwpebackend-fdo-1.6.1-r0: BAD signature
ERROR: hyphen-2.8.8-r1: BAD signature
ERROR: glib-networking-2.64.3-r0: BAD signature
ERROR: libwoff2enc-1.0.2-r1: BAD signature
ERROR: gsound-1.0.2-r1: BAD signature
ERROR: sbc-1.4-r0: BAD signature
ERROR: clutter-gtk-1.8.4-r1: BAD signature
ERROR: alsa-plugins-pulse-1.2.2-r1: BAD signature
ERROR: upower-0.99.11-r0: BAD signature
ERROR: libindicator-12.10.1-r0: BAD signature
ERROR: gnome-bluetooth-3.34.1-r0: BAD signature
ERROR: sane-udev-1.0.29-r0: BAD signature
ERROR: libpulse-mainloop-glib-13.0-r10: BAD signature
ERROR: startup-notification-0.12-r4: BAD signature

By Stephen Smith on 2020-07-30T14:48:31

Thanks for reporting. I can reproduce the "BAD signature errors". This is caused by mirrors from Alpine still having the old packages - they have the same filename, but different checksums.

As workaround, run with a specific alpine mirror, which has the new packages for sure. Run "update" to make sure the APKINDEX (index of packages from the repository) is downloaded, then use the mirror with "install":

$ pmbootstrap --mirror-alpine "http://dl-2.alpinelinux.org/alpine/" update
$ pmbootstrap --mirror-alpine "http://dl-2.alpinelinux.org/alpine/" install

EDIT: with these two commands, pmbootstrap install goes through for asus-grouper with UI: phosh (no idea if/how well phosh actually works with grouper, I just had it selected already).

EDIT 2: to explain it a bit better: by default, pmbootstrap uses the dl-cdn mirror, which (randomly?) selects one of the available mirrors - and therefore may give you one that still has the old packages. Usually this is handled much better, but with this breaking change, not all mirrors seem to have cought up properly. I'll bring this up with Alpine too to make sure they are aware.

EDIT 3: so dl-cdn is a geographical cache for some other mirror, not selecting one of the other mirrors at random. ikke from Alpine will purge the cache, that should help with the BAD signature errors. We will need further iterations of purging the cache when an old package that was not building in Alpine is now built against the new musl version and has a different hash. If somebody runs into the BAD signature errors again, just use the workaround above for now.

One can also delete the old apk files to free up disk space (should not be necessary to make it work AFAIK, only to save up disk space):

$ rm -rf ~/.local/var/pmbootstrap/cache_apk_armv7

List of mirrors: http://dl-cdn.alpinelinux.org/alpine/MIRRORS.txt

By Oliver Smith on 2020-07-30T16:31:34

Thanks. This seems to be working. However, initially it gave me an error stating that 'cryptsetup' could not be found, but a pmbootstrap zap resolved that.

By Newbyte on 2020-07-30T16:19:35

Thanks for this, I zapped my chroots and deleted the pmbootstrap folder, then re-initted and used pmbootstrap --mirror-alpine "http://dl-2.alpinelinux.org/alpine/" install --android-recovery-zip. This seems to have successfully built pmos-htc-m7.zip, but I'm having issues when installing it from TWRP. I'm not sure if it's at all related to this, but:

cp: can't stat '/emmc@*': No such file or directory
findfs: unable to resolve 'PARTLABEL=system'                                                                                                    
findfs: unable to resolve 'PARTLABEL=boot'
findfs: unable to resolve 'PARTLABEL=system'

I also tried pmbootstrap flasher --method=adb sideload as recommended in the wiki for the m7 but I get ERROR: Could not find dependency 'android-tools' in any aports folder or APKINDEX.

By Stephen Smith on 2020-07-30T17:18:59

Thanks for the follow-ups, @Newbyte and @stephen304!

I'm not sure if it's at all related to this, but:

A separate issue in https://gitlab.com/postmarketOS/postmarketos-android-recovery-installer/ would be the place to put this. The fastest way to figure this out, is probably looking at the source, figuring out where that cp statement is that doesn't work, and fixing it up to use the right argument. Good luck!

By Oliver Smith on 2020-07-30T17:27:41

It's booting to shell on grouper. I have to investigate why it's not booting to xfce4.

By gullradriel on 2020-07-30T18:19:14

Looks like it is a lightdm session issue

By gullradriel on 2020-07-30T19:00:24

Ok. it's finally building and flashing fine on grouper (both kernel and rootfs). BTW there is still no automatic adb push when trying to flash rootfs using pmbootstrap. OTOH now I'm on the system and everything seems fine, I still have some bad signatures when trying an update:

ERROR: libvpx-1.8.2-r0: BAD signature
ERROR: x264-libs-20191119-r1: BAD signature
ERROR: sbc-1.4-r0: BAD signature
ERROR: gsettings-desktop-schemas-3.36.1-r0: BAD signature
ERROR: libproxy-0.4.15-r6: BAD signature
ERROR: glib-networking-2.64.3-r0: BAD signature
ERROR: libsoup-2.70.0-r0: BAD signature
7 errors; 598 MiB in 372 packages

May I try with a different repo or it's still not ok ?

By gullradriel on 2020-08-01T07:47:52

Also got it fixed by using http://dl-2.alpinelinux.org/alpine/edge/ in apk repositories. As far as I can see everything looks fine on grouper now, good job !

By gullradriel on 2020-08-01T08:08:24

changed the description

By Oliver Smith on 2020-07-30T16:17:42

Any progress on armhf?

By Firstbober on 2020-07-30T16:46:03

I'm on it. For some reason, the armhf packages are not queued yet on build.postmarketos.org. I think I found the cause in the code, doing further testing and preparing a fix. I'll update this issue when there's something new.

By Oliver Smith on 2020-07-30T16:46:02

The bug is fixed (903cdd6d) and armhf packages as well as x86 packages get added to the queue now. You can see the progress at https://build.postmarketos.org. It will take time until they are all built.

By Oliver Smith on 2020-07-30T17:05:34

armhf repo is back :)

By Oliver Smith on 2020-08-03T08:54:04

I'm unable to install neither Phosh nor Plasma Mobile on armv7. Not sure what the Phosh issue is exactly, but Plasma Mobile gives me ERROR: Package 'kaccounts-providers': Could not find aport, and could not find this package in any APKINDEX! Tried 2 different mirrors and default.

Update: Phosh seems to install fine now

By Newbyte on 2020-07-31T18:59:53

marked the checklist item armhf as completed

By Oliver Smith on 2020-08-03T08:51:03

marked the checklist item x86 as completed

By Oliver Smith on 2020-08-03T10:29:20

I've released a new pmbootstrap version 1.22.1 that uses a different mirror by default. This avoids the "BAD signature" errors for the most part. (If it still happens after switching the mirror, then it is this issue - but only very few packages seem to be affected.)

Plasma mobile is temporarily disabled on armv7, because of missing binary packages (plasma-pa, plasma-workspace) in Alpine edge. This will likely be resolved soon. Some UI and other packages have been disabled in x86 as well for similar reasons, but that doesn't seem to be used much. Find details in the commit log.

All postmarketOS repositories are back online now!

Closing this issue. Thanks to everybody for their patience and understanding!

By Oliver Smith on 2020-08-03T10:36:54

closed