ci: qemu sometimes segfaults when build testing other archs

@JustSoup321 noticed this in a recent CI run (build-aarch64), the runner was ci-x64-runner3-docker-postmarketos (x86_64 CPU):

>>> device-pine64-pinephonepro: Analyzing dependencies...
qemu-aarch64-static: QEMU internal SIGSEGV {code=128, addr=0}
Segmentation fault (core dumped)
< that repeats a few more times ... >
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)
>>> ERROR: device-pine64-pinephonepro: builddeps failed

Important

If you noticed this failure, please comment below with the runner! The runner name is usually visible on the right side of the CI job page, or at the top of the CI job output, like:

Running with gitlab-runner 17.5.2 (c6eae8d7)
  on 5: ci-x64-runner3-docker-postmarketos hnB2RHmXb, system ID: s_5433359204a5
...

Knowing whether or not this happens on specific runners could be helpful for identifying the problem.

added ci label

I noticed script exited with error 127 on https://gitlab.postmarketos.org/postmarketOS/pmaports/-/jobs/1278265

ci-x64-runner4-docker-postmarketos

mentioned in merge request pmaports!5688 (merged)

Saw it happen on x64-runner-2, when building aarch64 packages: https://gitlab.postmarketos.org/postmarketOS/pmaports/-/jobs/1280001/raw

Saw it happen on x64-runner-3, when building armv7 packages: https://gitlab.postmarketos.org/postmarketOS/pmaports/-/jobs/1280002

I tried an experiment where I told CI to use aarch64 runners for ARM build testing (so no binfmt/qemu involved), and I see a failure that looks familiar. This was an aarch64 runner building stuff for armv7:

ERROR: busybox-1.37.0-r5.post-install: script exited with error 127
(4/24) Installing busybox-binsh (1.37.0-r5)
(5/24) Installing alpine-baselayout (3.6.8-r0)
Executing alpine-baselayout-3.6.8-r0.pre-install
ERROR: alpine-baselayout-3.6.8-r0.pre-install: script exited with error 127
Executing alpine-baselayout-3.6.8-r0.post-install
ERROR: alpine-baselayout-3.6.8-r0.post-install: script exited with error 127
(6/24) Installing ifupdown-ng (0.12.1-r6)
(7/24) Installing libcap2 (2.70-r0)
(8/24) Installing openrc (0.55.1-r2)
Executing openrc-0.55.1-r2.post-install
ERROR: openrc-0.55.1-r2.post-install: script exited with error 127
(9/24) Installing mdev-conf (4.7-r0)
(10/24) Installing busybox-mdev-openrc (1.37.0-r5)
(11/24) Installing alpine-conf (3.18.1-r2)
(12/24) Installing alpine-keys (2.5-r0)
(13/24) Installing alpine-release (3.21.0_alpha20240923-r0)
(14/24) Installing ca-certificates-bundle (20240705-r0)
(15/24) Installing libcrypto3 (3.3.2-r3)
(16/24) Installing libssl3 (3.3.2-r3)
(17/24) Installing ssl_client (1.37.0-r5)
(18/24) Installing zlib (1.3.1-r2)
(19/24) Installing apk-tools (2.14.4-r4)
(20/24) Installing busybox-openrc (1.37.0-r5)
(21/24) Installing busybox-suid (1.37.0-r5)
(22/24) Installing scanelf (1.3.8-r1)
(23/24) Installing musl-utils (1.2.5-r5)
(24/24) Installing alpine-base (3.21.0_alpha20240923-r0)
Executing busybox-1.37.0-r5.trigger
ERROR: busybox-1.37.0-r5.trigger: script exited with error 127

https://gitlab.postmarketos.org/postmarketOS/pmaports/-/jobs/1280045

I cannot reproduce this locally on my aarch64 laptop, when building for armv7

Saw it happen on x64-runner3, when building aarch64 packages: https://gitlab.postmarketos.org/postmarketOS/pmaports/-/jobs/1280596

closed

reopened

Happend on No 5 (hnB2RHmX) ci-x64-runner3-docker-postmarketos, in https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/jobs/1280550 as part of Pipeline for pmbootstrap!2472 (merged)

And another one.. on No 6 (3Ek4xzsq) ci-x64-runner4-docker-postmarketos, in https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/pipelines/209584 as part of Pipleline for pmbootstrap!2473 (merged)

Thanks for posting the reports. Please add new reports when you see more failures, this helps with finding a pattern and shows whether this is still relevant.

I've opened an issue with OSUOSL about it (support ticket ID #33621).

(feel free to edit and extend lists below)

EDIT: more segfaults:

• #6 (3Ek4xzsq) ci-x64-runner4-docker-postmarketos https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/jobs/1281440#L2720

EDIT: I also see mvdan.cc/sh/interp: /usr/lib/go/pkg/tool/linux_arm64/compile: fork/exec /usr/lib/go/pkg/tool/linux_arm64/compile: exec format error happening only sometimes, not sure if that is caused by the same problem:

• #7 (wMhJs9ZX) ci-x64-runner5-docker-postmarketos https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/jobs/1282049

moved from pmaports#3275 (moved)

changed milestone to %Remaining issues after gitlab migration

• https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/jobs/1283569
• https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/jobs/1283570

EDIT: actually that is a different issue unrelated to CI runners -> pmbootstrap#2497 (closed)

mentioned in issue pmbootstrap#2497 (closed)

I suspect this is a bug in the overlayfs kernel driver, possibly upgrading the kernel would fix it. OSUOSL recommended that we upgrade one node from Alma Linux 8 to 9 and see if that fixes it, so we'll do that.

FYI some time ago I added each runner name as a tag for each runner, so you should be able to target jobs on individual runners by using the tag... e.g. to only run jobs on runner4, in .gitlab-ci.yml add something like this for the job:

tags:
    - runner4

Switching to Alma Linux 9 did not fix it: pmbootstrap!2492 (comment 456347)

Not sure if we can upgrade the kernel or if it already is on the latest patch level. If that does not help it either, using BTRFS as storage driver is what we will try next.

mentioned in merge request pmbootstrap!2492

mentioned in commit Minecrell/pmaports@5752c31a

mentioned in merge request pmaports!5813 (merged)