Caleb Connolly authored 1 week ago and Oliver Smith committed 1 week ago

We currently always build kernel packages for the primary channel, which
will usually be systemd-edge. This will cause sideload to fail since it
correctly only looks in the edge repo.

Update envkernel to get the proper channel name and put the kernel
there, so they will always be in the local edge repo and not the
systemd-edge one.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Part-of: !2572

Anri Dellal authored 1 week ago and Oliver Smith committed 1 week ago

Part-of: !2575

Splitting makedepends in _host and _build only makes a difference when
building with cross-native2, as these packages end up in different
chroots.

For QEMU only and crossdirect we install all makedepends into the
foreign chroot. For cross-native (v1) we install everything into the
native chroot.

Show the warning only for cross-native2 so people don't think they have
to adjust their APKBUILDs here. Most of Alpine's APKBUILDs currently
don't have this split either.

Part-of: !2571

Part-of: !2577

Fix cross compiling where CPU emulation is not needed, for example
building x86 packages on x86_64 machines.

This is the case with both:
* CrossCompile.UNNECESSARY
* arch != Arch.native()

Fix the logic in CrossCompile().build_chroot that returned the native
chroot in that case instead of Chroot.buildroot(arch).

Fix error:
  /usr/bin/abuild: line 2692: x86_64-alpine-linux-musl-strip: not found

Part-of: !2577

Anri Dellal authored 1 week ago and Oliver Smith committed 1 week ago

Part-of: !2574

Ruff 0.11.1 is incorrectly printing the following types of errors:

pmb/helpers/logging.py:169:34: SyntaxError: Cannot use star annotation on Python 3.10 (syntax was added in Python 3.11)
    |
169 | def critical(msg: object, *args: str, **kwargs: Any) -> None:
    |                                  ^^^
170 |     logging.critical(msg, *args, **kwargs)
    |

But the code works fine in Python 3.10 (see pytest-python3.10 CI job).
This is a known upstream bug, pin to the previous version until it is
fixed.

Related: https://github.com/astral-sh/ruff/issues/16874
Part-of: !2578

Add a new environment variable that disables apk's caching feature for
space constrained environments such as the bpo image build jobs.

Currently we have a workaround in place in bpo: the apk cache is moved
to a tmpfs. But this is fragile and just disabling the apk cache is a
more elegant solution.

I've decided to make this an env var instead of a full pmbootstrap
option since this option is not relevant for normal users, only for CI
jobs in space constrained environments. Also we already have another
`PMB_APK_` env var.

Related: bpo issue 136
Part-of: !2553

Caleb Connolly authored 1 week ago and Oliver Smith committed 1 week ago

* Make CrossCompile a proper enum type rather than a string literal,
* Introduce methods to get the correct host/build chroots depending on the
  cross compile type and target architecture.
* Remove autodetect.chroot() since it doesn't do what we expect, adjust
  all users to use cross.build_chroot() instead.
* Refactor package building to correctly use cross.host_chroot() and
  cross.build_chroot().

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Part-of: !2568

Part-of: !2568

Part-of: !2568

Replace None with the three things it is actually used for to make the
code easier to follow:
* "autodetect"
* "unnecessary"
* "qemu-only"

I've used the term "unnecessary" instead of "native", because "native"
was previously used for "cross-native", then "cross-native2" and it
still sounds similar to these.

Part-of: !2568

This function doesn't handle the case of being called without a value
set for cross (it does not autodetect it) and callers don't expect that.
So unset the default value.

Part-of: !2568

As discussed with Caleb, it makes sense to keep using "cross-native" for
packages where we don't need to set up a foreign chroot because it is
faster.

Part-of: !2568

Use the same names as in the documentation for:
"cross-native" (not "kernel")
"cross-native2" (not "native")

Part-of: !2568

Building without !tracedeps is a problem with cross-native version 1 but
not version 2. Fix the warning.

Fixes: 2ee916f5 ("build: add pmb:cross-native2 (MR 2474)")

Part-of: !2568

Oliver Smith authored 2 weeks ago and Caleb Connolly committed 2 weeks ago

* makedepends_build is for the native chroot
* makedepends_host is for the foreign chroot

I had it the other way around in the documentation. The variables are so
far not documented here yet:
* https://gitlab.alpinelinux.org/alpine/abuild/-/blob/master/APKBUILD.5.scd
* https://wiki.alpinelinux.org/wiki/APKBUILD_Reference

But it now matches the comments Caleb had written for the cross-native2
implementation, and also what can be found in e.g. the abuild APKBUILD
in aports.git:

  makedepends_build="pkgconfig scdoc"
  makedepends_host="openssl-dev>3 zlib-dev"

Part-of: !2566
[ci:skip-build]: already built successfully in CI

Co-authored-by: Oliver Smith <ollieparanoid@postmarketos.org>
Part-of: !2538

This adds support for using an environment variable to set the FDE
passphrase, allowing us to automate image creation when using FDE.
The method used here was borrowed from how we set the password with when
using the --password arg: write to a temp file, call something in the
chroot to read/use it, then remove it.

Part-of: !2538

Frieder Hannenheim authored 2 weeks ago and Oliver Smith committed 2 weeks ago

Currently packages that are not in the systemd channel cannot be
sideloaded when using the systemd channel. This is because apk sideload
uses the global aports channel and not the channel of the package. Fix
it with this patch.

Newbyte authored 1 month ago and Oliver Smith committed 2 weeks ago

We fully expect to reach this code path within normal use of
pmbootstrap. It's not a bug if we get here. As such, use NonBugError to
indicate that and clean up the output.

Newbyte authored 1 month ago and Oliver Smith committed 2 weeks ago

We fully expect to reach this code path within normal use of
pmbootstrap. It's not a bug if we get here. As such, use NonBugError to
indicate that and clean up the output.

Newbyte authored 1 month ago and Oliver Smith committed 2 weeks ago

We fully expect to reach this code path within normal use of
pmbootstrap. It's not a bug if we get here. As such, use NonBugError to
indicate that and clean up the output.

Newbyte authored 1 month ago and Oliver Smith committed 2 weeks ago

The install action will initialise the native chroot, so this way we
don't have to initialise the chroot in this code path. It also gives
users feedback quicker if they haven't already initialised the rootfs
since this way they don't have to wait around for nbd to be installed
before being told that they need to generate the rootfs first.

Closes #2549

Cross compiling go programs with cross-native2 fails for armv7, riscv64
and armhf unless we use CGO_ENABLED=1 in the APKBUILD (see related
aports issue).

Add --sysroot to CGO_CFLAGS, so cgo can find the includes and doesn't
fail with:

  go build -buildmode=pie -modcacherw -trimpath -buildvcs=false -ldflags "-s -w -X main.Version=2.6.1" -o mkinitfs ./cmd/mkinitfs
  # runtime/cgo
  _cgo_export.c:3:10: fatal error: stdlib.h: No such file or directory

Add it to LDFLAGS too for linking against libraries from the sysroot.

Related: https://gitlab.alpinelinux.org/alpine/aports/-/issues/15809
Related: https://stackoverflow.com/a/38835938

This is needed for the linter to know about this option.

Fixes 2ee916f5

Fixes a bug where checkdepends are not being installed, and packages
that run tests that depend on something in checkdepends fail because
it's missing. For example, the `plasma-workspace` forked pkg in pmaports
tells abuild to run `xwfb-run` in `check()`, this binary comes from
`xwayland-run`, which is listed in the `checkdepends`. However this
isn't being installed and so `pmb build` ultimately fails to build the
package.

Add the cross compiling documentation to the pmbootstrap docs.

This is based on the following wiki pages, but updated to reflect new
changes with cross-native2 and to make it more readable:
* https://wiki.postmarketos.org/wiki/Pmbootstrap/Cross_Compiling
* https://wiki.postmarketos.org/wiki/Build_internals#Cross-compile_types



Co-authored-by: Caleb Connolly <caleb@postmarketos.org>

Speed up compressing packages after build by using pigz which does
parallelised compression!

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 1 month ago and Oliver Smith committed 1 month ago

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 2 months ago and Oliver Smith committed 1 month ago

Set things up so that we can run abuild on the native chroot and use
it's cross compilation features rather than running it and the build
system through QEMU. This massively speeds up building when it works.

cross-native used to be quite limited in functionality and didn't
integrate into abuild itself, this commit fixes that.

Packages can opt-in to this by adding pmb:cross-native2 to their options
and configuring makedepends_host and makedepends_build.

This also speeds up building packages like postmarketos-initramfs since
it entirely avoids running commands through QEMU (usually abuild itself
would be run through QEMU).

Lastly, we preserve the old pmb:cross-kernel options, this can be used to
enable the old cross compiler behaviour which is used for cross
compiling kernels in pmaports. This allows them to keep being supporting
while we adapt them to the new cross-native2.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 1 month ago and Oliver Smith committed 1 month ago

Rather than a list which we later convert to a set.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 2 months ago and Oliver Smith committed 1 month ago

Map architectures to the strings used for GOARCH for cross compiling Go
applications.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 2 months ago and Oliver Smith committed 1 month ago

The .supported() method errs on the side of exposing too many
architectures to make porting for new or less-common platforms easier.

It therefore isn't suitable for using as a list of supported
architectures we can probe the binary repository for.

Introduce a .supported_binary() method to export only the architectures
where we have a binary repository.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Instead of automatic python package discovery that for some reason still
picks up the test module even though it should be excluded, specify the
"pmb" module manually to be installed.

Follow-up to MR 2551.

Fixes: issue 2541

Newbyte authored 3 months ago and Oliver Smith committed 1 month ago

I didn't really understand mypy's overloading well when I wrote this. It
makes sense to have a third overload here with bool as type for
multiple_providers used in circumstances where a literal isn't provided
for that parameter.

Newbyte authored 3 months ago and Oliver Smith committed 1 month ago

Having a more generic overload like this fixes the issue where mypy
complains about no matching overloads at the call site in userm().

Anri Dellal authored 1 month ago and Oliver Smith committed 1 month ago

According to Python documentation, p.match checks
if characters at the beginning of string match the regex

This commit changes validation to full length of string