Oliver Smith authored 6 months ago and Luca Weiss committed 5 months ago

When running some commands like "pmbootstrap chroot" or even
"pmbootstrap work_migrate", pmbootstrap will attempt to read the
pmaports.cfg from origin/master *before* doing the work dir migration.

In order to do this, the "get_upstream_remote" function tries to find
the upstream remote by URL. Let it search with the outdated URLs too, so
this doesn't fail right before migrating to the new URLs.

(cherry picked from commit a6d34d66)

Newbyte authored 6 months ago and Luca Weiss committed 5 months ago

This could be re-purposed in the future in case we migrate URLs again.

Co-authored-by: Caleb Connolly <caleb@postmarketos.org>
Tweaked-by: Oliver Smith <ollieparanoid@postmarketos.org>
(cherry picked from commit 344d6f22)
[luca: backport to 2.3.x, add "2.x" suffix to workdir]

Try to finish the migration by changing various links and texts to point
to the new GitLab instance.

(cherry picked from commit 1153079f)

Newbyte authored 6 months ago and Luca Weiss committed 5 months ago

We moved from gitlab.com to gitlab.postmarketos.org.

(cherry picked from commit b1b6c52b)

Andrew Robbins authored 10 months ago and Oliver Smith committed 9 months ago

btrfs can only create a single subvolume at a time.

Fix the "upstream compat" test that is currently failing with:
  Increase 'postmarketos-ui-phosh' pkgrel (6 -> 7), missing depend(s): phosh-service

The mentioned dependency is in the postmarketos-ui-phosh-systemd
subpackage, and the phosh-service package is intentionally not provided
by anything in pmaports master right now. It will be there once we merge
the master_staging_systemd branch with master, which requires
pmbootstrap and bpo changes first.

Make the "upstream compat" test useful again by not failing on this for
now.

This patch is for the 2.3.x branch of pmbootstrap, which is used by CI
while we prepare for pmbootstrap v3.

Fixes: montoring issue 8

Luca Weiss authored 10 months ago and Oliver Smith committed 10 months ago

Tweaked-By: Oliver Smith <ollieparanoid@postmarketos.org>

Luca Weiss authored 10 months ago and Oliver Smith committed 10 months ago

Seems GitLab CI doesn't really care but still fix it.

Newbyte authored 10 months ago and Oliver Smith committed 10 months ago

Alpine 3.20 has released now which should contain everything we need to
run our CI.

Closes https://gitlab.com/postmarketOS/pmaports/-/issues/2747

Fixes: 0d6c03b2 ("PMB_APK_FORCE_MISSING_REPOSITORIES: new env var (MR 2318)")

This is needed to bring up the v24.06 repositories at
build.postmarketos.org. With the latest apk version, apk refuses to
operate if an URL from /etc/apk/repositories cannot be fetched.

Before the repositories are created for the first time, they do not
exist, so we will just set PMB_APK_FORCE_MISSING_REPOSITORIES=1 in bpo
to be not blocked here.

I've also spent significant time on alternative implementations, but
they have problems:
- Let bpo create an empty APKINDEX before building the first package,
  but this was a larger code change, leading to lots of adjustments in
  the tests, and ultimately it seems it didn't work properly (it seems
  apk/abuild doesn't create a valid signed APKINDEX for one that has no
  packages).
- Do not set the --mirror-pmOS argument for the "final" repository, only
  the "wip" repository, until the "final" repository is available for
  the first time. This works fine for x86_64, but not for foreign arch
  repositories because then the cross compilers from the x86_64
  repository are not available. I've also tried to make a different env
  var that ensures we don't write the non-existing repository to
  /etc/apk/repositories from within pmbootstrap if initializing a
  foreign arch chroot, but then we would find a sane way to do this only
  for the "final" repository and not for the "wip" repository which
  leads to a lot more complexity than this patch.

So this is not the nicest solution (apk still tries to fetch the indexes
and gets a 404), but it is the simplest one and unblocks us from working
on v24.06. Also it doesn't add more complexity which is important in the
middle of the feature freeze we are currently in.

Related: bpo issue 137
Related: https://gitlab.alpinelinux.org/alpine/apk-tools/-/commit/d76213e643c39da7d3b4e5534b721238ad7e8baf
Related: https://postmarketos.org/blog/2024/05/19/pmOS-update-2024-05/#pmbootstrap-230-and-feature-freeze

Clayton Craft authored 10 months ago and Oliver Smith committed 10 months ago

This option is required for kernels to support mounting the ESP as /boot
after pmaports bb6d7a05b

Val Packett authored 10 months ago and Oliver Smith committed 10 months ago

Fixes "UnboundLocalError: cannot access local variable 'filesystem'
where it is not associated with a value".

Robert Eckelmann authored 10 months ago and Newbyte committed 10 months ago

as reported in #2355 pmmbootstrap wheel installs files to /usr/lib/python3.12/site-packages/docs

Prepare for postmarketOS v24.06.

Set the current apk versions as min version.

Newbyte authored 11 months ago and Pablo Correa Gomez committed 10 months ago

See https://gitlab.com/postmarketOS/pmaports/-/merge_requests/5046

Clayton Craft authored 11 months ago and Newbyte committed 10 months ago

This is needed for sphinx autoprogram since that expects an
argparse.ArgumentParser, and arguments() returns some argparse
"Namespace" obj. Useful for sphinx/autoprogram and maybe other things
later that want to get at pmb's full args.

Pablo Correa Gomez authored 11 months ago and Clayton Craft committed 11 months ago

linux-lts and linux-edge in alpine use "utf-8", that might change, but
add the options regardless, so we can be safe (people can always edit them!)

Fixes https://gitlab.com/postmarketOS/pmaports/-/issues/2782
[ci:skip-build]: already built successfully in CI

Pablo Correa Gomez authored 11 months ago and Clayton Craft committed 11 months ago

We don't want executables, suid executables, or devices. We neither
want symbolic links in fat partitions. These have been taken from
systemd

Ref: pmaports#2782

Pablo Correa Gomez authored 11 months ago and Clayton Craft committed 11 months ago

The "fat16" and "fat32" mount types do not exist. It is "vfat"

See: https://gitlab.com/postmarketOS/pmaports/-/issues/2652

This test is *only* failing in gitlab CI and I haven't been able to
figure out why (see [1]). Everything else (e.g. bpo, building packages
locally) seems to work fine. Since this is blocking pmbootstrap MRs, I'm
going to disable the test until we can figure out wtf is going on with
running it in gitlab CI.

1. https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2346
[ci:skip-build]: already built successfully in CI

* Replace aports -> pmaports
* Make the "binary repo has newer version" message much shorter, and
  mention "pmbootstrap pull". If users didn't mess with their pmaports
  repository (checkout a custom branch, make commits), this command
  will update to the latest commit and resolve the warning. This also
  ties into MR 2294 where I removed a feature from "pmbootstrap status"
  that would complain about the last fetch of pmaports.git being too old
  and was also recommending "pmbootstrap pull". I think having it here
  in the warning makes more sense, as more people will see it.

* Remove obvious / not helpful line:
  "# Get package name, version, define start of debug message"
* Remove "old" in "Get old version from APKINDEX" because it may be
  newer than the version in pmaports
* Replace "Aports [folder]" with pmaports (in some parts of pmbootstrap
  code, pmaports is still referred to as aports, this was one of them)

Give more meaningful names to the variables:
* version_new -> version_pmaports
* version_old -> version_binary

This makes the code less confusing for the case where version_binary is
actually newer than version_pmaports. This is a relict from the time
before there was a binary package repository, in that case the version
from pmaports would always be the newer one, built from source, compared
to the local binary package that was probably built before.

Extend the testdata and test case to show that the parser can now handle
custom variables.

A common pattern in APKBUILDs, is to introduce custom variables prefixed
with underscores that get then used in makedepends and other variables.

For example:

  _wlrootsmakedepends="
	eudev-dev
  	# ...
  	"
  makedepends="
  	# ...
  	$_wlrootsmakedepends
  	"

Adjust the APKBUILD parser code, so it parses all top-level variables
and can use them further below when referenced inside other variables.
Before returning the parsed APKBUILD data, remove all variables that are
not in pmbootstrap's list of known APKBUILD parsing attributes (so the
result is the same).

I've compared "pmbootstrap apkbuild_parse" (which parses all APKBUILDs
in the currently checked out pmaports dir), before and after this
change, and the result is the same except for having more variables
successfully replaced.

- Performance Note-
This new implementation is actually faster than the previous one,
because we don't need to iterate through all known keys on each line of
the APKBUILDs. On my machine, average of 3 runs, parsing all APKBUILDs
of current pmaports master takes about half as long as with the previous
implementation.

  $ time pmbootstrap -q apkbuild_parse >/dev/null

-> old code: 0.954
-> new code: 0.483

Add the "args" pytest.fixture to each test as argument, so it runs the
related code of initializing logging and the cache (see top of the
test_parse_apkbuild.py). Without this, running the test file without any
prior tests (that do these initializations) is not possible:

  $ pytest -xvv test/test_parse_apkbuild.py
  …
  >       if path in pmb.helpers.other.cache["apkbuild"]:
  E       TypeError: 'NoneType' object is not subscriptable

Make the handling of the custom NonBugError and BuildFailedError
exceptions more consistent with the handling of other exceptions, by
printing "ERROR: " infront of the actual error text. Then we don't need
to duplicate that where we raise the errors. pmbootstrap prints "ERROR"
in red.

Clayton Craft authored 11 months ago and Oliver Smith committed 11 months ago

Useful for pmaports CI, see:
https://gitlab.com/postmarketOS/pmaports/-/merge_requests/4994#note_1842433422

Reimplement "pmbootstrap status" to be just a simple and useful status
overview. The previous version ran a bunch of checks every time, and
would fail on these even if pmaports was used for normal development:
  * "non-official" branch checked out in pmaports
  * pmaports.git is not clean

The information about aports.git was also considered not so useful upon
revisiting this command, since it is only used for "pmbootstrap
aportgen". Most users don't need this, and if the user runs this
command, it will tell if aports.git is outdated.

All of the above made the previous version unpleasant to use and I
suspect most people stopped using the command after trying it out a few
times and seeing the irrelevant but loud NOK complaints.

New version:

  $ pmbootstrap status
  Channel: edge (pmaports: master_staging_systemd)
  Device:  qemu-amd64 (x86_64, kernel: virt)
  UI:      console
  systemd: no (default for selected UI)

Old version (without --details it only shows NOK checks):

  $ pmbootstrap status --details
  [00:55:20] *** CONFIG ***
  [00:55:20] Device: qemu-amd64 (x86_64, "QEMU amd64")
  [00:55:20] Kernel: virt
  [00:55:20] User Interface: console
  [00:55:20]
  [00:55:20] *** GIT REPOS ***
  [00:55:20] Path: /home/user/.local/var/pmbootstrap/cache_git
  [00:55:20] - aports_upstream (master)
  [00:55:20] - pmaports (master)
  [00:55:20]
  [00:55:20] *** CHECKS ***
  [00:55:20] [OK ] Chroots zapped recently (or non-existing)
  [00:55:20] [OK ] aports_upstream: on official channel branch
  [00:55:20] [OK ] aports_upstream: workdir is clean
  [00:55:20] [OK ] aports_upstream: tracking proper remote branch 'origin/master'
  [00:55:20] [OK ] aports_upstream: up to date with remote branch
  [00:55:20] [OK ] aports_upstream: remote information updated recently (via git fetch/pull)
  [00:55:20] [OK ] pmaports: on official channel branch
  [00:55:20] [OK ] pmaports: workdir is clean
  [00:55:20] [OK ] pmaports: tracking proper remote branch 'origin/master'
  [00:55:20] [OK ] pmaports: up to date with remote branch
  [00:55:20] [OK ] pmaports: remote information updated recently (via git fetch/pull)
  [00:55:20]
  [00:55:20] NOTE: chroot is still active (use 'pmbootstrap shutdown' as necessary)
  [00:55:20] DONE!

Newbyte authored 11 months ago and Oliver Smith committed 11 months ago

In case the Docker image is outdated, this means we get the latest
bug fixes and such anyway.

Newbyte authored 11 months ago and Oliver Smith committed 11 months ago

Both Ruff and Codespell have been moved to the community
repository in Alpine:

 - https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/62711
 - https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/64338