Newbyte authored 6 months ago and Oliver Smith committed 1 month ago

We currently need this logic for at least two repositories, and the
number might grow with time. Let's deduplicate the code here.

pmbootstrap now depends on kpartx, which is provided by multipath-tools
in alpine.

Related: pmb MR 2450

Works around an issue in the CI after the migration where some dirs are
created that are owned by root and not writeable by any other groups/
users.

One way the CI failed from this is .git/refs/remotes was only
writeable by root, since root ran `git remote add` on the freshly cloned
repo causing the remotes dir to be created. Later in CI, when other
users tried to add remotes, these failed silently (thanks git) and the
CI job crashed when it tried to run `git fetch` on the remote that was
never added. At this point in the script, refs/remotes exists so this
chmod "fixes" the issue.

I don't have any idea why this is a problem *now* and not when we were
on gitlab.com.

Not sure why it is failing currently, add the log.txt to artifacts so we
can figure it out.

pmbootstrap master is now good enough to use as default in CI.

See https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2396

This reverts commit 755b5064.
[ci:skip-build]: already built successfully in CI

Add losetup, so "pmbootstrap shutdown" doesn't fail because it tries to
use busybox losetup with the --json parameter that it doesn't support:

	[11:29:44] % sudo losetup --json --list
	losetup: unrecognized option: json
	BusyBox v1.36.1 (2024-06-12 11:52:11 UTC) multi-call binary.
	Usage: losetup [-rP] [-o OFS] {-f|LOOPDEV} FILE: associate loop devices
		losetup -c LOOPDEV: reread file size
		losetup -d LOOPDEV: disassociate
		losetup -a: show status
		losetup -f: show next free loop device
		-o OFS	Start OFS bytes into FILE
		-P	Scan for partitions
		-r	Read-only
		-f	Show/use next free loop device

Currently running into this with bpo CI while working on
build.postmarketos.org#140.

Put coreutils in its own line, so it's easier to sort the list e.g. in
neovim.

In pmbootstrap v3, the --aports argument currently gets ignored in:
$ pmbootstrap --aports=.... init

As workaround, let pmbootstrap initialized with the default (which means
it clones pmaports to the default dir), and then change the config
afterwards to point to the pmaports dir we actually want to run CI in.

Related: pmb issue 2412
Related: pmb issue 2411

To enable the pmbootstrap refactor, move current master over to a new
2.3.x stable branch. We will then slowly move repos over to the new
version of pmb.

If cloning from a pmbootstrap fork with a different repo name, git will
clone it to an inconsistent directory. Specify the target dir when
cloning.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Fix for:
  $ apk -q add py3-pip
  $ pip3 -q --disable-pip-version-check install flake8
  error: externally-managed-environment

In latest git version a safety check was added that makes sure a git
directory is owned by the same user that tries to modify it. Since this
script runs only in CI, we can just mark it as safe directory and it
should work the same as before.

Fix for:
  Creating pmos user
  Found pmaports.cfg in current dir
  ERROR: failed to add original remote with git!
  fatal: unsafe repository ('/builds/anjandev/pmaports' is owned by someone else)
  To add an exception for this directory, call:
    git config --global --add safe.directory /builds/anjandev/pmaports

Related: https://gitlab.com/anjandev/pmaports/-/jobs/2328801751

Looks like there's a new shellscript version that finds more errors than
before. The errors are non-POSIX syntax that do work with busybox sh,
just ignore it for now since I want to fix another script.

Add tests for:
* installing dependencies
* verifying that binfmt_misc works
* both code paths of add_remote_origin_original

Run install_pmbootstrap.sh with sh -ex, so it's easier to see where it
fails.

Remove /tmp/pmbootstrap if it exists, and create the symlink to
/tmp/pmbootstrap/pmbootstrap.py with force.

Avoid error when running install_pmbootstrap.sh twice:
	mount: mounting none on /proc/sys/fs/binfmt_misc failed: Resource busy

In single runner setups, the gitlab CI runner code appears to keep the
git repositories and only re-initialize them. This preserves the remotes
that were added. If that is the case, just fetch the remote and don't
try to add it again.

Related: https://gitlab.com/postmarketOS/pmaports/-/jobs/748215671

Replace
	"$distfile: failed to find urihash!"
with
	"$distfile: not found in sha512sums! Forgot to generate checksums?"

Verify that each distfile has the same hash in a directory full of APKBUILDs.
Make is used for parallelization. This CI check is not postmarketOS specific.

Let pmbootstrap use the channel of the currently checked out pmaports
branch, if pmaports.git exists already. Otherwise, it would try to
change to the branch of the default channel as specified in channels.cfg
(currently "edge").

Make it easy to test if CI runs through for trivial changes.

Adjust to changes in pmbootstrap!1912: add the original repository, so
pmbootstrap can take the channels.cfg from there.

Don't check if the parent dir is the pmaports dir, but check if the
current dir is pmaports.

There's a pmbootstrap regression that broke pmaports.git CI. Allow to
override the tag, so I can verify my fix.

Let pmbootstrap properly kill processes that are timing out, instead of
having it display the busybox ps usage screen (as we are using arguments
of ps that busybox ps does not support):

[00:15:45] Process did not write any output for 900 seconds. Killing it.
[00:15:45] NOTE: The timeout can be increased with 'pmbootstrap -t'.
ps: unrecognized option: noheaders
BusyBox v1.31.1 () multi-call binary.
Usage: ps [-o COL1,COL2=HEADER]
Show list of processes
	-o COL1,COL2=HEADER	Select columns for display
[00:15:45] ERROR: Command '['ps', '-e', '-o', 'pid=,ppid=', '--noheaders']' returned non-zero exit status 1.

Example: https://gitlab.com/postmarketOS/pmaports/-/jobs/517281598#L6026

Put each dependency in a single line, sort them alphabetically and
change the code that prints the dependencies to still print them as
single line.

It does not make sense to use -q and --details to stdout. Remove -q and
redirect the output to another file, so we even have the log if
pmbootstrap fails before the workdir and logfile are created. Display it
if something goes wrong. While at it, update the message printed on
error.

If pmaports.cfg is not in current dir, then don't pass the current dir
to pmbootstrap as pmaports dir. Instead, let pmbootstrap clone pmaports.