Refactor the integration test script to support running a variety of
tests, and add CI jobs for them.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

We run pmbootstrap with --details-to-stdout now, so this isn't needed.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Write some basic parser tests for deviceinfo. The parser is quite basic
so there isn't much to test yet. Expecting this to gain features as we
look at e.g. making the parser the "source of truth" for the deviceinfo
format.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

This works around https://gitlab.postmarketos.org/postmarketOS/
pmaports/-/issues/3268, by always installing hexdump when building
packages. The full hexdump is small (like 80KB), and even though this
isn't required in most situations it seems like the best of a group
of bad workarounds, until upstream/busybox fix the bug in the pmaports
issue.

So we don't go through the pain of pushing 3 times to get 3 different
ruff failures in CI.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Add a test to parse some APKINDEX data.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

should be an int, but was always a string...

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

I guess this is leftover from some experimentation.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Also adapt pmb/install/recovery.py to new API for variables.py.

[ci:skip-build]: already built successfully in CI

Clayton Craft authored 5 months ago and Newbyte committed 5 months ago

Sometimes these fields are not set, e.g.:

pdb> p ret
{'depends': ['build-base', 'bash', 'bison', 'findutils', 'flex', 'musl-dev', 'openssl-dev', 'perl', 'python3'], 'arch': 'noarch', 'version': '20241022.022230', 'pkgname': '.makedepends-linux-lenovo-21bx', 'provides': []}

I hit this crash when I installed the build deps for the
linux-lenovo-21bx package in the native chroot, and then ran
`pmbootstrap stats`
[ci:skip-build]: already built successfully in CI

Newbyte authored 5 months ago and Casey committed 5 months ago

It doesn't make sense to both request the nconfig UI and the xconfig UI
at the same time. Make them mutually exclusive.

Jane Rachinger authored 9 months ago and Casey committed 5 months ago

Co-authored-by: Stefan Hansson <newbyte@postmarketos.org>

Closes postmarketOS/pmbootstrap#2402

Luca Weiss authored 5 months ago and Casey committed 5 months ago

* Make sure all Python modules are documented by adding a bit of shell
  to .ci/docs.sh
* Remove non-existing module references from .rst
* Fix various warnings from sphinx by adjusting Python docstrings
* Add class member docs to ApkindexBlock

Clayton Craft authored 5 months ago and Casey committed 5 months ago

When a `noarch` APKBUILD is changed in local pmaports, `pmb install`
will build it automatically but image creation will fail with the
following error:

```
$ pmbootstrap install
...
[12:00:43] Build is necessary for package 'postmarketos-base-ui': binary package out of date (binary: 28-r6, local pmaports: 29-r0)
[12:00:48] => edge/postmarketos-base-ui: Done!
...
[12:01:02] (rootfs_lenovo-21bx) % apk --no-progress --repository /home/clayton/.local/var/pmbootstrap/packages/edge add -u --virtual .pmbootstrap /mnt/pmbootstrap/packages/edge/aarch64/postmarketos-base-ui-wifi-iwd-29-r0.apk /mnt/pmbootstrap/packages/edge/aarch64/postmarketos-base-ui-qt-wayland-29-r0.apk /mnt/pmbootstrap/packages/edge/aarch64/postmarketos-base-ui-qt-tweaks-29-r0.apk /mnt/pmbootstrap/packages/edge/aarch64/postmarketos-base-ui-29-r0.apk /mnt/pmbootstrap/packages/edge/aarch64/postmarketos-base-ui-audio-none-29-r0.apk /mnt/pmbootstrap/packages/edge/aarch64/postmarketos-base-ui-networkmanager-29-r0.apk /mnt/pmbootstrap/packages/edge/aarch64/postmarketos-base-ui-wifi-iwd-29-r0.apk --no-interactive
...
[12:01:04] ERROR: Invalid architecture: 'noarch', expected something like: x86_64, aarch64, riscv64, armhf, x86, armv7
[12:01:04] See also: <https://postmarketos.org/troubleshooting>
[12:01:04] Traceback (most recent call last):
  File "/home/clayton/src/pmbootstrap/pmb/core/arch.py", line 42, in from_str
    return Arch(arch)
           ^^^^^^^^^^
  File "/usr/lib/python3.12/enum.py", line 757, in __call__
    return cls.__new__(cls, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/enum.py", line 1171, in __new__
    raise ve_exc
ValueError: 'noarch' is not a valid Arch

```

[caleb: fixed tests]

Luca Weiss authored 5 months ago and Casey committed 5 months ago

Make sure to annotate the parameters.

Luca Weiss authored 5 months ago and Casey committed 5 months ago

In the code where the object is made (pmb/parse/apkindex.py) depends
already will become an empty array when there's no dependencies.

And there's no point in having a differentiation between None and empty
array.

Try to finish the migration by changing various links and texts to point
to the new GitLab instance.

Clayton Craft authored 5 months ago and Newbyte committed 5 months ago

pmb.parse.apkindex.providers now returns a dict of string:ApkindexBlock,
probably from 71772b9b

This adds type hinting to package_provider to reflect this, and fixes
code that calls it to properly deal with the possible return types.
Without this, building FDE images is broken.

Use the new "Artifacts" directive for .ci/docs.sh and run the job in the
pmbootstrap chroot to avoid needing to install sphinx dependencies on
the host.

Fixes #2352

And make the docs job run in the chroot now and grab the 'public/'
directory so we have the built docs.

Since the return value is a file path, let's make it a Path object. Also
update the docstring to reflect reality.

This series breaks pmbootstrap, and that's not good[1]. A bug was filed
about this (#2465), and there have been multiple attempts to fix it (!
2435 and !2436). It kinda seems like we don't have time to fix/test this
for a while longer, which is fine, but given the impact this bug has I
think we should be revert this series until this issue is solved/tested.

1. pmb is broken in some specific cases, which means some workflows are
broken. One example is that this breaks the pmaports CI, so no work can
be done in pmaports.

Looks like an unintended leftover from debugging.

I tested $ pmbootstrap shutdown with this and observed no problems. All
the mounts disappear in lsblk as expected.

Closes #2423

And fix type errors.

All typing errors found with --check-untyped-defs have been fixed, so
let's make mypy stricter now to avoid regressing this.

I'm not sure why this was annotated as a Sequence rather than list. The
function clearly returns a list, not a Sequence, and the commit that
originally did it[1] provides no justification. This causes problems
with mypy checks in other places, so let's just annotate this as
returning a list.

 [1]: 198f302a

Otherwise mypy cannot figure out the type of these.

Newbyte authored 6 months ago and Newbyte committed 6 months ago

Alpine stable's mypy is currently outdated and has bugs in its
handling of overloads. As such, just use mypy from pip so we don't have
to worry about working aroud bugs in old mypy versions.

[ci:skip-build]: already built successfully in CI

Newbyte authored 6 months ago and Newbyte committed 6 months ago

Works around circular import.

Newbyte authored 6 months ago and Newbyte committed 6 months ago

And adjust other code.

Closes #2455