Caleb Connolly authored 4 months ago and Oliver Smith committed 4 months ago

We shouldn't overwrite config.aports if the user specified a custom
--aports directory. We only want to do this to update the default if
they're using a custom workdir.

Make the code here less aggressive, and correctly use the last entry in
config.aports as "pmaports" instead of the first.

Fixes: #2386
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 4 months ago and Oliver Smith committed 4 months ago

There was an oversight when this API was originally created and it
implicitly assumed that the pmaports repository was always named
"pmaports". This unfortunately broke some peoples workflows.

Introduce a new "pkgrepo_name()" function and adjust the codebase to use
it, as well as adjusting pkgrepo internally to special case the
"pmaports" repo so that it's always named pmaports no matter what the
directory itself is named.

This is probably more complexity than we should be dealing with here, we
should probably create a new type to encode this behaviour.

Fixes: #2412
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 4 months ago and Oliver Smith committed 4 months ago

When pmbootstrap starts up, it loads the config file (or uses the
defaults) and then merged some arguments (e.g. --work) into it. However
in config.init.frontend() we were creating yet another new Config, but
it's too late to merge args in at this point...

Rework this slightly to fix that major oversight and remove the totally
unnecessary hack of calling pmb.config.load() again.

This fixes calling pmbootstrap init with --work

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

This was always a hack, and it looks like there (hopefully) aren't any
placs where we still need to handle this.

Possibly expecting regressions... But then we have something to write a
test for.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

When running "apk add" we must always have --cache-dir set! We also
expect --no-interactive to be set.

Add some asserts so we don't regress here.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Since parse.deviceinfo() returns a custom class now, it can't be
trivially serialized as JSON.

This has been broken for months now so it's clear nobody is using this.
Let's take the chance to get rid of it.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Move the apk_static module from pmb/chroot to pmb/helpers which is a
more suitable location.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Since all apk invocations withing pmbootstrap now use apk.static by
default, apk_static.run() isn't called anymore, so let's remove it.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

In 0b4fb911 (chroot: always run apk static v2 (MR 2423)) we adjusted
install_run_apk() to run apk static on the host and pass in the local
binary repo with "--repository". This function can call apk in two ways,
either with the progress bar handling or without, the second case was
never updated and still ran apk inside the chroot incorrectly and with
an incorrect --repository flag.

Let's finish the job by refactoring helpers/apk.py to support all our
usecases and pointing everything to it, removing the last few situations
where we call "pmb.chroot.root(["apk", ...]).

The apk_with_progress() function is replaced by a generic "run()"
function which takes a boolean to indicate if we should render apk
progress.

Additionally, a new cache_clean() function is added so that "pmbootstrap
zap --pkgs-online-mismatch" can FINALLY be refactored to not rely on a
chroot existing. This requires some hacks but nothing serious, see the
comments in the function for details.

The chroot.init() code is now simplified since handling the --root,
--arch, --cache-dir, and --repository flags is now all done by
apk._prepare_cmd() as and when appropriate.

Lastly, this fixes a (previously unnoticed) bug where apk.static was
actually using /var/cache/apk on your host machine for its cache... This
is definitely not good behaviour....

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

This function better belongs here, especially as it will be used outside
of the context of a chroot() soon.

Additionally, it's adjusted to take the rootfs path as its first
argument rather than a chroot, since it could operate on any rootfs.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Lazy load pmb.config.styles and move the pmb.__version__ print elsewhere
so the logging module is (closer to) a standalone entity. This is
necessary to be able to import it in pmb/helpers/apk.py otherwise we get
a cyclical dependency.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

make mypy happy

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Make intellisense happy

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

This should match the previous behaviour and fix crashes when these
properties don't exist.

Tested with a htc-primou bootimg.

Fixes 826bb4f2
[ci:skip-build]: already built successfully in CI

Refactor the integration test script to support running a variety of
tests, and add CI jobs for them.

Additionally, move the integration tests to their own stage so we don't
potentially waste a bunch of CI time running them on broken branches.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

We run pmbootstrap with --details-to-stdout now, so this isn't needed.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Write some basic parser tests for deviceinfo. The parser is quite basic
so there isn't much to test yet. Expecting this to gain features as we
look at e.g. making the parser the "source of truth" for the deviceinfo
format.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

We pass this too, so might as well ensure we don't regress.

We pass this check now.

They should definitely not be None here, thus assert it so we don't need
to keep checking for None later.

It doesn't really make sense to check the blacklist against None, so
just skip the value if it's None. Appeases mypy.

Otherwise, if pmb.parse.deviceinfo().cgpt_kpart is an empty string, for
example, we end up passing an empty string to get_partition_layout()
instead of False.

pmaports_cfg.get() can return other types, too. As such, we need to
assert that it's a list to appease mypy.

Helps mypy understand the code better and also makes it easier to read
as a human in my opinion.

I think the "these types make no sense" comment were about
output_return_buffer, which defaulted to False but was to be set to
list[bytes] if used. I changed it so that it defaults to None, which
probably is more conventional. Other than that I didn't notice anything
weird about the types here.

Also check for None as necessary to appease mypy.

img_path_2nd is a Path, not a str, so we can't just rely on string
concatenation here.

img_path_2nd is a Path as that's what create_second_storage() returns
since 198f302a.

This could be done better with a real class instead of a TypedDict, but
it's better than a regular dict.

Useful for annotating places where a parsed APKBUILD is expected.

This makes the behaviour match the docstring which specifies that this
function returns integers.

This is equivalent to the previous definition, but more concise.

We don't want an actual Arch object in the environment object.

Can't construct a path by using the / operator on strings, so use the
Path constructor instead.

This is no longer used as of 1299c2ef,
so just remove this dead code.

If flavor is None here, something has gone wrong and the line after will
fail. Explicitly assert that it has the right value to appease mypy.

Reusing this variable might lead to confusing bugs later if this code is
changed in the future, and also leads to mypy being confused about the
type of size_mb as it then gets assigned both str and int values.

At some point, this function was updated to use the Arch type instead of
plain strings, but the return type was left omitted and the docstring
outdated.

No parameter called "now" exists for this function, so don't document
it.

This is already the default: https://mypy.readthedocs.io/en/stable/config_file.html#confval-color_output

And remove --check-untyped-defs from mypy.sh script to avoid
duplication.