Part-of: !2584

Clayton Craft authored 6 days ago and Casey committed 6 days ago

This fixes an issue where pmb lint tries to lint an apkbuild using a repo path that the apkbuild doesn't exist in.
For example, it fixes this:

```
[22:13:26] (native) linting postmarketos-ui-cosmic with apkbuild-lint
[22:13:26] *** apkbuild-lint output ***
Error: File APKBUILD does not exist
[22:13:26] *** apkbuild-lint output ***
[22:13:26] ERROR: Linter failed!
Traceback (most recent call last):
  File "/builds/postmarketOS/pmaports/.ci/lib/apkbuild_linting.py", line 26, in <module>
    common.run_pmbootstrap(["-q", "lint"] + packages)
  File "/builds/postmarketOS/pmaports/.ci/lib/common.py", line 52, in run_pmbootstrap
    subprocess.run(cmd, universal_newlines=True, check=True)
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pmbootstrap', '--aports', '/builds/postmarketOS/pmaports', '-q', 'lint', 'postmarketos-ui-cosmic']' returned non-zero exit status 2.
```

Part-of: !2584

Jens Reidel authored 1 month ago and Oliver Smith committed 1 week ago

Signed-off-by: Jens Reidel <adrian@travitia.xyz>
Part-of: !2559

Print to stdout only if config option has been really changed.
If the new setting is equal to old one, be silent.

Signed-off-by: Alexey Minnekhanov <alexeymin@postmarketos.org>
Part-of: !2539

Add a new environment variable that disables apk's caching feature for
space constrained environments such as the bpo image build jobs.

Currently we have a workaround in place in bpo: the apk cache is moved
to a tmpfs. But this is fragile and just disabling the apk cache is a
more elegant solution.

I've decided to make this an env var instead of a full pmbootstrap
option since this option is not relevant for normal users, only for CI
jobs in space constrained environments. Also we already have another
`PMB_APK_` env var.

Related: bpo issue 136
Part-of: !2553

Casey authored 2 months ago and Oliver Smith committed 1 month ago

The .supported() method errs on the side of exposing too many
architectures to make porting for new or less-common platforms easier.

It therefore isn't suitable for using as a list of supported
architectures we can probe the binary repository for.

Introduce a .supported_binary() method to export only the architectures
where we have a binary repository.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Anri Dellal authored 1 month ago and Oliver Smith committed 1 month ago

According to Python documentation, p.match checks
if characters at the beginning of string match the regex

This commit changes validation to full length of string

Casey authored 2 months ago and Oliver Smith committed 2 months ago

Add a flag to bind-mount in the necessary sysfs directories for USB
devices to be accessed inside the chroot. This is the same as how we do
it for "pmbootstrap flasher" but allows running arbitrary commands in
the chroot.

This is useful to allow using pmbootstrap chroot's as a sandbox for
custom flashing procedures.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

When installing pmboostrap, all tests are installed alongside it. There
doesn't seem to be any way to selectively exclude some files inside
python modules, so move all test into a separate module instead.

This is the typical convention in python projects.

See: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/79862
Fixes: postmarketOS/pmbootstrap#2541

Newbyte authored 3 months ago and Oliver Smith committed 2 months ago

See https://docs.astral.sh/ruff/rules/invalid-class-name

Newbyte authored 3 months ago and Oliver Smith committed 2 months ago

See https://docs.astral.sh/ruff/rules/unnecessary-placeholder

Will be used in the next commit.

Newbyte authored 3 months ago and Oliver Smith committed 3 months ago

And remove hack needed to accomodate this wrong type.

Looks like there was a missing comma when this was added initially.
Caught this while reviewing the formatting patch.

Newbyte authored 3 months ago and Oliver Smith committed 3 months ago

See https://astral.sh/blog/ruff-v0.9.0

Newbyte authored 3 months ago and Oliver Smith committed 3 months ago

See https://docs.astral.sh/ruff/rules/unnecessary-regular-expression

Newbyte authored 3 months ago and Oliver Smith committed 3 months ago

See https://docs.astral.sh/ruff/rules/collection-literal-concatenation

This is also slightly faster according to a microbenchmark that
shows it taking around 19% less time to run:
https://github.com/astral-sh/ruff/pull/1957#issue-1538092351

Newbyte authored 3 months ago and Oliver Smith committed 3 months ago

This shouldn't change at runtime, so avoid any potential issues with it
being a mutable class variable by making it immutable.

Newbyte authored 3 months ago and Oliver Smith committed 3 months ago

Integers automatically get converted to strings when used in f-strings
like this, so just get rid of this explicit conversion.

For some reason, gnome-shell-mobile depends on itself with current
resolving code. Ensure this does not happen.

Fix that repo_missing would pick the wrong package information if a
package is in both the normal repository and the split repository.

This would lead to having the wrong version and depends:

        "pkgname": "gnome-shell-mobile",
        "repo": null,  # not in the systemd repo
        "version": "99946.1-r1",
        "depends": [

Instead of the correct version:

        "pkgname": "gnome-shell-mobile",
        "repo": null,  # not in the systemd repo
        "version": "46.1-r1",
        "depends": [

Fix this by calling pmb.parse.apkbuild() directly with the APKBUILD path
when iterating over the APKBUILDs, instead of pmb.helpers.package.get().

Fix that abuild gets added twice in packages that exist in both the main
and split systemd repository:

        "pkgname": "gnome-settings-daemon-mobile",
        "repo": null,
        "version": "99946.0-r0",
        "depends": [
            "abuild",
            "abuild",
            "alsa-lib-dev",
            "colord-dev",

This fixes the following error in bpo when it tries to use the output of
"pmbootstrap repo_missing":

  UNIQUE constraint failed: package_dependency.package_id, package_dependency.dependency_id

Related: https://postmarketos.org/edge/2025/01/09/systemd-soon/

In the systemd repository, we currently have a forked version of
abuild, which needs to be used to build all other packages. Check if we
have a forked abuild, and if it is the case, add it to the dependencies
of all other packages.

Closes: issue 2401

The "pmbootstrap repo_missing" action is used exclusively by bpo. It
calls it only with these arguments:

  pmbootstrap repo_missing --built --arch "$ARCH"

A blocker for merging systemd into pmaports master is, that the
current repo_missing code cannot display packages that are both in
extra-repos/systemd and in another path. I have considered just not
supporting this and discussed doing that with Caleb and Clayton, but we
figured it would be a major obstacle in the future to not be able to
easily override packages with systemd specific versions (currently we
need this for 3 packages).

Integrating this into the existing repo_missing code would be hacks upon
hacks. Also the scope of the current repo_missing code has many extra
features that are not used and would be extra effort to carry along:
* Allow specifying a pkgname
* Running without --built
* --overview

So I decided to replace the repo_missing code with a much simpler, more
modern implementation, that does exactly what is needed:
* Duplicate packages in systemd and non-systemd dirs are displayed
* The output always include all packages, no matter if they are already
  built or not (same behavior as with --built)
* Removed --overview and selecting specific packages too
* The code for filling "repo" (either "systemd" or None) is more
  resilient now, as it can use proper relative paths to the root of
  pmaports. Unlike the previous implementation, it will not fail if
  subdirs are added to the systemd dir.

I have made sure that the output is exactly the same as before on
current pmaports master.

Related: bpo issue 144
Related: bpo issue 140

Newbyte authored 3 months ago and Oliver Smith committed 3 months ago

I tried my best to get the right author based on git commit history, but
in the case of code being moved from one file to make another I might
have gotten something wrong.

And fix some consequential type errors.

[ci:skip-build]: already built successfully in CI

Robert Eckelmann authored 4 months ago and Oliver Smith committed 4 months ago

this fixes the typos found in #2524

Signed-off-by: Robert Eckelmann <longnoserob@postmarketos.org>

Otherwise this will crash as you cannot directly concatenate str with
Arch, but f-strings don't have this problem.

This argument doesn't exist. Presumably this was copied from
filter_arch_packages() without adapting the comment.

The parameters of this function were needlessly complicated. While I
understand the intent of trying to replicate the API of retrieve()
without duplicating the parameter list, in practice this feels like
reducing duplication to a fault to me. This is due to that the third
parameter, <allow_404>, doesn't make sense in the context of
retrieve_json() as it would raise an exception[1] if retrieve() returns
None (which is the effect of <allow_404> when a 404 occurs), thus merely
changing the exception that gets raised to a less descriptive one
instead of actually behaving like retrieve() would with that parameter
set to True.

With <allow_404> eliminated, there's just two parameters left, and I
don't think duplicating those is a big deal. So make the function easier
to read and annotate by getting rid of the args/kwargs logic.

 [1]: json.loads() raises an exception if its first argument is None

Make codespell CI check pass again. This is trivial, so directly pushing
to master.

Anri Dellal authored 4 months ago and Newbyte committed 4 months ago

Add module that provides utilities and information related to localization.

Initial implementation includes code to generate keyboard configs
for locale set by user.

Do not abort if one or more APKINDEX files cannot be downloaded, if
PMB_APK_FORCE_MISSING_REPOSITORIES is set. This is needed when
bootstrapping new stable branches. The same environment variable is
already used in pmbootstrap code to pass --force-missing-repositories to
apk in pmb.chroot.apk.install_run_apk().

Previously, it would only fall back to parsing all APKBUILDs if a main
package was guessed for this potential subpackage. We do however want to
scan all APKBUILDs in case the package is provided by one of the
packages in pmaports.

Signed-off-by: Jens Reidel <adrian@travitia.xyz>

g++ is a subpackage of gcc, but cannot be detected as such easily by
pmbootstrap. This is because g++ is not gcc-g++ or any other variant of
such package names. Similarly, the detection for musl-dev-ppc64le and
other architectures is broken, since the -dev suffix detection does not
work if there is an architecture suffix.

To fix it, add special case handling for cross toolchain and packages
and have that fixup the -dev cases and hardcode g++ as a subpackage of
gcc.

To reproduce:
- Generate ppc64le cross packages:
  pmbootstrap aportgen gcc-ppc64le musl-ppc64le
- Build a package (to trigger building cross compilers):
  pmbootstrap build hello-world --arch=ppc64le

Signed-off-by: Jens Reidel <adrian@travitia.xyz>

Iuri Jikidze authored 5 months ago and Newbyte committed 5 months ago

Fixes bec2a4b1

[ci:skip-build]: already built successfully in CI