Newbyte authored 2 months ago and Oliver Smith committed 2 months ago

See https://astral.sh/blog/ruff-v0.9.0

The parameters of this function were needlessly complicated. While I
understand the intent of trying to replicate the API of retrieve()
without duplicating the parameter list, in practice this feels like
reducing duplication to a fault to me. This is due to that the third
parameter, <allow_404>, doesn't make sense in the context of
retrieve_json() as it would raise an exception[1] if retrieve() returns
None (which is the effect of <allow_404> when a 404 occurs), thus merely
changing the exception that gets raised to a less descriptive one
instead of actually behaving like retrieve() would with that parameter
set to True.

With <allow_404> eliminated, there's just two parameters left, and I
don't think duplicating those is a big deal. So make the function easier
to read and annotate by getting rid of the args/kwargs logic.

 [1]: json.loads() raises an exception if its first argument is None

Flush the progress bar displayed by the function before logging a
warning message for 404 not found responses from the server. Without
flushing, this message gets lost.

In addition to the warning, display a NOTE and ERROR at the end to
explain what went wrong:

  [20:16:50] Update package index for x86_64 (4 file(s))
  [20:16:51] WARNING: file not found: http://mirror.postmarketos.org/postmarketos_get_404_test/edge/main/x86_64/APKINDEX.tar.gz
  [20:16:51] NOTE: check the [mirrors] section in 'pmbootstrap config'
  [20:16:51] ERROR: getting APKINDEX from binary package mirror failed!

Without handling this properly, pmbootstrap would fail slightly later
with a much less useful error:

  [15:14:28] ERROR: expected str, bytes or os.PathLike object, not NoneType

Fixes: pmbootstrap issue 2424

Hugo Barrera authored 9 months ago and Oliver Smith committed 9 months ago

See: https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2324
Closes: https://gitlab.com/postmarketOS/pmbootstrap/-/merge_requests/2278

Caleb Connolly authored 9 months ago and Oliver Smith committed 9 months ago

Get rid of unused imports and such

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 9 months ago and Oliver Smith committed 9 months ago

This makes testing a lot more annoying.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 10 months ago and Oliver Smith committed 9 months ago

Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.

This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 10 months ago and Oliver Smith committed 9 months ago

Now we can run commands without needs args available!

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 1 year ago and Oliver Smith committed 9 months ago

With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 1 year ago and Oliver Smith committed 9 months ago

We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].

Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.

[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980



Signed-off-by: Caleb Connolly <caleb@postmarketos.org>

Caleb Connolly authored 1 year ago and Oliver Smith committed 9 months ago

Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).

Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:

builddir = chroot / "home/pmos/build"

The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).

The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.

As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.

Anjan Momi authored 4 years ago and Oliver Smith committed 4 years ago

fixes #1837

Oliver Smith authored 5 years ago and Alexey Minnekhanov committed 5 years ago

While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.

This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.

Luca Weiss authored 6 years ago and Oliver Smith committed 5 years ago

Fetch the content of a URL and returns it as string.

Happy new year \o/

Do not fail when an APKINDEX can not be downloaded, print a WARNING
instead. This matches, what apk does. Add a new allow_404 parameter to
pmb.helpers.http.download(), and use it in pmb.helpers.repo.update()
for downloading APKINDEX files. Cache the APKINDEX URLs that gave a 404
for the session, so we do not attempt to download these again.

This is needed for the new binary repository: the initial build is done
without existing APKINDEX files, so we must not fail in that case.

Describe all parameters of pmb.helpers.download().

* add my own build key
* enable the repo in the config
* update the README file
* Adjust testcase, that validates the keys and enable it in testcases_fast.sh
* Only save/load keys to/from the config file, which we ask for during
  'pmbootstrap init', so the binary repo gets used even if a config file
  already exists (this also removes a workaround, that deletes the work
  folder path from the config dictionary before writing it)
* Download missing APKINDEX.tar.gz files with Python code, before
  attempting to build packages (so we know which ones aleady exist in
  the binary packages repository)
* Consider APKINDEX files older than 4 hours as outdated and download
  them again (also in Python code)
* Provide 'pmbootstrap update' to force-update the APKINDEX files
* Travis: more logging output on failure
* Only allow keys from config_keys to be used by "pmbootstrap config"