Adjust CI scripts so they can be run with 'pmbootstrap ci'

In pmbootstrap master, there is a new action called pmbootstrap ci. It allows running CI scripts of the git repository that the user is currently working in (working directory), if they have a certain format. It already works with pmbootstrap.git. These patches adjust the scripts in pmaports.git so they fit that format.

More information: https://postmarketos.org/pmb-ci

todo

• add distfile-check too, so we really have all of them
  • currently this downloads a script from here
  • maybe download it only once and cache it, or rewrite it as python test and put it into this repo
  • EDIT 2022-11-04: rewrote it as python test, see commit messages. The nice thing is that I extended the sha512sums= parsing function for this and will be able to use this for an extended pkgver/pkgrel test in a follow-up MR.
• make sure all CI scripts still work in gitlab CI by making test changes to relevant files (some checks only run when specific files are modified)
  • Done! Created a test MR in my own gitlab namespace (https://gitlab.com/ollieparanoid/pmaports/-/merge_requests/1) that has all commits from here plus one test commit that changes a device package and a kernel config, everything passes.
• FIXME: the build tests are failing with the following error when running through pmbootstrap ci
  • this was a local error on my end in one VM - not sure what caused it but couldn't reproduce it elsewhere and should not block this MR.

-> Ready to go! Please review.

>>> hello-world: Installing for build: build-base
Waiting for repository lock
ERROR: Unable to lock database: Bad file descriptor
ERROR: Failed to open apk database: Bad file descriptor

demo output

$ pmbootstrap ci
[17:37:11] WARNING: this git repository has uncommitted changes
[17:37:11] Available CI scripts (15):
[17:37:11] * flake8: lint CI related python scripts/tests
[17:37:11] * apkbuild-lint: run apkbuild-lint on modified APKBUILDs
[17:37:11] * wiki: verify devices are documented in the wiki
[17:37:11] * ec: editorconfig-checker: lint for trailing whitespaces etc.
[17:37:11] * pytest: lint with various python tests
[17:37:11] * commits: check pkgver/pkgrel bumps, amount of changed pkgs etc
[17:37:11] * kconfig: check all kernel configs with 'pmbootstrap kconfig check'
[17:37:11] * grep: check various bad patterns with grep
[17:37:11] * shellcheck: lint all shell scripts
[17:37:11] * build-riscv64: build modified packages for this architecture (slow)
[17:37:11] * build-x86: build modified packages for this architecture (slow)
[17:37:11] * build-armhf: build modified packages for this architecture (slow)
[17:37:11] * build-aarch64: build modified packages for this architecture (slow)
[17:37:11] * build-x86_64: build modified packages for this architecture (slow)
[17:37:11] * build-armv7: build modified packages for this architecture (slow)
[17:37:11] Which script? [all]:

Also it's possible to specify scripts on the command-line:

$ pmbootstrap ci wiki ec pytest

assigned to @ollieparanoid

By Oliver Smith on 2022-11-01T16:44:29

mentioned in issue pmbootstrap#2169 (closed)

By Oliver Smith on 2022-11-01T16:45:12

added 45 commits

• 4652b271...4c3be3aa - 35 commits from branch master
• d2719003 - .ci/check_devices_in_wiki: remove <> around url (MR 3608)
• 1d626b7d - CI: wiki: adjust for 'pmbootstrap ci' (MR 3608)
• e61677ff - CI: flake8: adjust for 'pmbootstrap ci' (MR 3608)
• fabcd1d4 - CI: shellcheck: adjust for 'pmbootstrap ci' (MR 3608)
• 4d9f1c61 - CI: editor-config: adjust for 'pmbootstrap ci' (MR 3608)
• 8ff4a445 - CI: adjust .ci/common.py users to pmbootstrap ci (MR 3608)
• f362b61d - CI: kconfig adjust for 'pmbootstrap ci' (MR 3608)
• 24d4bc1e - CI: move_logs.sh: put in .ci/lib (MR 3608)
• 71aae70c - CI: make test_unreferenced_files more generic (MR 3608)
• c941f367 - CI: implement distfile-check as python test (MR 3608)

Compare with previous version

By Oliver Smith on 2022-11-04T06:59:47

marked this merge request as ready

By Oliver Smith on 2022-11-04T07:03:05

marked the checklist item add distfile-check too, so we really have all of them as completed

By Oliver Smith on 2022-11-04T07:03:05

marked the checklist item make sure all CI scripts still work in gitlab CI by making test changes to relevant files (some checks only run when specific files are modified) as completed

By Oliver Smith on 2022-11-04T07:03:05

marked the checklist item FIXME: the build tests are failing with the following error when running through pmbootstrap ci as completed

By Oliver Smith on 2022-11-04T07:03:06

Two quick comments, haven't done a deep analysis but in general looks quite cool!

• Why is build-x86_64.sh the main build file? What happens when we run pmbootstrap ci on e.g. aarch64? Maybe rename it build.sh or something and also symlink x86_64 there?
• Can we have a pmbootstrap ci preset that only runs the fast tests? Also for pmbootstrap itself I found that the pytest one takes quite a long time, compared to the rest. Because then also with even more tests here you need to manually write out all the tests that aren't the slow ones if you want to run the fast ones only.

By Luca Weiss on 2022-11-11T07:29:11

added ci label

added 36 commits

• c941f367...955ac29f - 26 commits from branch master
• b576bab9 - .ci/check_devices_in_wiki: remove <> around url (MR 3608)
• 5e0548f9 - CI: wiki: adjust for 'pmbootstrap ci' (MR 3608)
• 88cf6f4e - CI: flake8: adjust for 'pmbootstrap ci' (MR 3608)
• 2b683842 - CI: shellcheck: adjust for 'pmbootstrap ci' (MR 3608)
• 16965740 - CI: editor-config: adjust for 'pmbootstrap ci' (MR 3608)
• cc958ffe - CI: adjust .ci/common.py users to pmbootstrap ci (MR 3608)
• 4bc04ad9 - CI: kconfig adjust for 'pmbootstrap ci' (MR 3608)
• 286f6e5e - CI: move_logs.sh: put in .ci/lib (MR 3608)
• 01a7f41a - CI: make test_unreferenced_files more generic (MR 3608)
• d12bce13 - CI: implement distfile-check as python test (MR 3608)

Compare with previous version

By Oliver Smith on 2022-11-11T07:28:28

resolved all threads

By Oliver Smith on 2022-11-11T07:29:13

merging as discussed in meeting

By Oliver Smith on 2022-11-17T18:09:33

approved this merge request

By Luca Weiss on 2022-11-17T18:11:27

added 22 commits

• d12bce13...cb79ff57 - 12 commits from branch master
• 23790a35 - .ci/check_devices_in_wiki: remove <> around url (MR 3608)
• a2f75963 - CI: wiki: adjust for 'pmbootstrap ci' (MR 3608)
• 613aa1c6 - CI: flake8: adjust for 'pmbootstrap ci' (MR 3608)
• c88cfa51 - CI: shellcheck: adjust for 'pmbootstrap ci' (MR 3608)
• 8f78988b - CI: editor-config: adjust for 'pmbootstrap ci' (MR 3608)
• da52dace - CI: adjust .ci/common.py users to pmbootstrap ci (MR 3608)
• 196abb15 - CI: kconfig adjust for 'pmbootstrap ci' (MR 3608)
• 6dbe4fdc - CI: move_logs.sh: put in .ci/lib (MR 3608)
• 3c4080e8 - CI: make test_unreferenced_files more generic (MR 3608)
• a1242ed3 - CI: implement distfile-check as python test (MR 3608)

Compare with previous version

By Oliver Smith on 2022-11-17T18:12:11

enabled an automatic merge when the pipeline for a1242ed3 succeeds

By Oliver Smith on 2022-11-17T18:12:29

merged