@kcxt has lead a session with me, @Aelin, @fossdd, @PureTryOut at the hackathon about getting consistent package mirrors. Based on the notes, I'm creating this issue. ## Motivation * Alpine package repos go out of sync with each other (e.g. library bump in main) * Alpine goes out of sync with pmOS * When repos are broken we can't build immutable images (and have problems with building regular images as well, or just upgrading packages, etc.) * Packages forked from aports to pmaports need pkgver bumped (usually > 9000) * Keeping an archive of previous consistent states, which we can use for bisecting problems. * EDIT: see also #94 @kcxt proposed that we create custom Alpine and postmarketOS archive mirrors to solve this, that keeps all history (for a reasonable amount of time) and: * Alpine mirror: only update it when "main" and "community" in Alpine are consistent (for edge it would mirror "testing" too at this point, we don't expect "testing" to be consistent with the rest, good enough) * Remove the Alpine packages of which we have forked versions in pmaports from the mirrored Alpine APKINDEX * (This would affect systemd bootstrapping) * Keep a separate archive repo with all historical versions of packages * Could generate snapshots with version constrained world files for standard pmOS images * Enable "bisecting" the entire OS to find out what caused a regression ## Guesstimates We quickly estimated how expensive / feasible this would be, and concluded that it would be doable with these assumptions from previous experience: * One architecture of edge is about 50 GiB * 1 TiB can probably hold 2 arches of edge packages for 1 year * Stable much less as we have less changes there * 10 €/month per TiB on netcup ## Implementation plan We made a plan at the hackathon, and refined it today in a meeting with @fossdd, @Aelin, @PureTryOut and me, afterwards @kcxt also iterated over it and this is the result: * get a script that checks the consistency of main and testing, based on [Achill's checker script](https://codeberg.org/achill/apkindex-consistency-check/src/branch/main/main.py) * make a systemd timer that runs in the interval that Alpine infra tells us, the unit has a pre-condition to do the consistency check * Monitoring/reporting of breakages (to Matrix) * when this works, we get a server with cheap disk space and deploy it there * independent of this: We should switch to `/usr/lib/apk/repositories.d` * Can we keep the base-url defined in /etc so only the suffix (main/community/testing) are defined in /usr? * Complement consistent mirror with tooling to remove packages that are present in pmOS repos * Monitoring for when forked packages need to be updated * Potential automation through GitLab CI? * Could move relevant pkgver/rel/tag variables to a separate file which the APKBUILD sources and automatically trigger a job to update them, potentially a script per-package which can handle updating local files and other things which are package-specific. @kcxt, @Aelin, @fossdd: feel free to directly edit it if I got something wrong.