Split pmbootstrap and aports in two repositories

I'd be interested to start working on this packaging task. Would love to hear your thoughts on how we should handle this. Specifically the biggest pain point is the assumption that aports is under the working directory.

In this future split, where would be the right place to have aports? Within .var/local/pmbootstrap perhaps? Should pmbootstrap handle the loading of aports?

Note that currently in the setup.py proposed in #443 the use of the -p aports flag is required when running the installed executable entry point.

By Yuval Adam on 2017-08-22T20:57:35

Created by: ollieparanoid

Tough question and I'm almost out of postmarketOS time for today :/

I would like pmbootstrap to assist the user in getting the aports repository, so that no manual step is involved. It would need to happen before the first question in init, because that shows the list of devices, which we can only do when the aports folder is available.

The work path seems to be suitable for aports, so we would need to ask for that first.

Here's a rough draft from a user's point of view.

pmbootstrap init
ask for work folder
aports not found (specify with -p)! do you wish to download it to: $work/aports? [y]
- pulls it with git
- display instructions how to update them (make that convenient with a pmbootstrap shortcut?)
which device do you want to port?
...

By * postmarketOS Bot on 2017-08-23T20:57:11

Created by: kskarthik

Label this as Long term priority until we get plasma working on already green ports :) Then we are good to go

By * postmarketOS Bot on 2017-08-25T17:10:31

Created by: ollieparanoid

It has a future label already :) But if @yuvadm or someone else wants to work at this right now, why not? It is a blocker for getting pmbootstrap into pypi #327 (closed) after all and making the development more robust #382 (closed).

By * postmarketOS Bot on 2017-08-25T17:24:18

Created by: marmistrz

How about a very simple workflow: if the aports directory is missing, we just show the git clone ... command to the user.

This looks like a very small change which I could do myself, we'd just need to decide the default path for aports.

This issue actually breaks my packaging for AUR (Arch Linux) :)

By * postmarketOS Bot on 2018-04-30T11:26:24

I guess ~/.cache/pmbootstrap/aports or reuse ~/.local/var/pmbootstrap. I guess making pmbootstrap init fetch the latest aports makes it quite usable.

By Martijn Braam on 2018-04-30T12:59:40

Created by: ollieparanoid

Thanks for working on this @marmistrz! Since moving the aports to a dedicated git repository is a big change, there would be some tasks that need to be done beforehand:

set up the aports repository (for testing, this doesn't need to be done in the postmarketOS organization on GitHub)
- start with the pmbootstrap repository
- delete everything from pmbootstrap, except for the testing code
- adjust Travis CI testing code
  - only run the tests relevant for the aports
  - make sure building changed packages is still working (we will need to clone pmbootstrap for that!)
adjust the pmbootstrap repository (can be done in pull requests)
- rewrite test cases depending on the aports folder to work without that folder (or disable them if they still run in the "aports" repository)
- provide a good upgrade path for users (we are already versioning the work folder structure and have migration code, so we could add a new version and add migration code that requires the user to clone the aports folder)

I'm not sure if that covers everything, but that would be a good start from where we could do the first tests and fix whatever else comes up. So all this is quite some work, and while it would be nice if you could look into this, I don't think we should make such a big change in master at least until the next blog post is out at end of May (because preparing for that eats up a lot of resources already).

By * postmarketOS Bot on 2018-05-01T01:47:49

Created by: PureTryOut

Now we're moving to Gitlab and thus changing our infrastructure a bit, is it maybe a good time to split the two up? It seems like the perfect opportunity to me.

By * postmarketOS Bot on 2018-06-16T11:00:36

Created by: ollieparanoid

The hard part are the code changes (see above), so I would advise against mixing that up with the gitlab migration, because it would make the migration way more complex.

By * postmarketOS Bot on 2018-06-16T21:32:47

mentioned in merge request !1455 (closed)

By Martijn Braam on 2018-06-30T13:12:04

mentioned in issue #1622 (closed)

By Bart Ribbers on 2018-07-25T17:50:42

mentioned in issue #1419 (closed)

By Bart Ribbers on 2018-08-17T09:56:00

Could we maybe push this forward a bit? Having the aports separately allows for a bunch of stuff:

we can base the aports on Alpine releases (3.7, 3.8, edge, etc)
users will be able to easily switch between aport releases
issues with packaging in it's own issue tracker, providing a more clean issue tracker
pmbootstrap can be packaged for distributions and pip

We've been talking about this for quite a while, and I think it's a huge improvement to our current situation, but so far nothing has happened

By Bart Ribbers on 2018-08-17T09:59:43

First of all: my apologies, that we did not really get anywhere with this issue yet, despite that you and others have asked quite a few times over the course of the last months about it. Let's finally give this a big push!

we can base the aports on Alpine releases (3.7, 3.8, edge, etc)

This is what I want to have as well, but keep in mind that this requires big infrastructure changes. Packages need to be compiled and hosted for all supported branches then (which means changes in the pmOS-repo-scripts and folder structure in the binary repo mirror), and we need to copy everything from Alpine's aports/testing, which we are using to our aports/temp, because aports/testing is only built against edge in Alpine.

@MartijnBraam wrote on that note in #1622 (comment 90587200):

Having #383 (closed) fixed first would make it easier.

This sounds reasonable, so let's approach it from that angle first. After thinking about how to implement this in detail (below), I think I can do it within the next week (the separate aports repository without support for multiple branches).

Implementation ideas

pmaports repository

naming it "pmaports" instead of "aports" to avoid conflicts with Alpine's aports
has a pmaports.cfg file with compatibility information:
- pmaports version (pmbootstrap will have a minimum pmaports version stored and check against this value)
- minimum pmbootstrap version (again, for a check in pmbootstrap)
- compatible Alpine aports branch (to be used in the future for generating the appropriate mirror URL in /etc/apk/repositories)
- pmOS aports branch name (also used to generate the right mirror URL in the future - this value can be different from the git branch we're on inside this repository, if we are preparing a merge request in an extra branch for example)
folder structure:

- aports/
  - main/
  - device/
  - temp/
  - ...
- test/
- pmaports.cfg

new "pmbootstrap init" workflow

ask for the work dir as usually (we already do this as first question nowadays, but this was not always the case before)
initialize the work folder and start logging to it, so pmbootstrap log is already working (right now this only works after the first run of pmbootstrap init)
if $work/cache_git/pmaports does not exist:
- explain that sudo will be used to set up the chroot (because this is the first time for new users that it will happen)
- set up the native alpine chroot
- install git inside it
- clone the repository with git (note that we already have code to clone git repositoried with pmbootstrap, we use that to clone Alpine's aports for pmbootstrap aportgen gcc-armhf etc)
if pmbootstrap/aports does not exist:
- note: pmbootstrap/aports will not exist after cloning pmbootstrap, and be added to .gitignore
- try to create a symlink to pmbootstrap/aports, which points to the clone path from above
- that way the workflow doesn't change much for the user, they can find the aports in the same place
- creating the symlink will fail if pmbootstrap was installed system wide with a package manager - in that case, just display the path where the aports have been cloned
now continue as usually by asking for the device etc.

before executing any pmbootstrap action besides init:

check if the aports folder exists, raise an error that asks the user to run pmbootstrap init otherwise
check if the current pmbootstrap version is compatible with the checked out aports folder (by comparing pmaports.cfg with values in pmb/config/__init__.py) and show a helpful error otherwise that explains how to update the outdated component
sanity check: chroots should be zapped after switching branches (at least display a warning)

By Oliver Smith on 2018-08-18T06:20:44

mentioned in merge request !1671 (merged)

By Oliver Smith on 2018-08-19T19:40:30

In case you aren't aware, the pmaports repo can be easily created with all the history intact using git filter-branch:

git clone https://gitlab.postmarketos.org/postmarketos/pmbootstrap.git pmaports
cd pmaports/
git filter-branch --subdirectory-filter aports/

By Grant Miller on 2018-08-19T22:47:42

mentioned in issue #1645 (closed)

By Oliver Smith on 2018-08-20T20:56:27

That was very useful, thanks! WIP new repo is set up here: https://gitlab.com/postmarketOS/pmaports Merge request coming soon.

By Oliver Smith on 2018-08-23T19:46:59

mentioned in merge request !1685 (merged)

By Oliver Smith on 2018-08-23T21:43:22

closed via merge request !1685 (merged)

By Oliver Smith on 2018-09-05T05:57:39

assigned to @ollieparanoid

By Oliver Smith on 2018-09-05T05:57:46

For the record, I've used this script for rebasing (assumes the pmbootstrap/master branch is in the local branch pmbootstrap-master):

#!/bin/sh -e
cd ~/code/pmaports
git checkout master
git checkout pmbootstrap-master
git reset --hard pmbootstrap/master
git filter-branch --subdirectory-filter aports/ -f
git rebase master
git checkout master
git merge pmbootstrap-master

By Oliver Smith on 2018-09-05T14:47:37

mentioned in commit dd4e1b1b

By Oliver Smith on 2018-09-25T16:24:30

mentioned in issue #1906

By Oliver Smith on 2020-04-19T16:25:30

mentioned in issue #1914 (closed)

By Анатолий Михайлов on 2020-04-26T06:36:04

mentioned in commit 0be893d9

By Luca Weiss on 2022-06-24T12:59:34

mentioned in commit 58c39f2c

By Luca Weiss on 2022-06-27T06:16:27

mentioned in commit 46b70829

By Oliver Smith on 2023-04-14T23:53:14

Split pmbootstrap and aports in two repositories

Child items 0

Activity

Implementation ideas

pmaports repository

new "pmbootstrap init" workflow

before executing any pmbootstrap action besides init: