Project direction 2020: base on Alpine stable, get serious about supported devices

@PureTryOut: you've created this milestone about basing on Alpine stable: https://gitlab.com/groups/postmarketOS/-/milestones/2

With the TODOs above, the pmdevices.git task is actually several tasks, and there's a new bunch of tasks for "supported devices".

Would you like to change the milestone to "project direction 2020", so we can link all issues related to this issue to that milestone?

By Oliver Smith on 2019-12-10T10:19:01

Done!

By Bart Ribbers on 2019-12-10T10:19:01

changed the description

By Oliver Smith on 2019-12-13T18:22:13

changed the description

By Oliver Smith on 2019-12-13T18:35:53

changed the description

By Oliver Smith on 2019-12-21T20:24:11

changed the description

By Oliver Smith on 2019-12-23T19:42:28

marked the checklist item pmbootstrap: add pmbootstrap pull as shorthand for git pull in pmaports.git clone, if it is on master branch (pmbootstrap#1858) as completed

By Oliver Smith on 2020-01-26T18:40:33

mentioned in issue #16 (closed)

By Minecrell on 2020-02-07T19:49:56

changed the description

By Oliver Smith on 2020-02-07T20:17:04

@ollieparanoid I've been thinking a bit further after our recent discussion about how to handle the linux-postmarketos- kernels after splitting devices into a separate repository.

I think a short summary for that discussion was:

If we keep linux-postmarketos-* in pmaports we can have stable versions of the kernels.
On the other hand, this makes it impossible to make changes to device packages that depend on certain kernel changes.
So it might be better to keep the kernels in pmdevices, and offer a stable kernel through the pmbootstrap kernel selection instead.

Thinking this a bit further I wonder if the "making changes to device packages that depend on certain [kernel] changes" situation is a general disadvantage of using the same device packages for all stable branches.

Eventually we will have the situation that certain changes made to a device package will depend on changes made in the edge/master channel of Alpine/pmaports. The deviceinfo_min_postmarketos_channel property exists for that purpose, but increasing that property will completely disable the device for older stable branches (if I understand it correctly).

A few examples for these situations:

Upstream changes: Sometimes upstream changes require changes in device packages.
- With Alpine 3.11 the linux-vanilla kernel was moved to linux-lts and we had to change our device packages accordingly (see https://gitlab.com/postmarketOS/pmaports/-/merge_requests/814).
- The recent (minor...) alsa-lib 1.2.2 release broke the UCM files in soc-qcom-msm8916. The (inexistent) UCM specification seems to change quite a bit lately. The fixed UCM files do not work properly on the older alsa-lib version used on the stable branch. :/
In these situations, we can choose if we want to have the old stable release or edge/new stable working.
I think one of them will end up being broken. :(
Refactoring/cleanup: The nature of these changes is that they tend to cause many conflicts if they are not merged in time. However, larger refactoring/cleanup is not necessarily something I would want to push immediately to stable users. There is always the chance that something breaks that could be caught earlier by people testing edge/master.
- directfb can now automatically detect touchscreens, so the deviceinfo_dev_touchscreen property is no longer needed for most devices. We should remove it where possible! This works only on edge at the moment.
We could delay merge of related MRs for up to 7 months (next stable Alpine release plus one month support for older release), but uh... no 🤯
New features: There might be new features (e.g. an experimental GPS driver) that depends on new upstream Alpine packages. We could backport them but ideally they would only appear on stable devices after they were tested for a bit.

These problems don't seem too critical on short term, but I think on long term (when we want to provide a truly stable release) we should avoid them.

I think the only way to avoid them (without constantly worrying about breaking stable users) is to have separate branches for device packages as well. This requires additional work to (cherry-)pick relevant changes to the stable branch, but personally I don't think it is too much work. Supported devices are supposed to have a maintainer (see #16 (closed)), so this work could be distributed quite well.

Since these are more long-term problems I could imagine that we start with having the device packages shared across all stable branches (as suggested in your plan), and eventually switch to using separate branches later.

On the other hand, looking at your list above there seems to be quite some work involved to implement the functionality required to have device packages independently versioned. At this point I wonder if this is worth the effort.

What do you think? :)

By Minecrell on 2020-02-24T18:09:53

Eventually we will have the situation that certain changes made to a device package will depend on changes made in the edge/master channel of Alpine/pmaports. The deviceinfo_min_postmarketos_channel property exists for that purpose, but increasing that property will completely disable the device for older stable branches (if I understand it correctly).

Right, increasing that property will disable the device for older stable branches. We certainly can't afford to support old releases indefinitely, so for starters we have one - and in the transition phases between releases at most two - supported stable releases and cut support afterwards.

I had not written this down yet, but another mechanism I had in mind, is using install_if: we could create postmarketos-base-edge, postmarketos-base-3.11 subpackages indicating the channel, install those automatically with postmarketos-base on the appropriate branch, and then have install_if subpackages in the devices for code that must be different depending on the channel. Another tricky part would be, if the deviceinfo file really had to be different in edge/stable (and this isn't unlikely). In that case, we could teach pmbootstrap to use deviceinfo-stable and deviceinfo-edge files. I'll go through your examples one by one and think through if the above solution would catch them.

But first: do these necessary hacks justify the pmdevices repository, or should we just have branches for devices too? Honestly, I'm not so sure anymore. At least, the changes we have made so far for "pmbootstrap pull" and "pmbootstrap status" are useful, even if we don't end up adding a pmdevices repository.

With Alpine 3.11 the linux-vanilla kernel was moved to linux-lts and we had to change our device packages accordingly

Would work with install_if

The recent (minor...) alsa-lib 1.2.2 release broke the UCM files in soc-qcom-msm8916. The (inexistent) UCM specification seems to change quite a bit lately.

Would work with install_if

deviceinfo_dev_touchscreen property is no longer needed for most devices. We should remove it where possible! This works only on edge at the moment.

Would work with install_if ~~after also teaching pmbootstrap about deviceinfo-stable and deviceinfo-edge~~ (that's not necessary in this example, would only be needed for stuff that pmbootstrap parses and uses)

We could delay merge of related MRs for up to 7 months (next stable Alpine release plus one month support for older release), but uh... no 🤯

Well, regardless of how we do it (pmdevices repo or not), we would need to make an issue as reminder to the change for the stable branch or part of the packages in 7 months. Creating a MR so early would not be helpful. The point of the stable repository is, that not much changes. So this kind of change doesn't justify a backport in my opinion, just adjust our device packages when the time comes.

There might be new features (e.g. an experimental GPS driver) that depends on new upstream Alpine packages. We could backport them but ideally they would only appear on stable devices after they were tested for a bit.

Would work with install_if.

I think the only way to avoid them (without constantly worrying about breaking stable users) is to have separate branches for device packages as well. This requires additional work to (cherry-)pick relevant changes to the stable branch, but personally I don't think it is too much work. Supported devices are supposed to have a maintainer (see #16 (closed)), so this work could be distributed quite well.

Yeah, with your proposal in #16 (closed), it does suddenly seem more feasible to cherrypick necessary changes. Personally, I will probably take some time to get used to the new workflow, but then again, it is much cleaner if we handle the devices just like all the other packages.

On the other hand, looking at your list above there seems to be quite some work involved to implement the functionality required to have device packages independently versioned. At this point I wonder if this is worth the effort.

Luckily you brought this up just before work went into the pmdevices split... I'm definitively against a "split into pmdevices now, then merge pmdevices and pmaports again later" approach.

So... right now I'm leaning towards not using the pmdevices repository at all. Then we don't need to build so much extra code paths, and it would just be consistent with all the other packages we have. The more granulated categorization of device support with the additional maintainers it yields is really making the difference for me, without that we would only have a 1-3 really supported devices that would work well on edge and stable channels. But if we have the community supported devices, as you said, we have maintainers who can also make sure that it runs on stable.

As this is a really important decision, I will give it some more thought and think some more about how we would transition to get there and everything.

Thanks a lot for the input, highly appreciated that you think this through and write down your conclusions here, @Minecrell!

EDIT: of course a nice benefit of not splitting to pmdevices.git is, that we can cross out a lot of the TODOs from above and get towards stable faster :>

By Oliver Smith on 2020-02-24T18:13:59

It looks like just dealing with duplicated device packages makes it a lot easier/clearer than implementing the pmdevices repository.

If the devices are just in the branches for the pmaports repository then we might fall a bit behind on backporting things to stable but we don't have to take any extra care with keeping the device packages compatible. There's enough variations already with the device with mainline/downstream and the hacks that are added to make that work.

I also think that the majority of the devices don't have enough development on them to warrant backporting fixes anyway.

By Martijn Braam on 2020-02-24T18:23:20

Thanks for the suggestion with install_if! I think one of the primary differences of that approach is that it only allows to solve these situations, but it does not directly prevent them.

Without extensive CI checks the "worry" about eventually breaking stable users remains. Having separate branches allows to focus on improving the packages for the next release, without having to worry about the current stable users.

(Also: When a branch becomes unsupported we just throw it away. There is no cruft to clean up.)

So... right now I'm leaning towards not using the pmdevices repository at all. Then we don't need to build so much extra code paths, and it would just be consistent with all the other packages we have. The more granulated categorization of device support with the additional maintainers it yields is really making the difference for me, without that we would only have a 1-3 really supported devices that would work well on edge and stable channels. But if we have the community supported devices, as you said, we have maintainers who can also make sure that it runs on stable.

(Splitting pmdevices into an extra repository might still be desirable just to keep normal packages and device packages neatly separated. But it has a much lower priority. At the end the question is if splitting would just complicate changes that require changes in both normal and device packages...)

EDIT: of course a nice benefit of not splitting to pmdevices.git is, that we can cross out a lot of the TODOs from above and get towards stable faster :>

Less work for an approach that is likely going to cause us less worry later. :)

By Minecrell on 2020-02-24T18:40:25

Also from my side a comment: reading through it I don't know if the advantages of having a pmdevices repo have more impact than the disadvantages. I'd say that we should try an approach first that doesn't involve splitting pmaports in two - we can always change our approach later if we notice significant issues!

By Luca Weiss on 2020-02-25T19:48:14

So I've given this some more thought and I'm convinced now, that the best path forward is without pmdevices.git. My current "pmbootstrap status" related patches are finished once pmbootstrap!1882 is merged. I'll soon refactor the TODOs above to throw out the pmdevices.git related stuff, and go straight to making postmarketOS stable without that intermediate step.

By Oliver Smith on 2020-02-29T20:30:45

changed the description

By Minecrell on 2020-02-25T18:26:59

Hello World. While preparing an issue about ecology for my letter (https://opensourceweekly.org) I discovered your project and fallen in love.

I just finished to install pmOS on my Nexus 7, it was a really painful experience due to https://gitlab.com/postmarketOS/pmaports/issues/466, but after ~10 pmbootstrap install and some visits on XDA forum I finally succeeded.

Here is my feedback from a newcomer eye:

This project seems lacking proper project management
Hard to know which device works fine (#16 (closed) should be a major focus)
While working on my tablet, the UI (phosh) is really slow
Project seems lacking of focus compared to available resources.

Here are my idea to solve this:

Ask for Gitlab Gold for open source: https://about.gitlab.com/blog/2018/06/05/gitlab-ultimate-and-gold-free-for-education-and-open-source/
Start using epics or milestone.
find a good workflow (like https://www.buggycoder.com/project-management-with-gitlab/)
Focus on only 1 or 2 user interfaces (forget Desktop UIs, while cools, they will never be usable). I personally prefer phosh.

For example, we could create an organization epic: Roadmap then create sub epics

Librem 5, PinePhone, Nexus 7... one for each main devices
also create a subepic pmbootstrap which encompass all the major issues to have pmbootstrap working
also create a subepic for UIs like phosh which encompass all the issue to have phosh working.
and create a subepic apps which encompass all the tasks to have a proper and user friendly app distribution mechanism.

~~Finally, what I would love to see on the wiki's main page:~~ Edit: bad idea

Best regards, Sylvain

By Ghost User on 2020-03-06T13:52:18

Hard to know which device works fine (#16 (closed) should be a major focus)

Already being worked on.

While working on my tablet, the UI (phosh) is really slow

Not really a postmarketOS specific issue. Either the device has a not yet performant enough GPU driver or (probably both really) the upstream project needs performance optimizations.

Ask for Gitlab Gold for open source: https://about.gitlab.com/blog/2018/06/05/gitlab-ultimate-and-gold-free-for-education-and-open-source/

I'm against this. We should not use more than the open-source version of GitLab would offer us if we self-hosted it. We're a FOSS project with a focus on keeping everything we use FOSS, and we need to keep the possibility of self-hosting stuff like GitLab ourselves. We would never run proprietary code on our infrastructure.

Main devices (officially focused devices, #16 (closed))

I'm not sure we have to mention that explicitely on the wiki. We're already going to improve the visibility of our "main" devices by updating the site with them. Anyways, such things should be mentioned in #16 (closed), not here.

By Bart Ribbers on 2020-03-06T13:36:38

I'm against this. We should not use more than the open-source version of GitLab would offer us if we self-hosted it. We're a FOSS project with a focus on keeping everything we use FOSS, and we need to keep the possibility of self-hosting stuff like GitLab ourselves. We would never run proprietary code on our infrastructure.

While I understand the philosophy, good project management is really the difference between coding endless hours for 10 hackers, and having an impact for hundred of thousands people.

As I just found the project, I had to search in several repositories to find relevant issues.

Also the lack of a roadmap is really confusing as we can't know on which issue to focus without reading each and every issue, and what is missing to have the device working...

By Ghost User on 2020-03-06T14:21:11

I think roadmaps are part of the regular Gitlab offer. I agree we should use them more. Unfortunately, while I can (that is, have the permission to) create issues, I couldn't create milestones. I think up to now, we have been using (meta)issues for that. See for example #3 (closed). I think the issue has got more interesting content than the milestone https://gitlab.com/groups/postmarketOS/-/milestones/1.

By Antoine Fontaine on 2020-03-06T18:14:21

I think meta issue is inferior to milestones because this is not the level of granularity for which GitLab is designed.

On the other hand, We can effectively replace epics by milestones and using this page: https://gitlab.com/groups/postmarketOS/-/milestones as Roadmap. It allows to stay on the FOSS version of GitLab while drastically improving the project management visibility.

If you permit, I would love to Add a Roadmap link on the Wiki's homepage, pointing to the page above.

By Ghost User on 2020-03-06T18:24:02

I think it's a good idea, but I'm not the one deciding that. I think one problem is that you can't discuss on a milestone and people can't create their own, or add issues to existing milestones. For example, I'd like to create a milestone for Anbox support. This would make them not well maintained as people can't help to do that, while they can add add information as comment on an issue, or create a new one.

By Antoine Fontaine on 2020-03-06T18:43:53

maybe it would be possible to give permissions to create milestones as user? On the other hand, it may create milestones such as this one, that looks much less important than, say, pinephone support or making pmOS stable.

By Antoine Fontaine on 2020-03-06T18:46:14

Hmm I see.

I think only 'project managers' (core team members?) should be able to create / manage milestone. They are those with the more visibility on the projects and thus the more capable of deciding what should be prioritised. The rest (us?) then just have to pick an issue in an open milestone and work on it. If we think that some priority should be changed, or add an issue to a milestone, we ping a 'project manager'.

Good point regarding this milestone, maybe it should never have been a milestone in the first place?

Milestones should be groups of issues (see here for details https://www.buggycoder.com/project-management-with-gitlab/), indicating a clear step in the progress of the project.

For example, I think it's damaging to have both https://gitlab.com/groups/postmarketOS/-/milestones/1 and #3 (closed), it should be only a milestone (either requirements for proper PinePhone support on device launch or PinePhone as first daily driver) and then create issues to really track the progress.

By Ghost User on 2020-03-06T20:03:43

This project seems lacking proper project management

I think only 'project managers' (core team members?) should be able to create / manage milestone. They are those with the more visibility on the projects and thus the more capable of deciding what should be prioritised. The rest (us?) then just have to pick an issue in an open milestone and work on it. If we think that some priority should be changed, or add an issue to a milestone, we ping a 'project manager'.

As someone who has dealt with project managers professionally, both as a software engineer and as an actual project manager for a revenue product at a large company, not having project managers is a feature I'm quite OK with

I would be very, very careful with how this is implemented. Nothing scares away volunteer contributors than the arbitrary pressure/priorities that project managers place on tasks. One might be less inclined to finish a task if some project manager is bugging them constantly about it.

By clayton craft on 2020-03-07T02:54:28

@craftyguy I agree with you.

What I envisionned was more: 'project managers' create tasks and then everyone is free to pick any issue which need to be fixed.

Because I would have more visibility on what is needed, I would be more inclined to pick the 'right' issue by myself. Project manager would not be here to pressure people but to guide them. They won't give 'orders' to people, they will setup roadmap, and answer to contributors having questions.

Maybe it's not the best solution, but as a newcomer, here is my most important problems: lacking visibility on the path to success. If I can't visualize the path to success (having a device working?) how can I be motivated to work on an issue?

and lacking the visibility on what others are working on.

And of course, 'project managers' can also be code contributors, they just are those reponsible for the direction of the project and having the more visibility on what need to be done.

By Ghost User on 2020-03-07T07:46:20

Project direction 2020: base on Alpine stable, get serious about supported devices

NOTE: Current progress is tracked in the milestones.

Introduction

Where we are now

Road to daily driver in 2020

Base on Alpine's stable releases

Devices in separate repository

Get serious about supported devices

pmbootstrap changes

pmbootstrap init

pmbootstrap pull

pmbootstrap status

TODOs

pmdevices.git related

Stable branch in pmaports.git (depends on pmdevices.git)

Supported device selection (could be worked on in parallel to both above)

Streamlining

Child items ...

Activity