relocate /etc/deviceinfo to /usr/share, allow user overrides

mentioned in merge request !3349 (merged)

By clayton craft on 2022-12-14T06:03:31

Everything you're saying here makes sense. But it will be pain to find all places and all software that sources deviceinfo from /etc and change it everywhere..

By Alexey Min on 2022-12-14T21:59:30

Maybe temporarily one solution would be to install a copy of the file in both directories? You would still get an /etc/deviceinfo.apk-new file if the user modifies it, but it warranties that nothing will break, and user preferences will be respected over the shipped file.

The other option is to fix known programs, and wait to see what breaks :P

By Pablo Correa Gomez on 2022-12-14T21:59:30

I thought of something else that might work: make /etc/deviceinfo a script that sources from /etc/deviceinfo.d, /usr/share/deviceinfo and so on. so existing programs can continue to source /etc/deviceinfo without changes to them.

mkinitfs would need to pull in the other files, but that's probably(?) just a matter of adding a new file list for it to pull from

By clayton craft on 2022-12-18T18:49:55

If the idea is being more compliant with FHS, having a script under /etc/deviceinfo does not really sound like a great improvement. Although it might solve the user issue you were originally dealing with :)

By Pablo Correa Gomez on 2022-12-19T19:32:00

I would suggest including /etc/deviceinfo.d/* to include user changes.

By Raymond Hackley on 2023-01-03T04:23:59

So we discussed this in detail with @calebccff and @craftyguy. This is just some text so we do not totally forget about the discussion. We evaluated this two options:

Move /etc/deviceinfo to /usr/share/deviceinfo and create a new /etc/deviceinfo which sources /usr/share/deviceinfo and then /etc/deviceinfo.d/* where users could have their config.
Move /etc/deviceinfo to /usr/share/deviceinfo and keep /etc/deviceinfo for users to edit. Potentially, we could ship a default /etc/deviceinfo with instructions on how to modify it and maybe commented out line with all the options.

Option 1

PROS:
- Things sourcing /etc/deviceinfo will "just work".
CONS:
- Things parsing /etc/deviceinfo (postmarketos-mkinitfs at least) would need more work.
- We would instantaneously break mkinitfs for any user with an edited /etc/deviceinfo (since APK will keep their version). This is bad... But could be an option to get everybody that already edited get doing things the way we want :D

Option 2

PROS:
- Doing things this way is a standard. People are really used to the standard of being able to modify whatever in /etc, while people should already know that touching /usr/share will byte you back.
- Users that already edited /etc/deviceinfo will not see any difference in the short-term.
CONS:
- We need to modify everything that now sources /etc/deviceinfo to source first /usr/share and then /etc/deviceinfo. The good thing, is that this should not be much of a hazard. And is common in any program that has system and user configuration.
- User modifications to deviceinfo variables that contain long strings are likely to go wrong. Users would have to copy the whole string from /usr/share, they will most likely not do so and things like kernel_cmdline will most likely get out of sync and we will continue getting issues. This might be solvable, but needs further thought.

By Pablo Correa Gomez on 2023-02-19T15:41:39

With regards to moving deviceinfo to /usr/share, the FHS recommends that you store data in a subdirectory of /usr/share rather than /usr/share itself. /usr/share/misc/deviceinfo could be suitable: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s11.html

By Newbyte on 2023-02-19T15:41:39

True. I have the feeling that misc feels convoluted and nobody else seems to use it, but it makes sense to stick to the standards, so I'll do it if this proposal moves forward :)

By Pablo Correa Gomez on 2023-02-20T22:24:37

mentioned in issue boot-deploy#12 (closed)

By Pablo Correa Gomez on 2023-02-13T18:20:23

mentioned in merge request boot-deploy!24 (closed)

By Pablo Correa Gomez on 2023-02-13T19:39:26

mentioned in merge request boot-deploy!25 (closed)

By Pablo Correa Gomez on 2023-02-13T19:54:50

I created a "milestone" for this, since it'll require coordinating MRs across a number of projects under the gitlab.com/postmarketOS namespace: https://gitlab.com/groups/postmarketOS/-/milestones/17

I think we need to "finalize" on a plan on how to implement this. Here's what I propose, based on a few lengthy discussions with @pabloyoyoista :

1. Packaging changes in pmaports

Packages in pmaports create/own /usr/share/deviceinfo

All device packages can install deviceinfo to this new location and ignore /etc/deviceinfo
- [Added in Rev1] This is actually backwards compatible: Device packages that continue to install to /etc/deviceinfo should still be supported by apps/scripts even if they support reading from /usr/share, since they also have to continue to support reading from /etc/deviceinfo. There is no need to rebuild ALL device packages to make this work.
- [Added in Rev2] devicepkg-dev will make this change a lot easier, but packages have to be rebuilt in order to "get" the change in them
Apk will overwrite any changes to this file and not create an .apk-new, so we can be sure that device packages always have their desired deviceinfo "applied" on the target system.

2. Existing /etc/deviceinfo

[Edited in Rev1] The /etc/deviceinfo file is ignored by packages in pmaports (i.e. none of them install this file). Users can modify this file at-will to overwrite variables set in /usr/share/deviceinfo

[Added in Rev1] As a bonus, if the user has any modifications, then apk will not remove it either once the device package no longer installs it.

3. Changes to parsing deviceinfo

Introduce some new deviceinfo variables, for the purpose of appending values from /etc/deviceinfo, without requiring users to manage long lists of things in the values.

One example is deviceinfo_cmdline, it can be quite long, and if a user copies it from /usr/share/deviceinfo to /etc/deviceinfo, then they risk their local changes becoming "out of sync" with the deviceinfo from the device package. Plus it's annoying having to maintain a large list if you just want to add a new module, cmdline param, whatever
apps/scripts that use variables that have an _append equivalent should properly append values from them to the non-_append version. E.g. cmdline="$deviceinfo_cmdline $deviceinfo_cmdline_append"
device packages should not set deviceinfo_*_append variables in /usr/share/deviceinfo, since users are now encouraged to override it in /etc/deviceinfo
- Maybe some CI can grep for that pattern and fail, so that we don't accept device packages/changes that modify these variables in /usr/share/deviceinfo ?
New variables added to deviceinfo:
- deviceinfo_cmdline_append
- deviceinfo_modules_initfs_append
- others?
Other variables in deviceinfo, e.g. that are bools, ints, strings, whatever, can be overwritten. Appending to these doesn't make sense.

Applications/scripts that use deviceinfo will need to:

source / get values from /usr/share/deviceinfo
source / get values from /etc/deviceinfo, overwriting any re-definitions of vars done here.
for variables that have _append variants, properly append those before using the value (see 3 above)

By clayton craft on 2023-02-15T19:39:10

Revisions

Rev1:

Added clarification to section 1 about packaging changes to deal with the new file location
Edited and existing point and added a new one to section 2 about how the existing /etc/deviceinfo file is treated
Added section titles

Rev2:

Added note about deviceinfo-dev in section 1

By clayton craft on 2023-02-13T23:36:30

i really like this proposal, although the _append variables just don't feel quite right. I've tried thinking of ways to not have them but it always ends up being more complex...

i think the reasoning for that is mostly because we want to be able to append to but also replace the cmdline options.

how about we define a list of variables that should be appended by default, drop it as a bash flavoured config file containing a deviceinfo_append_options=..., and allow users to override it with their own config containing a deviceinfo_never_append=... option.

stuff can either parse this file or alternatively we shove the parsing behaviour somewhere and allow them to just invoke a parse_deviceinfo script to spit it all out.

i think this would be relatively straightforward, not add too much complexity (the only places we need to deal with this are on device where everything could call out to this script).

By Caleb Connolly on 2023-03-05T18:48:37

but also replace the cmdline options

We decided strictly against a specific option for this. On one side, because as you say, it adds complexity. On the other side, because this is something that to our knowledge is not available anywhere else (e.g: grub only has the "append" variable). This means that the need to replace cmdline options is definitely not widespread and at most affects a little amount of users. If these users have a real need to remove some entry in the cmdline, there are two options: replace the cmdline variable in /etc, and make sure it's kept consistent with upstream changes, or bring the change upstream like you have done removing the serial console from sdm845 kernel options.

Is that more convincing?

I have summarized it down here, but we would only be adding _append variable to kernel_cmdline.

By Pablo Correa Gomez on 2023-03-05T21:08:07

added discussion label

@pabloyoyoista I was thinking about the _append var idea, because I still don't really like it...

In the last pmOS team meeting, we all kinda agreed that maybe we should have a "compiler" script that apps can "source $(getdevinfo)" or something to get a generated list of deviceinfo key/value pairs. Any complicated merging or whatever could happen in that "compiler", and it would give us the flexibility change what that complicated logic does without affecting any apps that consume deviceinfo. (e.g. for supporting generic images)

So what I was thinking is that maybe we should just move all variables out of deviceinfo that we don't feel comfortable just simply replacing entirely by sourcing /usr/share then /etc, i.e. the vars we think require _append variants.

For the variables that might be more complicated to merge, there are other options for managing those in a more sane way. For example, I just rediscovered (lol) that in mkinitfs, I (already, lol) easily allow users to add modules by just writing to files in /etc/postmarketos-mkinitfs/modules/ (I just need to add support for iterating through /usr/share too). mkinitfs can, today, successfully merge multiple files with modules listed in them into one set of modules in import into the initramfs. For kernel cmdline options, those are already handled with the recent boot-deploy changes.

Pros

no need for adding _append vars, and dealing with them
Barely increases the complexity of existing apps that source deviceinfo
Greatly simplifies how existing apps would use a deviceinfo setup after relocating it to /usr/share
Yes, the last three points above are all basically saying the same thing

Cons

Might not be backwards compatible, the changes would be a little more involved... i.e. apps using the vars that have been removed, like device_kernel_cmdline, will need to use the "new" config.
- I'm pretty sure the cmdline and modules vars are used mostly by boot-deploy and mkinitfs...
I haven't thought through all the pmbootstrap changes that would be required, and if it's possible to support this
Might not work with generic images (cc @calebccff) ?
The sacred deviceinfo is no longer the single source of truth for system's entire configuration. initfs_modules would live on as /etc/postmarketos-mkintfs/modules and /usr/share/postmarketos-mkinitfs/modules/, the kernel cmdline can live on as the recent cmdline.txt stuff in boot-deploy, and so on.
- But, was it ever really? Some configuration has to generated at runtime, like with the librem 5's uboot script choosing the correct dtb on each boot.

Thoughts?

By clayton craft on 2023-02-20T22:22:13

I have the feeling that this is solution is a bit too convoluted for the current task at hand. To my understanding, the issue this was originally opened for was to move deviceinfo out of /etc, since users tend to modify it, APK treats changes there as something special and this causes issues with updates not getting through.

The previous proposal was to execute exactly that: move deviceinfo to /usr/share, fixing some of the fallback of that migration with specific solutions for specific needs (the _append solution). I sent #1836 (comment 1277043534) to specifically show that the change was possible, reasonable, and limited. This new proposal, however, goes much further by considerably rethinking the goal of the deviceinfo, and I am not totally sure that is the right approach to solve the specific problem this issue was opened for. On the other hand, I have the feeling that your rationale for this proposal goes far beyond this issue (mkinitfs 2.0 😉). The question is whether we should consider these problems together, or for simplicity one at a time.

Still, in general terms, of this proposal:

I like the merging or initfs modules. Mostly because is something people are used to. I would however still leave those in the deviceinfo as the default ones (maybe rename to "default" modules?), since like you say, it keeps the deviceinfo as a source of truth
I do not like the merging of cmdline fragments. Maybe it's just my experience, but I am used to just have the option to append to the cmdline, and that possibly solves 99% of the usecases. Breaking deviceinfo for what I see as 1% feels unnecessary, but maybe you can convince me otherwise :)
A "compiler" script that apps can "source $(getdevinfo) makes sense if we have more than 2 users. Right now I can only think about boot deploy (mkinitfs would have to re-implement in go anyway). Do we have or plan a second user? If so, it makes total sense :)

By Pablo Correa Gomez on 2023-02-20T22:22:13

Regarding deviceinfo as the "source of truth", it's not today. I mean, it mostly is, but there are some "leaks". For example, mkinitfs can slurp up module lists, in addition to what's listed in deviceinfo. I'd actually rather seem mkinitfs use module list files entirely and not depend on deviceinfo for getting device modules... but that would mean taking modules out of deviceinfo (I think mkinitfs is the only consumer of the modules list in that file?) Boot-deploy is dealing with kernel cmdline options from an external cmdline.txt now, along with cmdline from deviceinfo (which may be a mistake, but I merged it, so... the cat is already out of the bag)...

I'm trying to rely as little as possible on deviceinfo in mkinitfs, here's a commit in a secret branch to demonstrate that: https://gitlab.com/postmarketOS/postmarketos-mkinitfs/-/commit/547fb1a11c92bdeea0b53461afe4fdde057ff618

The question is whether we should consider these problems together, or for simplicity one at a time.

That's well said. I think we should consider these problems together or else we risk having to redo things again when we try to solve the "other problems", if that makes sense.

By clayton craft on 2023-02-20T23:18:35

That's well said. I think we should consider these problems together or else we risk having to redo things again when we try to solve the "other problems", if that makes sense.

It makes sense. But I am also getting a bit confused on which is your exact rationale, but I guess we can summarize after agreeing on something :P

Because of what you mention in the commit you are providing, I think moving the initfs modules out of the deviceinfo makes sense, but also want more details of what you want to do with deviceinfo. I wonder how convenient would it be for packagers might be a question, since now they might have to add an additional file to every device package. But that's more like a detail. If that makes sense to you, I would take it.

But I am still really against the idea of creating the cmdline from fragments. In that specific case, falling back to the regular deviceinfo var and append makes much more sense to me. And yes, the logic is already implemented, but it was implemented as a way of circumventing the issue we are trying to solve now. I'd rather revert it and fix the core of the problem instead of sticking to a "circumvention" that ignores some of the design decisions at the core of pmOS and unnecessarily spills configuration all around the place.

By Pablo Correa Gomez on 2023-02-21T00:10:00

if I were to design this all today, I would probably do something like:

deviceinfo in the paths we talked about previously (packaged in /usr/share, user configurable in /etc)
deviceinfo only contains "simple" types, things that are override-able like deviceinfo_zram_swap_pct=60. It would not contain variables that might require "merging" like deviceinfo_modules_initfs
More complicated types (e.g. deviceinfo_modules_initfs) should be in separate files, the app that uses them can define its own "merge" strategy, or whether to allow users to modify them, etc.
Apps use deviceinfo like we talked about above, first source from /usr/share, then source from /etc. That's it.

Right now I think the things in deviceinfo that might require merging (so, candidates for not having in deviceinfo using my "ideal" design at the top of this comment), are:

deviceinfo_modules_initfs
deviceinfo_kernel_cmdline
(more? those are the only two I could find...)

Coincidentally, users who end up modifying deviceinfo are probably trying to change one or both of those two variables... heh.

And yes, the logic is already implemented, but it was implemented as a way of circumventing the issue we are trying to solve now. I'd rather revert it and fix the core of the problem instead of sticking to a "circumvention" that ignores some of the design decisions at the core of pmOS and unnecessarily spills configuration all around the place.

Ya, we can revert that stuff, there's still time imho. But I want to have an agreed plan in place + patches to replace that functionality (letting users modify the cmdline) when we revert it.

By clayton craft on 2023-02-21T00:35:56

Here's output from a new update to mkinitfs I've been working on:

2023/02/20 17:09:37 Generating for kernel version: 6.2.0-0-edge
2023/02/20 17:09:37 Output directory: /boot
2023/02/20 17:09:37 == Generating initramfs ==
2023/02/20 17:09:37 - Creating directories specified in /usr/share/mkinitfs/dirs
2023/02/20 17:09:37 -- Creating directories from: /usr/share/mkinitfs/dirs/00-initramfs-base.dirs
2023/02/20 17:09:37 - Creating directories specified in /etc/mkinitfs/dirs
2023/02/20 17:09:37 - Including file lists from /usr/share/mkinitfs/files
2023/02/20 17:09:37 -- Including files from: /usr/share/mkinitfs/files/00-initramfs-base.files
2023/02/20 17:09:37 -- Including files from: /usr/share/mkinitfs/files/30-postmarketos-bootsplash.files
2023/02/20 17:09:37 - Including file lists from /etc/mkinitfs/files
2023/02/20 17:09:37 - Including hook scripts from /usr/share/mkinitfs/hooks
2023/02/20 17:09:37 - Including hook scripts from /etc/mkinitfs/hooks
2023/02/20 17:09:37 - Including kernel modules
2023/02/20 17:09:37 -- Including modules from: /usr/share/mkinitfs/modules/00-default.modules
2023/02/20 17:09:37 - Writing and verifying archive:  initramfs
2023/02/20 17:09:37 initramfs completed in: 121.840949ms
2023/02/20 17:09:37 == Generating initramfs-extra ==
2023/02/20 17:09:37 - Including file lists from /usr/share/mkinitfs/files-extra
2023/02/20 17:09:37 -- Including files from: /usr/share/mkinitfs/files-extra/00-initramfs-extra-base.files
2023/02/20 17:09:37 - Including file lists from /etc/mkinitfs/files-extra
2023/02/20 17:09:37 - Including hook scripts from /usr/share/mkinitfs/hooks-extra
2023/02/20 17:09:37 - Including hook scripts from /etc/mkinitfs/hooks-extra
2023/02/20 17:09:37 - Writing and verifying archive:  initramfs-extra
2023/02/20 17:09:37 initramfs-extra completed in: 75.999916ms
2023/02/20 17:09:37 mkinitfs completed in: 199.432126ms

If the modules are just a list in a file, it's easy to slurp up. But I agree that the change to move device packages from using deviceinfo to list modules to installing a new file for it is going to be a non-trivial amount of work. But maybe it's worth doing that work if it reduces out technical debt in the long term...

By clayton craft on 2023-02-21T01:13:13

Now I'm totally convinced about your rationale for having the list of modules outside the deviceinfo. What you propose makes sense, and if the team agrees that it's sensible for new and existing ports, I'm also fine with that.

However, I am not convinced that kernel_cmdline should be treated in the same way. The list of modules is exactly and simply that, a list of modules. It does not matter the order and every module can simply and only be a plain string. There is also often the need to pick modules from different packages (e.g: the hooks), and the need to merge them all is real. I see the cmdline variable differently. It can not just contain strings, but also some sort of variables. So what happens when one fragment configures ps=1 and the next one ps=0? If one sets quiet the next verbose? Like sure, we just say we sort things alphanumerically... But does that anybody really need that? Can we not live with a single append variable that boot-deploy can very easily parse? Do we need a complicated solution to solve a no-problem, or even the same problem that we can solve with a simple solution? Like I get the long-term benefits for the modules, but I do not for the cmdline.

By Pablo Correa Gomez on 2023-02-21T17:33:11

changed the description

By clayton craft on 2023-02-17T01:07:46

changed the description

By clayton craft on 2023-02-17T01:08:24

Somewhat related: I've been thinking about if we want to revamp how we handle device pages. Currently, there's some degree of overlap between the information contained in deviceinfo and our wiki pages, resulting in duplication. Additionally, wiki pages themselves often contain information that is virtually identical between different device pages since installation instructions often are more or less the same on different devices. What I had in mind was if deviceinfo could be extended to include sufficient information to automatically generate a basic device page with installation instructions and an information box with specs and a list of working features, but not all the notes that the current wiki pages have since it wouldn't make sense to include all that in deviceinfo. This would then be hosted on postmarketos.org rather than the wiki, and the downloads page there would contain a list of links to these pages rather than a list of links to direct downloads. Furthermore, this information could also be used by a hypothetical future desktop installer for postmarketOS.

The benefit of this as I see it is that it would reduce duplication of information, and make device pages more straightforward to users only interested in trying out postmarketOS. It would also let us improve the download page which I personally think is a bit lacklustre right now. I think the current split between the wiki and postmarketos.org is rather awkward. And, as mentioned, it would help a future desktop installer.

I do realise that this proposal definitely deserves its own issue for separate discussion, but since we're making significant changes to how deviceinfo works I wanted to throw this out there in case we want to take this idea into consideration when we're changing deviceinfo anyway. There are some other issues I would like to discuss regarding this idea (primarily, how we handle all the other information that device pages currently contain), but since that's not really related to deviceinfo I think that is better discussed about elsewhere.

By Newbyte on 2023-02-20T23:11:46

First off, I'm glad that you're trying to consolidate things and make feature status more current/consistent. I think that adding static device specs, like the SOC model, to deviceinfo is fine. I guess it would also give apps a local source for all of the device specs, that they can use however they want.

I'm not so sure about adding "supported features & status" to deviceinfo though. I'm worried that it won't be maintained any more than the wiki pages are now. It seems like it would increase the burden for maintainers to maintain it (submitting patches) and for us to review/merge them. I totally get that having "good" status is key to getting more people to use it more confidently, but here "good" means 1) "everything is working!" or 2) "this is being tested very frequently for regressions".

Ideally, features would be tested in a CI / board farm, and the output from that could be used to indicate status "real time" on the wiki or some other page/site, and it would do this frequently to catch regressions fast. I think that until we reach the point where we can have automation tell us how good a thing is, any effort spent on trying to keep more accurate device status right now might be better spent on trying to get us towards some automated future.

For devices that aren't tested like that then manually updating the status, what we do today, is "fine" I guess. The current system for getting device feature status is janky, but you also still have to be a fairly technical user[1] (most of the time it seems) to keep a daily driver install going even if you run a stable release.

Another thing to consider is that if we do use automation to help in the future, having the feature status for a device in the deviceinfo means that we either give automation push access to pmaports (ewww!) or we have to review/merge a ton of patches to maintain status in deviceinfo.

And in some cases (boot-deploy, others?) we're using deviceinfo as a configuration file in apps that are used in other distros. Adding "feature status of this device in pmOS" to the spec makes even less sense there.

someone in chat the other day was very unhappy that they had bought a OP6 and couldn't use it as a daily driver, I forget the actual issue that brought them there, but it seemed obvious to me that they weren't going to have a great time running pmOS as it is today.

By clayton craft on 2023-02-20T23:11:46

To summarize the current status of the agreement between @craftyguy and me on how to solve this issue:

deviceinfo will be moved to /usr/share (or /usr/share/misc if we want to be fully compliant with FHS).
Packages using deviceinfo will read deviceinfo frist from /usr/share and then from /etc, merging a giving preference to the second one. This warranties backwards compatibility and allows to solve the source of problems having deviceinfo in /etc.
The merging strategy becomes quite complex with long multi-valued deviceinfo variables that contain strings. The main problem is that adding things prevents from updates getting through, since the user would have to keep both in-sync. The two most relevant ones, with their treatment are:
- deviceinfo_modules_initfs: The proposal is to move this variable out of deviceinfo and instead place the list of necessary modules into a file that can be dropped somewhere into /usr/share/mkinitfs. This is not completely breaking since mkinitfs already reads from some of those places. It is some implementation work, but it could be scripted and the mkinitfs part is already in place, so not an issue. The only thing to consider is the change in the process necessary by device porters. Somebody with experience on this should provide their input :) Users can drop extra modules in /etc/mkinitfs, so user customization is still possible.
- deviceinfo_kernel_cmdline: The proposal is to create a new deviceinfo_kernel_cmdline_append variable that users can use to append more things to the cmdline. Users shall be advised to use that variable in /etc/deviceinfo, for example by providing that file with some default commented-out documentation. More complicated merge strategies in the cmdline are on purpose not considered due to the complexity (e.g: how would one merge cmdline fragments with opposite values like quite verbose or ps=1 ps=0). Neither it is possible to remove options from the cmdline, since that's not an option anywhere else and will most likely break kernels.
After this process, users own /etc/deviceinfo and still have reliable ways to configure the kernel cmdline and the modules that should go into the initfs. In most ways, the behavior of the programs that deal with this can be kept backward compatible, reducing the chances for breakage :)

By Pablo Correa Gomez on 2023-03-01T06:51:50

This proposal sounds great, I like how this avoids a rather complex merging strategy of variables.

The only thing to consider is the change in the process necessary by device porters. Somebody with experience on this should provide their input :) Users can drop extra modules in /etc/mkinitfs, so user customization is still possible.

To change the porting process / documentation in general, we would do this:

explain the new method in the device specific package wiki page
adjust the deviceinfo reference:
- move the option to obsolete
- link to the new documentation in the description of why it's obsolete
mark deviceinfo_modules_initfs as obsolete after we adjusted all devices in edge, so CI will complain when it is still used: .ci/testcases/test_deviceinfo.py:deviceinfo_obsolete()
adjust pmb/aportgen/device.py in pmbootstrap to not have the deviceinfo_modules_initfs option anymore

deviceinfo_kernel_cmdline

Sounds reasonable to me as well. CC @jenneron who recently added /etc/boot/cmdline.txt, which fulfills a similar purpose (but without the option to append to the default kernel cmdline for the device) and would be replaced by that.

By Oliver Smith on 2023-03-01T06:51:50

Sounds reasonable to me as well. CC @jenneron who recently added /etc/boot/cmdline.txt, which fulfills a similar purpose (but without the option to append to the default kernel cmdline for the device) and would be replaced by that.

Yes, we discussed this in https://gitlab.com/postmarketOS/boot-deploy/-/merge_requests/25

By Pablo Correa Gomez on 2023-03-05T18:01:41

one addition to this that we talked about it a team meeting: we should provide a default /etc/deviceinfo file, with config in it that we think users will want to modify, with the vars commented out. This won't change any config on the system, but serve as a starting point for those who want to modify deviceinfo config. The file should also reference the deviceinfo wiki page.

cc @calebccff

By clayton craft on 2023-04-12T18:43:54

That makes total sense. I'll start working on it, let's see how this works out 😄

By Pablo Correa Gomez on 2023-04-12T21:59:36

Can this new approach account for building generic images in the future? We're still quite a ways off that but I figure this is worth considering now as it's already a large migration.

The big issue is that device packages conflict, I suggest that when adjusting the deviceinfo paths we also rename the file to include the device codename in vendor,device format, with an optional arbitrary variant suffix, we also assign a "component type" for future use. Right now we'd only have "device" components (though I plan to add a "platform" component to de-dup some things in the future).

The variant suffix would be used where multiple versions of a device with different deviceinfo properties are needed, I'm not sure if this actually applies to any devices currently in pmaports but it seems sensible to include "just in case". Pmbootstrap would treat each variant as its own unique device, and how it's determined by mkinitfs is TBD for now (deviceinfo helper quirks).

We'd rename all the deviceinfo files to deviceinfo-device-<vendor>,<codename>[_variant] in pmaports, e.g. deviceinfo-device-oneplus,enchilada. The file is installed to /usr/share/deviceinfo/ (so making it a directory). For devices which use devicetree the correct deviceinfo can easily be determined this way. We do not yet have generic images, so anything parsing deviceinfo can just glob for deviceinfo-device-* and get the first match.

As an initial step this lets us install multiple device packages which is pretty big. It makes choosing the correct deviceinfo file possible too for a lot of devices.

Of course /etc/deviceinfo still behaves exactly as described in the proposal, this should just be a name change and not affect the behaviour described by the proposal. I suggest doing it now because it is already necessary to adjust the behaviour of everything that parses deviceinfo to implement the new proposal, applying this new naming scheme should make future changes to this system a whole lot easier.

why/how/future ideas

Everything below here is just fruitless pondering about how to utilise this.

I remember we discussed the idea of having a helper utility, but discarded the idea because it has to be parsed in pmbootstrap and other tools too. I think it might make sense to reassess this to get generic images working. Having multiple fragments be combined to produce a final deviceinfo configuration would be necessary to de-dup and make sure everything is in sync. I think a simple "include" directive works here, yes this breaks compatibility but I think thats ok, a helper utility makes it possible to properly formalise everything and can just as easily be invoked from a shell script to set all the variables.

I think having a tool to implement all this logic (and handle variant matching) makes a lot more sense here. The use of deviceinfo by pmbootstrap means we need to be able to "build" a deviceinfo from just the pmaports source as well as on the device rootfs and come up with the same answer. By using the component "type" identifiers I suggested above and having a unique part to the file name (e.g. platform-qcom,sdm845 to define all the sdm845 generic options) we can keep parsing to O(n) time by finding all files in pmaports that begin with "deviceinfo-". pmbootstrap just needs to produce a boot image for every possible combination so there would be no complexity in selecting the right deviceinfo.

It shouldn't be necessary to have more than a single device and platform component to create a full deviceinfo, but this approach makes it possible to generalise further if that becomes necessary by introducing new component types which have their own behaviour.

I think this makes partially generic device images possible - like what Mobian provide. Where a single rootfs is associated with a selection of Android boot images. We can then have pmbootstrap build an image from a list of device packages, generating a boot image for each device in turn. This would eventually let us have a single rootfs image for the OnePlus 6, 6T, SHIFT6mq and Poco F1.

By Caleb Connolly on 2023-04-16T17:36:37

For devices which use devicetree the correct deviceinfo can easily be determined this way.

What about x86{,_64} ? If we're going to support a unified image, it would make sense to also support platforms with this arch too, there are a lot of tablets, etc that are similar, but still require some differences in deviceinfo (e.g. for cmdline, dimensions, etc). But there's no identifier really, unless we get it from smbios tables maybe?

so anything parsing deviceinfo can just glob for deviceinfo-device-* and get the first match.

This, combined with having ID the platform and then gather files from multiple locations, makes the task "parse deviceinfo" a bit more complex, and it would need to be implemented everywhere we "parse deviceinfo". It's more complex than what the original proposal called for: 1) source deviceinfo from /usr/share, 2) then source deviceinfo from /etc to overwrite anything from (1)

This would eventually let us have a single rootfs image for the OnePlus 6, 6T, SHIFT6mq and Poco F1.

How many other devices would we be able to group up like this into a single rootfs? I'm guessing that it's not many... The L5, PP/PPP and others all require additional services, udev rules, kernels, etc for enabling things specific to those devices, so it seems like those (potentially hundreds of devices?) would never be able to share a rootfs, is that right?

Besides saving a little disk space on our download infra, and saving a little time by having to build 1 image instead of 4, what other practical benefits does supporting generic images have for pmOS? I'm worried about further increasing complexity in our codebases to support this, and not sure that the benefits are worth it, so I need some more convincing 😅

By clayton craft on 2023-04-16T19:09:48

Installing devices deviceinfo under /usr/share/deviceinfo/device-vendor,codename is something I like. Allows for future extensibility, and opens the door for some of the things Caleb proposed. However, many of the other things I believe are out of scope of this proposal, which is aiming to solve a very specific issue. For now, I'd say the best way to keep compatibility would be to create a symlink /usr/share/deviceinfo/deviceinfo pointing to the only one existing (this should be easy with devicepackage-dev that I have to touch anyway). @craftyguy, I think this opens the door for future improvements, with minimal changes to what we were planning. And then leaves the complex task for the future. Would that make sense as a middle point? I'm still like one or two weeks away from starting to work on this, so there's some time to discuss :P

By Pablo Correa Gomez on 2023-04-17T10:01:03

I could probably have done a better job at being explicit above. My motivations for suggesting this are that I anticipate some kind of rename like this to be useful in the future. It's a whole lot easier to do it now for every device given that we're already modifying all device packages than it would be to do it on a per-device basis down the line.

If we rename all deviceinfo files from deviceinfo to deviceinfo-device-vendor,codename (both in pmaports and in the device package) then we finally make it possible to install multiple device packages at once. Obviously this doesn't just immediately let us do generic images, some devices have unconditional quirks or different kernels - things which we ought to fix regardless and that might benefit from a little bump in motivation.

The -device- part is meaningless right now but could be really really handy in the future. IF we end up introducing more complexity with other "types" of deviceinfo file then it allows those to be differentiated. If not then no matter, it can just be ignored. It's much easier to add it now and play it safe.

So to be clear, what I'm suggesting we commit to right now is just renaming the file and creating a symlink. The only complexity this adds is in the merge request which makes this change. Parsing deviceinfo would be identical to the original proposal due to the symlink from the device-specific file to the generic /usr/share/deviceinfo/deviceinfo

As far as the future goes, if we want generic images then yes, this adds complexity. I don't see any way around this and the tradeoff seems more than worth it, for UX even if nothing else. We can encapsulate that complexity into a deviceinfo helper utility and gradually transition everything to use it (because this change is entirely backwards compatible with the existing proposal).

By Caleb Connolly on 2023-04-17T16:01:08

But... there's no reliable way to detect vendor,codename for all devices without the deviceinfo, right? You can't use values from deviceinfo when trying to figure out which deviceinfo to read...

...

On Mon, 17 Apr 2023 10:01:10 +0000 "Pablo Correa Gomez (@pabloyoyoista)" gitlab@mg.gitlab.com wrote:

Pablo Correa Gomez commented on a discussion: #1836 (comment 1354968718)

Installing devices deviceinfo under /usr/share/deviceinfo/device-vendor,codename is something I like. Allows for future extensibility, and opens the door for some of the things Caleb proposed. However, many of the other things I believe are out of scope of this proposal, which is aiming to solve a very specific issue. For now, I'd say the best way to keep compatibility would be to create a symlink /usr/share/deviceinfo/deviceinfo pointing to the only one existing (this should be easy with devicepackage-dev that I have to touch anyway). @craftyguy, I think this opens the door for future improvements, with minimal changes to what we were planning. And then leaves the complex task for the future. Would that make sense as a middle point? I'm still like one or two weeks away from starting to work on this, so there's some time to discuss :P

By clayton craft on 2023-04-17T17:03:26

Ok so if all you're proposing is adding a symlink, then that's not so bad, but I think we should do that later when we have a better idea of how it will be used. I think that if we add it now, someone will start using it... so if we decide later that we want to name it differently then we'll have to introduce a breaking change or keep around the "old" symlink. That's my 2 cents. I wouldn't NAK any patch that adds just the symlink you described though, I just think it might warrant some more thinkin/discussions (i.e. my comment elsewhere about how we can't reliably figure out some things on some platforms for even finding it)

...

On Mon, 17 Apr 2023 14:16:02 +0000 "Caleb Connolly (@calebccff)" gitlab@mg.gitlab.com wrote:

Caleb Connolly commented on a discussion: #1836 (comment 1355476074)

I could probably have done a better job at being explicit above. My motivations for suggesting this are that I anticipate some kind of rename like this to be useful in the future. It's a whole lot easier to do it now for every device given that we're already modifying all device packages than it would be to do it on a per-device basis down the line.

If we rename all deviceinfo files from deviceinfo to deviceinfo-device-vendor,codename (both in pmaports and in the device package) then we finally make it possible to install multiple device packages at once. Obviously this doesn't just immediately let us do generic images, some devices have unconditional quirks or different kernels - things which we ought to fix regardless and that might benefit from a little bump in motivation.

The -device- part is meaningless right now but could be really really handy in the future. IF we end up introducing more complexity with other "types" of deviceinfo file then it allows those to be differentiated. If not then no matter, it can just be ignored. It's much easier to add it now and play it safe.

So to be clear, what I'm suggesting we commit to right now is just renaming the file and creating a symlink. The only complexity this adds is in the merge request which makes this change. Parsing deviceinfo would be identical to the original proposal due to the symlink from the device-specific file to the generic /usr/share/deviceinfo/deviceinfo

As far as the future goes, if we want generic images then yes, this adds complexity. I don't see any way around this and the tradeoff seems more than worth it, for UX even if nothing else. We can encapsulate that complexity into a deviceinfo helper utility and gradually transition everything to use it (because this change is entirely backwards compatible with the existing proposal).

By clayton craft on 2023-04-17T17:07:47

But... there's no reliable way to detect vendor,codename for all devices without the deviceinfo, right? You can't use values from deviceinfo when trying to figure out which deviceinfo to read...

We could possibly use some sort of compatible to vendor-codename mapping.

By Newbyte on 2023-04-17T17:53:06

Also mentioned this earlier.. but not all platforms have "compatible" (e.g. x86/x86_64)

...

On Mon, 17 Apr 2023 17:53:08 +0000 "Newbyte (@Newbyte)" gitlab@mg.gitlab.com wrote:

Newbyte commented on a discussion: #1836 (comment 1355869844)

But... there's no reliable way to detect vendor,codename for all devices without the deviceinfo, right? You can't use values from deviceinfo when trying to figure out which deviceinfo to read...

We could possibly use some sort of compatible to vendor-codename mapping.

By clayton craft on 2023-04-17T17:59:42

Also mentioned this earlier.. but not all platforms have "compatible" (e.g. x86/x86_64)

Yeah, sure, but you have other ways of identifying devices there. For example, the DMI table.

By Newbyte on 2023-04-17T18:51:47

But... there's no reliable way to detect vendor,codename for all devices without the deviceinfo, right? You can't use values from deviceinfo when trying to figure out which deviceinfo to read...

for the rename we can just use the device package name. For doing this at runtime, no a perfectly generic solution isn't possible. We'd migrate to a deviceinfo helper thing and then have a lookup table and various mechanisms for identifying a device, or more simply we do devicetree and if that's not available then we just shove it in the kernel cmdline. Though, again, to what extent this would be applicable to an x86 device I really don't know.

I get that I'm not offering proper answers here, I don't want to go down a rabbit hole of solving a problem we don't actually have. ACPI/x86 devices make up a TINY minority of the devices we support, and this change doesn't even effect them negatively.

Like, this will only be an issue if we want to have a generic image that supports a few different specific ACPI devices that have no way of being differentiated from each other (either by a deviceinfo helper utility at runtime or some openrc service that sets up symlinks during boot). That would absolutely be a real issue that someone would have to solve in order to be able to do generic images for those devices. But like, how does preventing multiple device packages from being installed at once make this any easier?

Ok so if all you're proposing is adding a symlink, then that's not so bad, but I think we should do that later when we have a better idea of how it will be used. I think that if we add it now, someone will start using it...

The only way I can forsee this being used is for generic images, and if someone starts working on that then that's a good thing right? If folks come up with a fun unexpected way to depend on how these files are named and we want to rename them then I don't think anyone would be too upset, like I don't really understand the concern here.

Doing the rename later should be alright, I don't want it to be rushed through prematurely or anything, it just seems sensible to chuck it in while we're here. It doesn't make any sense to me not to rename deviceinfo to something device specific and now is a perfect time. I wish I had come up with this idea a bit earlier.

By Caleb Connolly on 2023-04-17T18:56:09

We'd have to add and maintain some mapping for that... which supports the point I keep repeating here: this is going to increase complexity, and I'm not sure it's worth it.

...

On Mon, 17 Apr 2023 18:51:48 +0000 "Newbyte (@Newbyte)" gitlab@mg.gitlab.com wrote:

Newbyte commented on a discussion: #1836 (comment 1355921443)

Also mentioned this earlier.. but not all platforms have "compatible" (e.g. x86/x86_64)

Yeah, sure, but you have other ways of identifying devices there. For example, the DMI table.

By clayton craft on 2023-04-17T19:06:25

Caleb Connolly commented on a discussion: #1836 (comment 1355924401)

I get that I'm not offering proper answers here, I don't want to go down a rabbit hole of solving a problem we don't actually have. ACPI/x86 devices make up a TINY minority of the devices we support, and this change doesn't even effect them negatively.

Ya, and the set of devices that could even use generic images is even smaller, right? x86 would be a good candidate for generic images, so it would be nice if the design of this didn't preclude them. I don't know what the answers are either, but my intent is to point out that supporting generic images requires a bit more discussion to make sure that the design is sound.

The only way I can forsee this being used is for generic images, and if someone starts working on that then that's a good thing right? If folks come up with a fun unexpected way to depend on how these files are named and we want to rename them then I don't think anyone would be too upset, like I don't really understand the concern here.

Don't assume people won't use things in ways that you didn't intend :) https://m.xkcd.com/1172/

Doing the rename later should be alright, I don't want it to be rushed through prematurely or anything, it just seems sensible to chuck it in while we're here. It doesn't make any sense to me not to rename deviceinfo to something device specific and now is a perfect time. I wish I had come up with this idea a bit earlier.

It's just a symlink, we can (and IMHO, should) add it later once we are sure about the name (and if it's even the right approach...), and so on.

But I think trying to sort this all out right now is distracting the original intent of this proposal: to fix how we install deviceinfo so that user changes don't conflict with apk

If there's nothing in the current proposal here that would prevent moving forward with the generic images thing, then I think it would be best to file a new issue to discuss changes that would be needed on top of this proposal for supporting that. A deviceinfo handler, symlinks, etc can all come later if they are actually required, and I think that supporting generic images requires more thought about the design before we start implementing any of it... Do you agree?

...

On Mon, 17 Apr 2023 18:56:11 +0000 "Caleb Connolly (@calebccff)" gitlab@mg.gitlab.com wrote:

By clayton craft on 2023-04-17T19:22:29

I talked with @calebccff over chat about this some more, and I think the changes they are proposing to add on aren't as intrusive as I initially understood. So, I agree with them + @pabloyoyoista that this is (relatively) low-hanging fruit that should just be included now to make future progress on other projects (generic images) easier.

By clayton craft on 2023-04-17T20:47:42

Lovely to see you two figure it out :D In worst case, I'd have argued that /usr/share/deviceinfo/deviceinfo is a better location than /usr/share/misc/deviceinfo. But that seems out of scope now :)

By Pablo Correa Gomez on 2023-04-17T21:21:00

assigned to @pabloyoyoista

By Pablo Correa Gomez on 2023-03-05T18:03:10

mentioned in merge request !3670 (closed)

By Oliver Smith on 2023-04-14T07:39:51

mentioned in commit boot-deploy@a437c9ba

By Pablo Correa Gomez on 2023-04-24T19:31:39

mentioned in merge request boot-deploy!29 (merged)

By Pablo Correa Gomez on 2023-04-24T19:32:35

mentioned in commit boot-deploy@ca28897e

By Pablo Correa Gomez on 2023-04-24T19:48:17

mentioned in commit boot-deploy@535ed507

By Pablo Correa Gomez on 2023-04-24T20:01:33