Turn the Wiki into a knowledge database and decouple the device database

Idea: Turn the Wiki into a knowledge database and decouple the device database

Summary

The postmarketOS Wiki is currently in a weird state where a lot of low-quality, low-effort device pages exist that are just stubs or pages without a postmarketOS port that they would help describe. In my opinion, a Wiki should serve as a knowledge base, not as a device database. By decoupling the device database (i.e. the device wiki pages) from the Wiki and moving them to a Git-Repository (pmaports), we enjoy multiple benefits:

• Device feature table status mirrors pmaports
• Non-booting, non-packaged devices won't have device pages (as opposed to currently, where making matters worse, a lot of pages don't specify booting = no and packaged = no)
• Quality control is easy to maintain and changes can be discussed beforehand (GitLab MR reviews as opposed to people patroling recent Wiki changes and feeling permitted to revert changes)
• We can validate structured data with a schema (as opposed to Infobox changes silently breaking hundreds of device pages)
• A statically generated devices website that people can browse through in a design of our choice (not limited by MediaWiki)
• The Wiki can actually focus on being a source of information on things like UIs, App recommendations, specific configurations like the firewall, etc. rather than being a database of infoboxes

Specification

I propose the following:

• Create a subdirectory devices/ in pmaports, inside which for every supported device we have a directory [vendor]-[codename] with 4 files:
  • data.yaml or data.toml (any structured data format that can be validated against a schema, really, I'd prefer TOML) containing the information we currently keep in infoboxes
  • flashing.md for installation instructions specific to the device, like unlocking the bootloader, flashing U-Boot or an EDL loader that is required
  • quirks.md for device-specific quirks (like "5GHz WiFi requires running the following nmcli command", etc.)
  • image.jpg for a picture of the device
• Issues should reside in GitLab issues now and be tagged for each device and assigned to the maintainers
• When a device is updated (e.g. its kernel or device package change feature support), the information in the devices/ directory is also updated in the same MR
• We set up a static site generator of choice that generates a devices.postmarketos.org page with the information in the directory and deploys on every pmaports commit that touches the directory
  • We could even do funky things like having the generator parse our kernel packages for the devices and look up the kernel versions from there instead of hardcoding it in the data.toml!
• Migration would happen as follows:
  • Parse the devices cargo table in the wiki
  • Filter for devices currently in pmaports
  • Generate data.toml from the infobox contents, download the device image as needed
  • Parse the Wiki page, find a section matching Installation or Install and put that in flashing.md (with manual review)
  • Keep quirks.md empty / not have one at all (this should be equivalent)
  • Maintainers would perform a proper update of this info next time their device is updated in pmaports
  • This way migration can be done mostly automated initially and will be up to date quite soon for devices that are maintained

Drawbacks

• The bar for changing info for a device is higher - you need to go through the pmaports MR workflow now
  • I would argue that this is a benefit, since it encourages higher quality changes
  • Information wouldn't need to be updated often, known issues are bug reports on GitLab instead and not a MR to edit a markdown file
• We would probably lose out on some info in the migration process, but I think having a much cleaner Wiki and all the other benefits heavily outweigh this downside

changed the description

Thanks for writing this down! I would probably do some things differently in the implementation, but overall I think this is a very good idea, and IMHO it is worth making a PMCR for this.

Thoughts (no need to reply to them, this can be discussed in detail in the PMCR):

• I would not store image files of the devices in pmaports.git, this just blows up the size needlessly which means users need to clone all that additional data, and also CI jobs need to clone that. Even if you do a shallow clone, you would get all the images in their current state. Maybe it would make sense to keep images in the wiki, but put the link in some config/yaml file?
• Personally not a fan of yaml, maybe toml instead?
• Storing the metadata in pmaports has the advantages that they can be updated as e.g. bugs are fixed / features are enabled, and they can now document the current state of a given branch, which is great! When we build the docs and publish them somewhere, I would also make this branch specific, so users can read the docs for the release (or edge) they are using.
• We already duplicate information from the wiki in some places, e.g. in bpo configs (device codename -> full name), and the website. It would be really nice to have this in a proper machine readable format in one place.
• Directory name: I'd name it docs/devices or something, so it is clear that this directory does not contain packages, only documentation / metadata.

We discussed this idea informally during the FOSDEM hackathon, and there's some related ideas I want to bring up:

Combine deviceinfo and the device database

This would remove the duplicated data, and give us a good excuse to finally migrate away from shell for deviceinfo which would give us additional flexibility in the format.

Standardising installation instructions

Something else which is quite inconsistent on the wiki is install instructions. These are in a purely human readable format and requires duplicating a lot of common advice/guidance between pages for similar devices.

We could also take this opportunity to adopt the ubports installer config format and use it to describe installation instructions. This makes it machine readable allowing for use in interactive installers as well as to produce consistent looking installation instructions, plus we can abstract away common information like handling A/B slots or interacting with lk2nd.

Publishing device info on the website

Putting this all together, we could publish elegant and consistent device pages on the postmarketOS website with installation instructions and direct download links just like the lineageOS website.

This would be a HUGE upgrade for accessibility, allowing us to provide much more interactive and user friendly guidance, better written guides, and easily integrate things like the web flasher for supported devices too.

What's left of the wiki?

It would still be nice to produce device feature tables on the wiki with the info sourced from pmaports, but moving installation instructions to the website would free up device pages for documenting quirks, workarounds and developer info.

The primary downside I see to all of this is that it introduces additional separation between end-users and contributors. Today it's expected that if you use postmarketOS you will visit the wiki and read at least the page for your device, but if that's no longer necessary then people may pay less attention to the wiki or be less likely to visit it which could have some knock-on effects.

Overall I suspect the tradeoff is probably worth it, having all this info integrated seamlessly into the website would really make us look a lot more professional and make the installation process a lot more seamless I think.

Implementing all this

I think it makes sense to go something like this:

1. Decide where to host device images. A git repo probably doesn't make sense, maybe the wiki? Or some simple web service where folks upload images that get manually approved by pmOS mods (https://serve.postmarketos.org ?)
2. Define a TOML format for deviceinfo + feature table, adjust devicepkg-dev to generate the classic deviceinfo file from the TOML. Data currently stored on the pmOS website and BPO about images/uis/etc should be moved here too.
3. Add a CI job to pmaports master that updates device pages on the wiki based on the feature table in pmaports (this will be necessary for the migration at least).
4. Rework the postmarketOS website to have a dedicated page for each device which contains the feature table and download links.
5. Rework BPO to use info from pmaports as well
6. Write ubports installer configs for devices and start to add generated installation instructions to the website with agreed upon process

Combine deviceinfo and the device database

The issue with that is generic ports. Currently, devices supported by generic ports generally have separate device pages that list working features for that specific device. If deviceinfo and the database are merged, it would create false expectations because the feature support wouldn't match the actual features present on the device.

I think combining deviceinfo and the device database is probably kind of possible. In my view the devices/ directory would contain directories for the generic device as well as the specific devices supported by it. The individual pages would contain feature status and refer to the generic package, which could then be used for the flashing instructions. Sure, this will need some more detailed ironing out, but I think it's possible

Nice! This idea is something we've discussed multiple times. One cool thing that maybe would be good to think about in this case, is how to make the "device database" as generic as possible. So maybe, at one point, we could have others reusing it if possible. Not totally sure how feasible that is, but generally something to think about for a PMCR

Also, thanks a lot for looking into this! Given the responses, I think we're quite excited to have a PMCR to work towards this :D

Some related random thoughts:

• What does this mean for generic ports? You'd have a page for the generic port itself, sure, but what about the devices it supports, but which don't have separate device packages? (Unrelated - would be nice to have a thing in pmbootstrap where typing in a device codename auto-selects the correct generic port...)
• SoC pages can contain SoC-specific quirks which apply to all devices with an SoC. Do we want to move those as well or do they stay on the wiki? If they stay on the wiki, how do we handle listing devices that use them (or do we just drop it)?

What does this mean for generic ports?

See my reply to Aster above.

SoC pages can contain SoC-specific quirks which apply to all devices with an SoC. Do we want to move those as well or do they stay on the wiki? If they stay on the wiki, how do we handle listing devices that use them (or do we just drop it)?

We can make use of a templating engine on top of Markdown to include a SoC-specific quirks.md in the device-specific ones.

While I like the idea of having a device database, the suggested implementation needs more work due to reasons listed further.

First of all, do you really have to move all device-related information to pmaports? A huge disadvantage of such solution is that the contribution process will be much slower compared to wiki + it puts a huge workload on maintainers. Let's say, I want to add something to quirks.md. I have to wait until it gets reviewed and merged and sometimes this process takes a lot of time even for one-liners from my experience. Same goes for support table and other info, some basic info could be stored in pmaports, but not the entirety of it.

Non-booting, non-packaged devices won't have device pages (as opposed to currently, where making matters worse, a lot of pages don't specify booting = no and packaged = no)

I strongly oppose this one, since there are lots of useful pages, even if the device is not submitted to pmaports, a page can be useful:

• If the device boots, but is not present in pmaports, the page can be used to advertise current work, inform users on its state and get more contributors - such approach is rather common for mainline forks.
• If the device doesn't boot, but the page lists information on what was tried to boot the device or why the device doesn't boot yet. Such work can be easily picked up by other contributors instead of doing double amount of research.

Having flashing.md is questionable given how many devices have a common way to install pmOS. This issue can be solved by using templates/properties rather than copying the same text across the repo. E.g. there are some deviceinfo properties which describe install partitions, there could be more properties which can be used by both device database and pmbootstrap. Let's say, deviceinfo_supports_android_recovery_zip is added to properties, we can set it to false for a certain device, provide a warning/prompt when --android-recovery-zip is used and show that recovery installation is not possible in installation template.

As conclusion, my suggestion is:

• Store a minimum (necessary) amount of information in pmaports, mostly things which can be reused by another tooling, such as pmbootstrap and bpo, while other info goes to another source. It could be either a separate repo (if you still want to use MR process to ensure higher quality of contribution) or wiki or device db as a web resource where info can be edited interactively, just like in wiki, but, as you suggested, with different design, improved search(wiki drilldown is very bad), possible pre-moderation or automated validation of the page content.
• Allow storing information about devices not included in pmaports either in wiki or in device db. If the information is stored in device db, it has to be pre-moderated to avoid useless pages and spam.

First of all, do you really have to move all device-related information to pmaports?

Yes.

the contribution process will be much slower

And of higher quality.

it puts a huge workload on maintainers

Elaborate? Ticking out some feature boxes when you update the kernel isn't a "huge workload" and they're expected to keep the Wiki pages in sync already anyways.

I have to wait until it gets reviewed and merged and sometimes this process takes a lot of time even for one-liners from my experience.

Merge requests taking a long time to get reviewed is an issue entirely unrelated to this PMCR.

Same goes for support table and other info, some basic info could be stored in pmaports, but not the entirety of it.

Again, elaborate.

If the device boots, but is not present in pmaports, the page can be used to advertise current work, inform users on its state and get more contributors - such approach is rather common for mainline forks.

If it boots and is not present in pmaports, it should just be in pmaports. It isn't healthy to have large device work outside of pmaports (source: done that myself for multiple years and eventually got tired of people asking me about it). The best way to get more contributors is pmaports.

If the device doesn't boot, but the page lists information on what was tried to boot the device or why the device doesn't boot yet. Such work can be easily picked up by other contributors instead of doing double amount of research.

In most cases these pages have a link to a downstream Android kernel. I can find that in a single search online and it is really information that has no added value.

Having flashing.md is questionable given how many devices have a common way to install pmOS

Let me quote my issue text here to answer all your concerns:

flashing.md for installation instructions specific to the device, like unlocking the bootloader, flashing U-Boot or an EDL loader that is required

First of all, do you really have to move all device-related information to pmaports?

Yes.

That could have been a valid answer if you explained why it has to be only in pmaports and why all information has to go there.

the contribution process will be much slower

And of higher quality.

I agree that MR process gives higher quality as you can see from my previous feedback, but I hope to find balance between contribution workload and quality. Higher quality can be achieved by other means too such as template/properties usage, so you don't have to write as much raw text if you have to write any, and they can be easily updated across multiple devices if needed.

it puts a huge workload on maintainers

Elaborate? Ticking out some feature boxes when you update the kernel isn't a "huge workload" and they're expected to keep the Wiki pages in sync already anyways.

I should have phrased it differently, I'm not talking about device maintainers, it's more about pmaports people who have to review documentation changes. Changes in documentation are not always tied to kernel updates. Imagine that you found important information about the device (e.g. that cellular works only after flashing a certain version of stock firmware). Now, to add this information, you have to go through MR process. Wait until someone reviews it, approves and merges. As I said, sometimes that takes a significant amount of time. And I said "huge workload", because there will be lots of such MRs. Some with broken formatting, some with wrong spelling; factual errors - that has to be addressed by reviewers too.

Merge requests taking a long time to get reviewed is an issue entirely unrelated to this PMCR.

It is related, because this PMCR will make things worse in this regard if implemented in current form.

Same goes for support table and other info, some basic info could be stored in pmaports, but not the entirety of it.

Again, elaborate.

Example: user tests the device, assumes that feature X is not working and marks as such. Later on, same user or a different one finds out that the feature does in fact work. The reasons can be different: the user didn't test the device properly or there was some update of userspace program which fixed the bug preventing the feature to work. The question is do we have to review and spam the git log with such minor changes.

If the device doesn't boot, but the page lists information on what was tried to boot the device or why the device doesn't boot yet. Such work can be easily picked up by other contributors instead of doing double amount of research.

In most cases these pages have a link to a downstream Android kernel. I can find that in a single search online and it is really information that has no added value.

While I agree with the booting ports point in general, I can't agree with this one. If majority of such pages are of low quality, it doesn't mean all such pages have to be removed. I'm talking about valuable info, e.g. testpoints/uart, schematics, u-boot forks info, bootloader unlock, bypassing secure boot, etc. Some of this info cannot be found on the internet and some is hard to find.

Having flashing.md is questionable given how many devices have a common way to install pmOS

Let me quote my issue text here to answer all your concerns:

flashing.md for installation instructions specific to the device, like unlocking the bootloader, flashing U-Boot or an EDL loader that is required

Some of this info is generic and can be replaced with templates, the rest goes to quirks.

Thanks both for the comments. I think you've hit quite some pain points that certainly need more discussion. But given how much interest this has raised, I bet it would be more useful to wait to address them in a PMCR, so we can have a discussion with a proposal at-hand.

Seeing all the positive initial responses makes me very hopeful that this will succeed. I'll be putting something together for the PMCR text and try to take into account the constructive feedback that was put up here in the next days. I'll try to include a working demo of what I envision for the data.toml and so on as part of the PMCR.

mentioned in issue wiki#75