Add support to different deviceinfo variables in mainline/downstream subpackages

TODO:

• Test whole workflow: from pmbootstrap init, kernel selection, flashing to device and verify

Changes:

• New version of devicepkg-dev with changes described in https://github.com/postmarketOS/pmbootstrap/issues/1368#issuecomment-387809557
• Added the logic suggested by @ollieparanoid (https://gitlab.com/postmarketOS/pmbootstrap/issues/1368#note_85245801)

...always installs the deviceinfo file, but the subpackage then clears it from the package in case it will get installed in the kernel subpackages

• It's not limited to the kernel_cmdline but works with every variable
• Added a validation check: if there is a variable with the kernel subpackage suffix, it should not exist the same variable without the prefix

How to test:

pmbootstrap -y zap
pmbootstrap build devicepkg-dev --force --arch armhf

# Test device with both cmdlines
pmbootstrap build device-samsung-klte --force --arch armhf

pmbootstrap chroot -b armhf -- find /home/pmos/build/pkg/
# output should contain the deviceinfo file in both subpackages and not in the device package

pmbootstrap chroot -b armhf -- cat /home/pmos/build/pkg/device-samsung-klte-kernel-downstream/etc/deviceinfo
# output should contain the lines:
#    deviceinfo_kernel_cmdline="androidboot.boot_recovery=1"
#    deviceinfo_bootimg_qcdt="true"

pmbootstrap chroot -b armhf -- cat /home/pmos/build/pkg/device-samsung-klte-kernel-mainline/etc/deviceinfo
# output should contain the lines: 
#    deviceinfo_kernel_cmdline="console=ttyMSM0,115200,n8 PMOS_NO_OUTPUT_REDIRECT"
#    deviceinfo_bootimg_qcdt="false"


# Test device with normal cmdline
pmbootstrap build device-samsung-maguro --force --arch armhf

pmbootstrap chroot -b armhf -- find /home/pmos/build/pkg/
# output should contain the deviceinfo file in the device package as usual

pmbootstrap chroot -b armhf -- diff -s /home/pmos/build/pkg/device-samsung-maguro/etc/deviceinfo /home/pmos/build/src/deviceinfo
# output should contain the exact content as the source file


# Test device without cmdline
pmbootstrap build device-samsung-i9070 --force --arch armhf

pmbootstrap chroot -b armhf -- find /home/pmos/build/pkg/
# output should contain the deviceinfo file in the device package as usual

pmbootstrap chroot -b armhf -- diff -s /home/pmos/build/pkg/device-samsung-i9070/etc/deviceinfo /home/pmos/build/src/deviceinfo
# output should contain the exact content as the source file

Closes: https://gitlab.com/postmarketOS/pmbootstrap/issues/1368

added 1 commit

• a825d47e - Fix shellcheck errors

Compare with previous version

By Daniele Debernardi on 2018-09-29T20:41:12

added 1 commit

• 559a8c73 - Bump versions

Compare with previous version

By Daniele Debernardi on 2018-09-29T20:49:54

I think it would be a good idea to add an automatic test for devicepkg_subpackage_kernel.sh inside this package. Maybe create a "testdata" folder with example inputs and output, then run devicepkg_subpackage_kernel.sh on it and compare with diff if the output matches. that could be done in check(). Would you like to do that?

Thanks a lot for the MR @drebrez!

By Oliver Smith on 2018-10-02T05:23:05

added 1 commit

• 271f4bf2 - Add comments

Compare with previous version

By Daniele Debernardi on 2018-10-02T20:47:34

@ollieparanoid I've added some comments like you suggested.

I will look at the automatic test in the next days.

By Daniele Debernardi on 2018-10-02T23:44:14

added 23 commits

• 271f4bf2...f490d353 - 19 commits from branch master
• 19b8c47c - Add support to different deviceinfo variables in mainline/downstream subpackages
• b2b4a137 - Fix shellcheck errors
• 67135e6b - Bump versions
• 091c92fd - Add comments

Compare with previous version

By Daniele Debernardi on 2018-10-04T21:14:00

While working on the tests I found some issues, today I will try to fix them and rebase

By Daniele Debernardi on 2018-10-09T08:19:21

added 1 commit

• 8cfe5c64 - Fix script and add tests

Compare with previous version

By Daniele Debernardi on 2018-10-09T20:13:29

changed the description

By Daniele Debernardi on 2018-10-09T20:42:09

changed the description

By Daniele Debernardi on 2018-10-09T20:46:33

added 15 commits

• 8cfe5c64...50ac7a4d - 9 commits from branch master
• 0ac97052 - Add support to different deviceinfo variables in mainline/downstream subpackages
• 12e343b2 - Fix shellcheck errors
• b3a83164 - Bump versions
• 597f3aed - Add comments
• 30ce9422 - Fix script and add tests
• 3ca0181d - Bump version and fix mistake during rebase

Compare with previous version

By Daniele Debernardi on 2018-10-09T20:47:25

@ollieparanoid I've added the check function with a test case, rebased on master, updated the how to test section.

The output deviceinfo of the downstream kernel is exactly like it was before so I guess that it still works. For the mainline deviceinfo I changed to use the non QCDT version and the cmdline with the serial output but I don't have my linux laptop with me right now and cannot test flashing it to the device (I'm working with a VM right now).

Nonetheless it seems that the devicepkg_subpackage_kernel.sh script works as expected

By Daniele Debernardi on 2018-10-09T21:05:19

resolved all discussions

By Oliver Smith on 2018-10-11T04:18:18

Awesome work, code looks good!

Let's merge this once you've confirmed that it's booting

By Oliver Smith on 2018-10-11T04:20:24

approved this merge request

By Oliver Smith on 2018-10-11T04:20:32

added 6 commits

• 3ca0181d...f5f34402 - 5 commits from branch master
• d29bcee1 - Merge branch 'master' into feature/devicepkg-support-different-variables

Compare with previous version

By Daniele Debernardi on 2018-10-11T22:12:43

@ollieparanoid I tried with the downstream kernel with weston and everything work,

but with the mainline kernel I have some trouble:

• with deviceinfo_dtb="qcom-msm8974-samsung-klte" and deviceinfo_bootimg_qcdt="false" the bootloader doesn't start the kernel anymore (it should append the dtb to the kernel and create the boot.img without the dt.img), I was doing this manually some time ago and was working, I will investigate
• with deviceinfo_dtb="qcom-msm8974-samsung-klte" and deviceinfo_bootimg_qcdt="true" it's even worse, I think that our mkinitfs script will combine both (dtb appended to the kernel and dt.img appended to the boot.img). This combination is probably something that nobody will ever need and we can probably handle it directly in postmarketos-mkinitfs (if qcdt = true then use kernel and not kernel-dtb?). This is the same combination as it is now with the downstream kernel but the vmlinuz-dtb file is just a copy of the normal kernel (https://gitlab.postmarketos.org/postmarketos/pmaports/blob/master/main/postmarketos-mkinitfs/mkinitfs.sh#L360)
• with deviceinfo_dtb="" and deviceinfo_bootimg_qcdt="true" it boots but it fails with

Mount boot partition (/dev/dm-0)
ERROR: initramfs-extra not found!

and that's strange because adding the debug-shell and doing the commands manually the initramfs-extra is there and the extraction works fine. So probably this is something that have to be fixed in the kernel itself (I suppose some regulator voltage in the dtb)

If you want you can merge it, since the issues I'm having are not related to the devicepkg-dev features added with this MR and I can create new MR once I understood what's exactly happening with the mainline kernel.

By Daniele Debernardi on 2018-10-12T00:17:59

I did some other tests and discovered that the bootloader on the Samsung S5 only accepts the dtb appended to the boot.img (deviceinfo_dtb="" and deviceinfo_bootimg_qcdt="true") when there are the msm-id and board-id properties (see commit https://gitlab.com/postmarketOS/linux-postmarketos/commit/09efb4857f1a4119eced855912043817bd96e064).

If I remove those 2 lines the bootloader allows to boot the kernel with the dtb appended to the kernel itself (with deviceinfo_dtb="qcom-msm8974-samsung-klte" and deviceinfo_bootimg_qcdt="false").

By Daniele Debernardi on 2019-05-22T22:25:24

Sorry for necro-posting, what do you think should be the correct way to go?

• Keep qcdt=true deviceinfo_dtb="" and store msm-id, board-id in dts?
• Or remove msm-id, board-id, set qcdt=false and deviceinfo_dtb to a valid file?

By Alexey Min on 2019-05-22T22:25:24

@minlexx I personally prefer the qcdt=false without the msm-id/board-id because it's less dependent on the bootloader, maybe there are some model variants with different ids and that means updating the devicetree?

The dtb appended to the kernel itself is the solution implemented by the kernel to remedy the problem of old bootloaders who were not aware of these devicetree, I suppose.

By Daniele Debernardi on 2019-05-23T17:18:25

unmarked as a Work In Progress

By Oliver Smith on 2018-10-15T19:52:41

approved this merge request

By Oliver Smith on 2018-10-15T19:52:52

Code looks good to me. Thank you very much @drebrez! And it's impressive that you have figured out why your S5 would not accept the dtb appended to the boot.img.

By Oliver Smith on 2018-10-15T19:54:08

merged

By Oliver Smith on 2018-10-15T19:54:16

mentioned in merge request !226 (merged)

By Daniele Debernardi on 2019-02-19T22:24:49

mentioned in issue #260 (closed)

By Zhuowei Zhang on 2019-05-17T06:55:33

mentioned in merge request pmbootstrap!2043 (merged)

By Luca Weiss on 2021-03-31T11:54:36