HTC One M8 Progress/Help

Created by: ollieparanoid

@fooforever: Thanks for creating this detailed issue!

@osm0sis: you have quite some knowledge about all the boot.img formats. Maybe you can answer these questions if you have a minute:

Is there anything special about the HTC devices (the One M8 specifically), that we need to look out for regarding boot.img and dtb files?
Looks like the extract-dtb tool we use can't handle extracting these dtb files. Can you recommend us a tool (or could author @PabloCastellano take a look at this case and try to fix it)?

By * postmarketOS Bot on 2018-02-05T20:41:19

Judging from the most recent here: https://download.lineageos.org/m8, seems like a normal QCDT to me. AIK with repackimg.sh --original repacks an identical image. That's using my unpackbootimg and mkbootimg, of course.

By osm0sis on 2018-02-05T21:54:17

Created by: ollieparanoid

@fooforever: Please double-check if you have QCDT enabled.

If you have it enabled, and get the same behavior, you could try @osm0sis' mkbootimg implementation (it will be the default soon, see #1193):

Check out the branch feature/mkbootimg
Clean your chroots: pmbootstrap zap
Run pmbootstrap install again
Try to flash the kernel again

By * postmarketOS Bot on 2018-02-05T22:11:37

Created by: ollieparanoid

Uh right, the branch is not part of the main repository. This should work:

cd ~/code/pmbootstrap
git remote add -f lramage94 https://github.com/lramage94/pmbootstrap
git checkout feature/mkbootimg

or if that fails:

cd ~/code/pmbootstrap
git checkout master
wget https://github.com/postmarketOS/pmbootstrap/pull/1193.patch
patch -p1 < 1193.patch

By * postmarketOS Bot on 2018-02-05T22:21:41

Created by: fooforever

I'm still getting the same dtb not found issue when trying to do flasher boot and still ust rebooting to fastboot after flasher flash_kernel and a reboot. I ran zap, rebuilt device and linux, then ran install. So there's something else going on here.

By * postmarketOS Bot on 2018-02-05T22:43:28

Created by: ollieparanoid

Just to be 100% sure that the DTB file has been attached to the kernel image, could you run pmbootstrap bootimg_analyze on the boot.img file generated by postmarketOS? (Get a shortcut to it with pmbootstrap export.)

It should give you exactly the same output as running the command on the known working boot.img file.

By * postmarketOS Bot on 2018-02-06T19:55:38

Created by: ollieparanoid

After asking @drebrez, who successfully ported the HTC desire, we have some new information: https://wiki.postmarketos.org/wiki/Troubleshooting:boot#Note_about_HTC_devices

By * postmarketOS Bot on 2018-02-06T23:15:05

Created by: fooforever

Just got chance to have a look at this, pulled the up to date master first then zapped and rebuild everything and I'm still having the same problem, also checking the generated boot image against the twrp one with bootimg_analyze gives the same output for both.

By * postmarketOS Bot on 2018-02-25T17:34:50

Created by: ollieparanoid

Summary:

after flashing it reboots right into fastboot
pmbootstrap flasher boot: "FAILED (remote: dtb not found)"
osm0sis' mkbootimg (which is the default now) had the same result
bootimg_analyze gives the same output for both boot.img files
extract-dtb fails to extract the files

Some thoughts:

After reading the extract-dtb error again: It tries to extract files to a subfolder which does not exist. Pull request https://github.com/PabloCastellano/extract-dtb/pull/4 will fix this, here is the raw extract-dtb.py script which you could download and use as standalone until we have properly integrated this change back into postmarketOS. I would be interested what happens if you run this version on both boot.img files, the one from pmOS and the one from TWRP - I guess the DTB files are different? (and there are a lot of them, a recursive diff will help with comparing them)
Not that helpful for debugging, but to work around this issue, it should be possible to use abootimg to put the postmarketOS and kernel (from pmbootstrap export) into the known working boot.img file.

By * postmarketOS Bot on 2018-02-25T19:00:56

Created by: fooforever

So some results, still no apparently compatible image. But I've run the extract dtb script over everything and this is what I found, file listings for all the extractions here: https://pastebin.com/i3aW2q8v. pmbootstrat seems to generate a similar but not identical dtb to the twrp image, it has a few less file and things are name/numbered differently(maybe because there are files missing?) However when I do an aboot -u twrp.img -k /tmp/postmarketOS-export/vmlinuz-htc-m8 -r /tmp/postmarketOS-export/initramfs-htc-m8 the dtbs seem to be concatenated or at least both are included without duplicated because there are 51 files vs 36 in the twrp image and 30 in the pmos image.

By * postmarketOS Bot on 2018-02-27T19:22:17

Created by: fooforever

Ok so I found there is a dmesg function the the HTC fastboot commands fastboot oem dmesg. After installing the pmOS image via ADB sideload from TWRP I rebooted to try and load the image but predictably got dropped back into fastboot mode. Then I ran dmesg, output here: https://pastebin.com/erKTAw92 This seems to confirm the dtb problems, with the messages stamped [3387014] to [3392359] spewing out dtb output and errors about not finding a valid device tree address. This must mean the dtb is being appended to the boot image incorrectly in some way.

By * postmarketOS Bot on 2018-02-27T19:38:04

Created by: ollieparanoid

Cool trick with the dmesg output and thanks for the detailed analysis of the dtb files.

However when I do an aboot -u twrp.img -k /tmp/postmarketOS-export/vmlinuz-htc-m8 -r /tmp/postmarketOS-export/initramfs-htc-m8 the dtbs seem to be concatenated or at least both are included without duplicated because there are 51 files vs 36 in the twrp image and 30 in the pmos image.

This would mean the vmlinuz file has appended dtbs... is that the case?

For the dtb files not matching up, I have two explanations:

The kernel is different, and produces different dtb files (I'd like to run some more tests before we can rule that out, see below).
Our tooling is broken and doesn't append the right dtbs to the kernel, although we are using the same source.

I'm pretty sure that the kernel sources are different when using the TWRP boot.img, but the LineageOS sources. I've tried to find the kernel sources, that the TWRP image used, but it only says "Builds with OmniROM android-5.0 tree" here and the omirom kernel repository doesn't seem to be there anymore. At least not for Android 5.0 when looking into the branches.

But we have to source to the LineageOS 14.1 kernel, which you are using. And here is a zip file, which should have a boot.img file inside, generated from these kernel sources. I hope that the following helps with finding out more:

Can your device boot that lineageos image?
How does the output differ when using extract-dtb on that?
Does the abootimg approach work with that one?

I think we're getting closer, thanks for not giving up

By * postmarketOS Bot on 2018-02-27T22:15:16

Created by: fooforever

The plot thickens, I pulled the lineage 15 build from here dated 18/2 which according to the thread matches extactly the commit I'm pulling sources from. This boots on the device no problem. I can't seem to repack the kernel and ramdisk from pmos into the lineage boot.img because they are too big. BUT Looking the the dtb extracts from the lineage boot image and the kernel generated by pmbootstrap, they're almost identical! take a look. So Ollie you were right the kernel does have a dtb already appended to it. The only thing that is different between these two dtb extracts is that number 1-5 should be 11-15 and vice versa. So the question now is how do I stop all the other dtb files from being generated and potentially messing things up? And what determines the numbering of the files in the dtb? and do they actually matter?

By * postmarketOS Bot on 2018-02-28T22:20:38

Created by: ollieparanoid

So the question now is how do I stop all the other dtb files from being generated and potentially messing things up?

That can all be adjusted in the kernel's aport:

boot.img dtbs

The dtbs that get appended to the boot.img file get placed in /boot/dt.img (that's the location after you install the kernel package). And they get generated here (should be somewhere in your kernel APKBUILD):

dtbTool -s 2048 -p "scripts/dtc/" -o "arch/arm/boot/dt.img" "arch/arm/boot/"

vmlinuz dtbs

The dtbs appended to the vmlinuz file can be configured in the menuconfig I think. From what I know, first the regular vmlinuz file gets created, and afterwards the dtbs get appended and the file gets renamed. Also in the linux APKBUILD there's a function that tries to find the correct vmlinuz file, and prefers ones with dtb files appended (because that is usually what we want). You could hardcode the file to take at this point:

    # zImage (find the right one)
    cd "$builddir/arch/$_carch/boot"
    _target="$pkgdir/boot/vmlinuz-$_flavor"
    for _zimg in zImage-dtb Image.gz-dtb *zImage Image; do
        [ -e "$_zimg" ] || continue
        msg "zImage found: $_zimg"
        install -Dm644 "$_zimg" "$_target"
        break
    done
    if ! [ -e "$_target" ]; then
        error "Could not find zImage in $PWD!"
        return 1
    fi

And what determines the numbering of the files in the dtb?

Probably the naming of the files and/or folders on disk. Ideas:

you could add commands right before executing dtbTool that moves the files so they get packed in the right order
the dtbTool program used for building that is just a Python script, so you could hardcode the order there for testing

and do they actually matter?

I hope they do, since that's the best lead we have right now

By * postmarketOS Bot on 2018-02-28T23:43:03

Created by: fooforever

I disabled packing the dtb into there kernel image by unsetting Build a concatenated zImage/dtb by default in the menuconfig. This gets me a kernel without the dtb appended as expected, which makes the extracts look almost like they should except for the numbering as mentioned above. This image still won't boot because of dtb problems and neither does the pmos kernel repacked into the lineage image. But it seems like the pmos image does better than the repacked one(dmesg. I don't know what an actual boot should look like because I can't capture the bootloader dmesg because it actually boots android and I can only get a linux dmesg. Do you think it could be the numbering? Is there anyone else around here that might know more about boot images?

By * postmarketOS Bot on 2018-03-01T13:22:09

Created by: ollieparanoid

Another idea for debugging would be copying the dt.img file from the extracted LineageOS boot.img to ~/.local/var/pmbootstrap/chroot_device-your-device/boot/dt.img (after pmbootstrap install). Then you can generate a new boot.img with pmbootstrap mkinitfs (better verify that it used the right dt.img file by extracting the resulting boot.img from postmarketOS again). Maybe that gets it booting as a stop-gap solution.

If that doesn't get you anywhere, in #postmarketOS-lowlevel you might get more help by other people (e.g. McBitter, not tagging you so you don't need to read the whole thread now if the solution above is possibly helping). (And I'm still around to help out with more ideas of course, but I'm not an expert on the subject.)

By * postmarketOS Bot on 2018-03-01T20:54:14

Created by: fooforever

If I change the dt.img with the one from the lineage boot (as above suggested) I do get a boot image that appears to get further than before. So I'm not not being dropped straight back into fastboot, but I get stuck on the boot screen. I don't get any network device show up on the pc though. Although it does re appear as a new usb device (id 0bb4:0f25) which doesn't seem to match when its in fastboot or recovery modes (with mtp and mass storage enabled, disabled). I did have to go round about to get the system image on the device (process below) so I'm not sure if that will have messed with things. This is because pmbootstrap refused to build the flashable zip after swapping out the dt.img ( I'm guessing the chroot is checksummed somewhere or something).

To get this far:

build everything as normal
make flashable zip (can't flash system from fastboot)
adb sideload system via the flashable zip
swap out the dt.img file in the chroot for the lineage os boot.img one
pmbootstrap initfs to rebuild boot.img with new dt.img
flash new boot.img

By * postmarketOS Bot on 2018-03-01T22:50:49

Created by: ollieparanoid

Hey, that is great news! Now we know that the problem is dtbTool not producing not the right dt.img file!

Regarding the networking issue, see: https://wiki.postmarketos.org/wiki/Troubleshooting:boot#Kernel_boots_without_network_interface

Here are instructions on how to use an alternative dtb tool, for reference (I'd recommend to fix networking issue first): https://wiki.postmarketos.org/wiki/Dtbtool

@MoreRobustThanYou: Do you know how to contact Skales, the author of the dtbtool we're currently using to hell him about this bug?

By * postmarketOS Bot on 2018-03-03T21:46:27

Created by: steamp0rt

@ollieparanoid no i don't know anything about how to contact skales. all i know is CAF is involved somehow

By * postmarketOS Bot on 2018-03-03T22:14:26

Created by: fooforever

No luck with the lineage dtbtool either

By * postmarketOS Bot on 2018-03-14T20:31:10

Created by: ollieparanoid

Heh, I thought skales is a person. But it's just the name of the repository, and Stephen Boyd (@bebarino) made the first commit and it seems like he's maintaining it.

@bebarino: Sorry to drag you in from the side, but we could need your advice here if you have a minute. Here's a quick summary:

@fooforever is porting the HTC One M8 to postmarketOS
This is a QCDT device, which means we use the dtbTool from the skales repository to generate a dt.img file
We add the dt.img to the boot.img (that step was double checked and works)
The device does not boot up with our boot.img file.
When extracting the generated dt.img file, and comparing it to the one from LineageOS (who build their kernel from the same source), they are slightly different:

Lineage OS boot image dtb extract
 00_kernel                 06_dtbdump_M8_WHL.dtb          12_dtbdump_M8_XB
 01_dtbdump_M8_WL.dtb      07_dtbdump_M8_WHL_XB           13_dtbdump_M8_XD.dtb
 02_dtbdump_M8_WL_XB.dtb   08_dtbdump_M8_WHL_XD.dtb       14_dtbdump_M8_XE.dtb
 03_dtbdump_M8_WL_XC.dtb   09_dtbdump_M8_WHL_XE.dtb       15_dtbdump_M8_XF.dtb
 04_dtbdump_M8_WL_XD.dtb   10_dtbdump_M8_WHL_XF.dtb
 05_dtbdump_M8_WL_XE.dtb  '11_dtbdump_!htc,msm8974.dtb'
 
Our boot.img dtb extract (dtb.img generated with skales dtbTool)
 00_kernel                      06_dtbdump_M8_WHL.dtb      12_dtbdump_M8_WL_XB.dtb
'01_dtbdump_!htc,msm8974.dtb'   07_dtbdump_M8_WHL_XB       13_dtbdump_M8_WL_XC.dtb
 02_dtbdump_M8_XB               08_dtbdump_M8_WHL_XD.dtb   14_dtbdump_M8_WL_XD.dtb
 03_dtbdump_M8_XD.dtb           09_dtbdump_M8_WHL_XE.dtb   15_dtbdump_M8_WL_XE.dtb
 04_dtbdump_M8_XE.dtb           10_dtbdump_M8_WHL_XF.dtb
 05_dtbdump_M8_XF.dtb           11_dtbdump_M8_WL.dtb

The only thing that is different between these two dtb extracts is that number 1-5 should be 11-15 and vice versa.

So it looks like a bug in the dtbTool, which produces the files in the wrong order. @bebarino: What do you think about that, and do you have any advice for proceeding further?

@fooforever: I hope everything was summarized right here, please correct me if necessary.

By * postmarketOS Bot on 2018-03-14T22:40:46

Created by: steamp0rt

try --version=1

By * postmarketOS Bot on 2018-03-14T22:43:07

Created by: bebarino

Dtbtool from skales does a sort of all the IDs in the QCDT blob and then writes them out in a sequence. If I read this right it sounds like that sort is wrong? Is there a difference in how the sorting is done between the two versions of dtbtool?

By * postmarketOS Bot on 2018-03-15T04:20:27

Created by: fooforever

@bebarino You sound like you might know more about this than me, does the numbering even matter? And if it does where are the IDs defined, maybe I can track down the issue

By * postmarketOS Bot on 2018-03-27T20:51:30

Created by: bebarino

I was thinking that the qcom,msm-id and qcom,board-id and qcom,pmic-id properties were being improperly sorted somehow and that's causing the QCDT image to have the wrong order of dtb blobs.

I believe the order matters because those properties in each dtb blob are used to sort the blobs into a "priority" order that the bootloader can then run through in order to find the closest match to the hardware that the boot.img is running on without requiring the bootloader to sort the blobs in memory at boot.

Can you extract all those IDs and then list them next to the file names for the dtbs? That would be useful to see what sort of ordering is happening and if that is somehow broken.

On Tue, Mar 27, 2018, 1:51 PM Nick Boone notifications@github.com wrote:

@bebarino https://github.com/bebarino You sound like you might know more about this than me, does the numbering even matter? And if it does where are the IDs defined, maybe I can track down the issue

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/postmarketOS/pmbootstrap/issues/1206#issuecomment-376671056, or mute the thread https://github.com/notifications/unsubscribe-auth/AACXsARD_3DZz_zZqYukWRBCB9kwEhxBks5tiqZUgaJpZM4R4wRy .

By * postmarketOS Bot on 2018-03-28T05:02:55

Created by: fooforever

I've extracted all the individual DTBs intso DTSs the extracted all the IDs. There's 2 id fields qcom,msm-id and htc,project-id. The qcom,msm-id is exactly the same for every file, so it must be sorting on the htc,project-id field. There does seem to be something weird going on with the sort on the pmos extracted ones. The lineage dtb extracts are sorted in ascending order by the htc,project-id field but, the pmos ones don't seem to be. They have the same or very close IDs but are sorted incorrectly. Dropped the data into a spreadsheet & csv so you can have a look.

ID Coparisons.tar.gz

By * postmarketOS Bot on 2018-03-28T20:38:56

Created by: bebarino

The mkbootimg in skales isn't looking at an htc,project-id property so maybe that needs to be added into the sorting? I've never heard of that property, but if the bootloader looks for it then it may need to be used too.

On Thu, Mar 29, 2018, 4:39 AM Nick Boone notifications@github.com wrote:

I've extracted all the individual DTBs intso DTSs the extracted all the IDs. There's 2 id fields qcom,msm-id and htc,project-id. The qcom,msm-id is exactly the same for every file, so it must be sorting on the htc,project-id field. There does seem to be something weird going on with the sort on the pmos extracted ones. The lineage dtb extracts are sorted in ascending order by the htc,project-id field but, the pmos ones don't seem to be. They have the same or very close IDs but are sorted incorrectly. Dropped the data into a spreadsheet & csv so you can have a look.

ID Coparisons.tar.gz https://github.com/postmarketOS/pmbootstrap/files/1857660/ID.Coparisons.tar.gz

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/postmarketOS/pmbootstrap/issues/1206#issuecomment-377028238, or mute the thread https://github.com/notifications/unsubscribe-auth/AACXsKF-i-8JbbjZK8OZOIWXCompFE4Fks5ti_TogaJpZM4R4wRy .

By * postmarketOS Bot on 2018-03-29T10:25:48

Created by: fooforever

What's the fallback sorting method then? I've just double checked and the qcom id is exactly the same in every file

By * postmarketOS Bot on 2018-03-29T10:56:21

Created by: fooforever

@ollieparanoid Just seen the referenced PR, would M8 be suitable for merge too? I'm using the same trick, but was waiting to get a booting kernel right out of pmbootstrap if we found a fix.

By * postmarketOS Bot on 2018-04-03T16:43:01

Created by: ollieparanoid

@fooforever: yep, that would be suitable. Just make sure to mention in the documentation how to work around the issue (actually that information was missing from the htc-a5ul page and I've just added it - good that you mentioned it!)

@bebarino: so the htc-a5ul has the same issue, see the diff @Aerotinge posted in https://github.com/postmarketOS/pmbootstrap/pull/1378#issue-178547080:

# Compare the appended dtb files

/tmp/extract-dtb-pmOS # cp /boot/boot.img-htc-a5ul /tmp/extract-dtb-pmOS/boot.img
/tmp/extract-dtb-pmOS # extract-dtb boot.img
Dumped 00_kernel, start=0 end=8269
Dumped 01_dtbdump_`<@M@0q__.dtb, start=8269 end=10297344
Dumped 02_dtbdump_MSM_8926_A5UL.dtb, start=10297344 end=10485760
Dumped 03_dtbdump_MSM_8x26_A5_China.dtb, start=10485760 end=10674176
Dumped 04_dtbdump_MSM_8926_A5DWGL.dtb, start=10674176 end=10862592
Extracted 4 appended dtbs + kernel to dtb

/tmp/extract-dtb-pmOS # cd /tmp/extract-dtb-known-working
/tmp/extract-dtb-known-working # extract-dtb boot.img
Dumped 00_kernel, start=0 end=8241
Dumped 01_dtbdump_`<@M@0__.dtb, start=8241 end=10575872
Dumped 02_dtbdump_MSM_8x26_A5_China.dtb, start=10575872 end=10764288
Dumped 03_dtbdump_MSM_8926_A5UL.dtb, start=10764288 end=10952704
Dumped 04_dtbdump_MSM_8926_A5DWGL.dtb, start=10952704 end=16777216
Extracted 4 appended dtbs + kernel to dtb

(A5_China.dtb and A5UL.dtb are switched)

What's the best path to fix this upstream? Thank you very much for your time!

By * postmarketOS Bot on 2018-04-04T00:22:01

mentioned in commit wiki@1ec55091

By postmarketOS Wiki bot on 2018-08-28T20:42:33

moved to pmaports#63 (closed)

By Bart Ribbers on 2018-09-17T22:04:14

HTC One M8 Progress/Help

Child items ...

Activity

boot.img dtbs

vmlinuz dtbs