Automated GCC bisect for Linux kernels

Some downstream kernels don't boot anymore after being compiled with newer kernel versions.

@mberndt wrote in !132 (comment 131543422):

I have a crazy idea: we could do a kind of binary search by compiling one half of the kernel source files with gcc 6 and the other half with gcc 4.9.4 and then linking it all together – I think the object files are supposed to be compatible. If that results in a working kernel, we know the error must be in the part of the code compiled with gcc 4.9.4, otherwise we know it must be in the other half.

As @mirh found out, this technique was already used successfully.

So... while that would only help downstream kernels, which we want to migrate away from to mainline wherever possible, it would be a fun hack nevertheless. Maybe somebody is interested in looking into automating this.

Bonus points for pmbootstrap integration (pmbootstrap gcc-bisect fairphone-fp2 gcc6 gcc8?), and even more for automatically enabling Qemu kernel config options, confirming that the kernel boots in Qemu when compiled with the known working GCC, and then doing the bisect in a fully automated fashion without even flashing to the device.

added help wanted label

If we are really assuming qemu is enough to spot this kind of bugs, then I guess bisecting of gcc itself could also be doable (--plz-i-want-to-die mode). I mean, say for cases where you'd put more blame on it than your code. Or maybe, who knows, gcc6 and gcc8 struggle to link together.

I'm not really sure on how version branching of gcc works though.. It seems like they have a perennial master tree (and maybe it would be better to stick with this "monotonic" thing?), with a branch for every version (or maybe this would be more easygoing, assuming backports are trustworthy enough not to cause regressions themselves?)

p.s. check the post for bloopers

By mirh on 2019-01-18T13:45:10

I have written a program to do this and used it successfully to figure out why my HTC Desire Z won't boot when compiled with gcc > 4.9.4. It's quite ugly and hacky, so I want to polish it a bit more before publishing it, and it needs to be documented, otherwise it won't be of any help to anyone other than me.

Needless to say, it doesn't have any of the fancypants features that @ollieparanoid suggested. It only copies the object files around and launches the kernel build system to produce the zImage. But hey, it did the job.

By Matthias Berndt on 2019-02-07T23:42:47

@mberndt I've read about how you managed to find the compiler issue in !132 (merged) and I probably need to do something similar for the u-boot of the galaxy nexus which I tried to compile with both gcc8 and gcc6 but doesn't boot, while it boots successfully when compiled with gcc4.

In the next days I'll create a gcc4 package as @ollieparanoid suggested here and do you think we also need the gcc5?

Then we can start thinking about how to implement this "gcc-bisect" functionality.

So the main logic of the functionality would be to compile the same package with 2 different gcc version and keep the generated object files somewhere and then start copying some file over, link them, ask the user if it works, and then continue until just one or more file have been identified to cause the issue, right?

By Daniele Debernardi on 2019-02-11T11:43:27

Hey @drebrez,

I didn't get around to cleaning up my program yet, but I don't know when (or if) I will, so I might as well show you what I have, perhaps it helps you. It's quite hacky though, and I called it Frankenkernel.

https://gitlab.com/mberndt/gcc-bisect

It currently works only with the Linux kernel, because it makes some assumptions about the build system and its output. You need to have Stack installed.

There are three relevant subcommands: init, good and bad. You start by running ./frankenkernel.hs init. This will take a while when you first run it, because stack will download and compile a bunch of libraries I'm using.

init takes three parameters:

• the "good command", which is the shell command that you use to compile the kernel that works. In my case this was PATH=~/x-tools/arm-unknown-eabihf4.9.4/bin/:$PATH make ARCH=arm CROSS_COMPILE=arm-unknown-eabihf- -j3
• the "bad command", which is the shell command that you use to compile the kernel that doesn't work. In my case this was PATH=~/x-tools/arm-unknown-eabihf5.4/bin/:$PATH make ARCH=arm CROSS_COMPILE=arm-unknown-eabihf- -j3
• the "test command", which is the shell command that you use to compile the kernel that may or may not work, depending on the object files. In my case it was the same as the "bad command"

Frankenkernel will then set up build directories called build-good and build-bad and will compile a kernel in each of them by running the corresponding command with the suffix O=build-good or O=build-bad. It will parse the build output, looking for lines like CC foo/bar/baz.o to obtain a list of the object files (I could use the find command or similar, but that would find some .o files generated by HOSTCC, which I'm not interested in). Then it copies build-bad to build-test and the init phase is done.

Now, do the following:

• test the kernel in the build-test directory
• if it works, run ./frankenkernel good, otherwise run ./frankenkernel.hs bad. Frankenkernel will then copy a bunch of object files from either build-good or build-bad to the build-test directory and launch the kernel build system to link the whole thing together.
• repeat this until Frankenkernel tells you that you're done. It will then print a list of the files that are miscompiled by the new compiler.

If you made a mistake, you can start over by simply running ./frankenkernel.hs init again.

Note that after the init command, you basically always need to run bad next, because at that stage the build-test is just a copy of build-bad, which we know is broken. This should probably be done automatically…

By Matthias Berndt on 2019-02-11T20:02:13

@mberndt thanks for sharing the script, I'm not very familiar with haskell syntax but I will take a look more in details in the next days probably.

In the meantime I've packaged the gcc4 and the gcc4-armhf cross-compiler.

I'm thinking about how to implement this functionality in pmbootstrap, maybe we can just simply package your script and use it as a wrapper in the chroot instead of re-implementig the whole logic in python. @ollieparanoid what you think about it?

By Daniele Debernardi on 2019-02-11T23:58:16

Closing issue due to inactivity.

By Robert Eckelmann on 2024-08-14T13:33:53

closed