Adjust to GCC8 upgrade in Alpine edge

mentioned in merge request !4 (merged)

By Oliver Smith on 2018-09-26T06:43:31

GCC has been upgraded for all arches now. I'm not entirely sure if we have to rebuild all our packages that required GCC to build, does anyone know? I assume most stuff will compile fine (I know for a fact Phosh and it's dependencies did) except most of the kernels.

By Bart Ribbers on 2018-09-26T19:25:09

I've started the branch feature/gcc8 today which currently contains regenerated gcc and binutils aports and a commit that renamed the compiler-gcc6.h file to compiler-gcc8.h in the FP2 kernel and adjusts the devicepkg-dev package to that filename. Sadly the kernel does not compile due to very helpful errors

/tmp/ccCNJBpc.s: Assembler messages:
/tmp/ccCNJBpc.s:678: Error: .err encountered
/tmp/ccCNJBpc.s:728: Error: .err encountered
/tmp/ccCNJBpc.s:1101: Error: .err encountered
/tmp/ccCNJBpc.s:8200: Error: .err encountered

By Luca Weiss on 2018-09-26T22:15:19

GCC has been upgraded for all arches now. I'm not entirely sure if we have to rebuild all our packages that required GCC to build, does anyone know?

I think it would be better to to that now than finding out that the packages don't compile with GCC8 as soon as we are trying to update them.

I'll be online later today (in 8-9 hours) to work on that.

@z3ntu: does the patch from here fix this? https://lkml.org/lkml/2018/7/26/187

(In general, if it's not much extra effort, it would be nice if we could turn compiler-gcc*.h into a regular .patch file now that we need to touch it everywhere anyway.)

By Oliver Smith on 2018-09-27T07:21:03

Update: The patch (adjusted) fixes the compilation but the kernel doesn't boot and I even with earlyprintk I didn't see any output out of the serial connection so I'm saying "it doesn't work".

By Luca Weiss on 2018-09-27T12:22:34

Maybe this has something to do with it:

[2402860] booting linux @ 0x8000, ramdisk @ 0x1000000 (1278235), tags/device tree @ 0x100
[2402870] undefined abort, halting
[2402870] r0  0x00000000 r1  0x00000000 r2  0x00000100 r3  0x00008000
[2402870] r4  0x11000000 r5  0x00000100 r6  0x00008000 r7  0x00000000
[2402870] r8  0x0073a800 r9  0x0f99dd00 r10 0x0f9297cc r11 0x00000004
[2402870] r12 0x60000193 usp 0x00000000 ulr 0x00000000 pc  0x00008000
[2402870] spsr 0x20000193

Thing is, it actually just appears when doing fastboot boot and not when booting the flashed kernel.

By Luca Weiss on 2018-09-27T12:27:52

mentioned in commit 8cd9e694

By Oliver Smith on 2018-09-27T17:19:42

After throughout research, I can't find any patch or even any mention of the error message you've posted.

It would be nice if we could all kernels work with GCC8 in theory. But when it starts with something like that, it doesn't seem feasible, and it doesn't really seem worth wasting a lot of time on this.

So I'm going the route of keeping gcc6 around, exclusively for the legacy vendor kernels.

TODO:

nice to have:

test case: gcc6 packages must have the same version
test case: linux aports that depend on gcc6 should have the same HOSTCC=... lines (see below)

By Oliver Smith on 2018-10-05T07:29:48

I just noticed that Linux 3.16 is still being maintained over at kernel.org. Maybe a relevant patch can be found here; however I did not find one as I looked through the results. So unless somebody is able to fix the error @z3ntu reported, we'll need to go with the GCC6 aport solution.

By Oliver Smith on 2018-09-27T21:02:25

I tried the downstream kernel (3.1) for samsung-p4wifi and it boots postmarketOS. I had to apply the two patches below.

https://github.com/Decatf/android_kernel_samsung_p4/commit/8c8187d41b99acd4f2078d0fff6807e0eeb47407
https://github.com/Decatf/android_kernel_samsung_p4/commit/a7bfe0a2832e66f3d732e4210b5d53c81cc7a85f

I don't have serial for this device so I don't know if it even had the boot loader issues mentioned above

I haven't used this downstream kernel a long time since there is a mainline(-ish) kernel for this device. I used this kernel for Android and had it compiling on GCC 5.x (iirc) before I stopped using it. There's more GCC related patches in the commit history but I don't remember all the details. The branch is here:
https://github.com/Decatf/android_kernel_samsung_p4/tree/aosp-ng-gcc8

Only other patch I can remember is this
https://github.com/Decatf/android_kernel_samsung_p4/commit/bd0772df2132bf56cda78e7204facf434cd5aa40

Which was a really messy merge of this:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=cb984d101b30eb7478d32df56a0023e4603cba7f

By ryang on 2018-09-28T01:11:33

Thank you very much @ryang2678!

bd0772df2132bf56cda78e7204facf434cd5aa40 sounds like a good solution to getting rid of the compiler-N.h files. But it is unlikely that this would resolve the boot problem, right?

By Oliver Smith on 2018-09-28T05:33:01

mentioned in commit f667bebe

By Oliver Smith on 2018-09-28T06:16:52

mentioned in commit eae15aec

By Oliver Smith on 2018-09-28T06:24:12

mentioned in merge request !24 (merged)

By Oliver Smith on 2018-09-28T06:24:44

mentioned in commit b53afb3a

By Oliver Smith on 2018-09-28T06:26:01

I'm making good progress with adding support for GCC6, see the checkboxes above. The pmbootstrap aportgen gcc6-armhf etc. code lives in the feature/aportgen-gcc6 branch right now, and the changes necessary were rather simple. It's still WIP though, the resulting gcc6-armhf package is conflicting with gcc6 right now:

ERROR: gcc6-armhf-6.4.0-r0: trying to overwrite usr/bin/gcc6 owned by gcc6-6.4.0-r0.
ERROR: gcc6-armhf-6.4.0-r0: trying to overwrite usr/gcc6/lib/libcc1.so owned by gcc6-6.4.0-r0.
ERROR: gcc6-armhf-6.4.0-r0: trying to overwrite usr/gcc6/lib/libcc1.so.0 owned by gcc6-6.4.0-r0.
ERROR: gcc6-armhf-6.4.0-r0: trying to overwrite usr/gcc6/lib/libcc1.so.0.0.0 owned by gcc6-6.4.0-r0.
ERROR: gcc6-armhf-6.4.0-r0: trying to overwrite usr/gcc6/bin/gcc6 owned by gcc6-6.4.0-r0.
ERROR: gcc6-armhf-6.4.0-r0: trying to overwrite usr/gcc6/bin/cc owned by gcc6-6.4.0-r0.

I'll continue working on this as soon as I find time for it. If somebody else wants to play with this in the meantime, take a look at the TODOs and go ahead!

By Oliver Smith on 2018-09-28T07:17:56

Little status update, MRs coming as soon as this works.

Cross compilers can be generated now, they compile and they don't conflict.
gcc6 doesn't get installed to usr/bin/gcc6 anymore. Instead, all gcc6 programs get symlinked to /usr/bin/gcc6-... - that way they don't conflict with anything.
I've added gcc6 to the builddepends, and check for that in the Python code of pmbootstrap. When that is the case, it installs the gcc6 cross compiler.
So all in all it's pretty close to working as PoC. I need to fine tune the Linux APKBUILD now, so it really uses the cross compiler, and maybe a few other fixes.

Looks like this could be ready on monday.

By Oliver Smith on 2018-09-29T20:45:22

mentioned in commit pmbootstrap@3e7c95e8

By Oliver Smith on 2018-09-30T02:31:29

mentioned in merge request pmbootstrap!1699 (merged)

By Oliver Smith on 2018-09-30T02:33:10

mentioned in commit 27f31bcc

By Oliver Smith on 2018-09-30T02:40:45

I've compiled a modified linux-samsung-i9100 with GCC6 successfully \o/

To reproduce that (should work the same for other devices):

Use latest pmaports repository
Generate cross compiler aports: pmbootstrap aportgen gcc6-armhf (mind the 6!)
Modify your linux-* aport:
- add gcc6 to makedepends
- modify the "HOSTCC" lines as follows:

-HOSTCC="${CC:-gcc}"
-HOSTCC="${HOSTCC#${CROSS_COMPILE}}"
+
+# old kernel needs GCC6
+CC="gcc6-$CC"
+HOSTCC="gcc6-gcc"
+CROSS_COMPILE="gcc6-$CROSS_COMPILE"

That's enough for today next up comes streamlining, such as providing this by default in new linux aport templates, and updating the existing aports. Maybe the CC line is not needed for example, this can probably be streamlined a bit.

By Oliver Smith on 2018-09-30T02:47:30

mentioned in commit 537ffa55

By Oliver Smith on 2018-09-30T02:57:34

Alright, the cross compilers are building now. That's really it for today. If someone wants to compile their kernel, it should be possible by modifying the linux APKBUILD as mentioned above.

I'll take a break of this tomorrow (sunday), and then get back at it on monday.

If someone has time and interest to help out, the following would be helpful:

check if a kernel generated that way is actually booting
prepare a merge request, that updates all old vendor kernels to compile with GCC6 as outlined above
ideally test if they really compile

By Oliver Smith on 2018-09-30T03:03:06

mentioned in commit daba27dd

By Oliver Smith on 2018-10-01T06:47:24

@danct12 reported in the channel that the z00vd kernel boots up with gcc8. It's a 3.18.x based kernel, the kernel @z3ntu tested for the fairphone 2 is based on 3.4.0. My conclusion from this is, that it's probably a good idea to make using the latest GCC the default for new kernels. If they do boot, we don't need to install the GCC6 dependency to compile them, and at one point in the future we'll have 4.x kernels coming in, which won't need gcc6 either.

Status update:

Created a wiki page explaining how to switch between GCC6 and the latest GCC in the kernel APKBUILDs: https://wiki.postmarketos.org/wiki/Vendor_kernel_specific_package
I've set all 3.x kernels to compile with GCC6 in this branch: fix/vendorkernels-use-gcc6. Typically the patch for one APKBUILD looks like this:

-makedepends="perl sed installkernel bash gmp-dev bc linux-headers elfutils-dev xz dtbtool"
-HOSTCC="${CC:-gcc}"
-HOSTCC="${HOSTCC#${CROSS_COMPILE}}"
+makedepends="perl sed installkernel bash gmp-dev bc linux-headers elfutils-dev xz dtbtool gcc6"
+
+# Compiler: this kernel was only tested with GCC6. Feel free to make a merge
+# request if you find out that it is booting working with newer GCCs as
+# well. See <https://postmarketos.org/vendorkernel> for instructions.
+CC="gcc6-$CC"
+HOSTCC="gcc6-gcc"
+CROSS_COMPILE="gcc6-$CROSS_COMPILE"

I'm compiling all kernels in that branch manually to make sure that they work as expected
Then I'll merge it and change the default aportgen code to use GCC8 by default.

By Oliver Smith on 2018-10-01T06:57:26

mentioned in merge request !30 (merged)

By Oliver Smith on 2018-10-01T06:59:20

mentioned in commit e4aa6332

By Oliver Smith on 2018-10-01T08:01:11

mentioned in commit bf2fa4f0

By Oliver Smith on 2018-10-01T15:16:10

mentioned in commit 7dfa5923

By Oliver Smith on 2018-10-01T16:19:02

mentioned in merge request !20 (merged)

By Oliver Smith on 2018-10-02T04:45:39

mentioned in merge request pmbootstrap!1697 (merged)

By Oliver Smith on 2018-10-02T07:08:32

mentioned in commit 5ab14cf0

By Oliver Smith on 2018-10-03T07:51:52

mentioned in commit 71a669a4

By Oliver Smith on 2018-10-03T10:07:13

mentioned in commit a9169619

By Oliver Smith on 2018-10-03T16:00:10

mentioned in commit d32d9c3d

By Oliver Smith on 2018-10-04T04:23:00

All kernels from the repo are compiling again. these will be the finishing touches:

adjust devicepkg-dev to ship compiler-gcc.h (I'll install it to /usr/share, so we don't need to ship it with each aport, making reviews easier)
adjust aportgen code:
- link to the vendorkernel reference
- add the gcc8 patches
- remove compiler-gcc6.h
test by recreating the i9100 aport
make it compile and test on the device

By Oliver Smith on 2018-10-04T08:01:01

The envkernel.sh script won't work for gcc6 kernels. The cross compile environment variables would need to be adjusted appropriately.

By ryang on 2018-10-04T16:35:32

True. But is anyone using envkernel.sh with old kernels? I thought the use case would be for mainlining (where GCC8 should work just fine).

By Oliver Smith on 2018-10-05T05:10:52

mentioned in merge request !35 (merged)

By Oliver Smith on 2018-10-05T07:00:54

mentioned in merge request pmbootstrap!1702 (merged)

By Oliver Smith on 2018-10-05T07:16:11

status update

Right now, "pmbootstrap aportgen linux-..." creates broken aports that don't work with either GCC6 or GCC8. This is the last important piece to this issue and will be resolved by the following two merge requests:

They only work together and can be used already if somebody wants to create a new linux aport right now.

I'll merge these next monday, if someone wants to they can be reviewed in the meantime. (I'm still operating in this "emergency fix mode" until we are through with this issue, so porting new devices is working again. After that, I'll put my commits up for proper review like everyone else again).

TODO:

test the kernel on the lg-mako
merge both MRs
increase the pmbootstrap and pmaports versions (and their checks), so people don't get additional breakage by only updating one of the repositories

PS: I've tried to do the proof of concept with the i9100 first, but its kernel has problems when not using a different build directory, and I'm spending a lot of time on this issue already so I chose to go with lg-mako.

By Oliver Smith on 2018-10-05T07:25:08

changed title from Adjust to GCC8 upgrade in Alpine edge to WIP: Adjust to GCC8 upgrade in Alpine edge

By Oliver Smith on 2018-10-05T07:23:52

changed title from WIP: Adjust to GCC8 upgrade in Alpine edge to Adjust to GCC8 upgrade in Alpine edge

By Oliver Smith on 2018-10-05T07:24:00

True. But is anyone using envkernel.sh with old kernels? I thought the use case would be for mainlining (where GCC8 should work just fine).

I thought it would be useful to quickly compile and test code changes with any kernel. I've done it a lot when experimenting with a downstream kernel.

By ryang on 2018-10-05T13:16:14

Oh, happy to hear that it is useful for that case!

If you have time and interest for this, would you like to add an argument to it, which lets it optionally use GCC6?

By Oliver Smith on 2018-10-05T15:01:47

mentioned in commit 28d29826

By Oliver Smith on 2018-10-05T15:51:11

mentioned in commit e9bbd50c

By Oliver Smith on 2018-10-05T15:51:11

mentioned in commit 3aa1a7cd

By Oliver Smith on 2018-10-05T15:51:14

mentioned in commit 6d82b400

By Oliver Smith on 2018-10-05T15:51:14

mentioned in commit 370914c2

By Oliver Smith on 2018-10-05T15:51:16

Adjust to GCC8 upgrade in Alpine edge

Child items 0

Activity

status update