qt5-qtwebengine for aarch64 and armhf isn't in the binary repo

Created by: ollieparanoid

qt5-qtwebengine is the whole chromium browser, with a bit of QT glue. It takes hours to compile, even with a fast CPU. I've tried to compile it for x86_64, armhf and aarch64.

Status

• x86_64: compiled, and is available in the repo now (I did not test using it, that would be helpful)
• armhf failed at the very end when linking libQt5WebEngineCore

linking ..../lib/libQt5WebEngineCore.so.5.9.3
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
compilation terminated
distcc[12345] ERROR: compile (null) on localhost failed

• aarch64: hangs forever (left it there for hours) on [327/237] LINK gn fairly early for some reason.

NOTE: I gave it 50 GB of SWAP just in case, so running out of memory shouldn't be the problem (and that would have gotten other output).

How to help

Try to compile it on your own PC for foreign arches. Especially for aarch64 it fails early for me, so it would be good to know if it's just me or not. If it's possible, trying to compile on a native device (e.g. @opendata26 with his Shield TV) would also be helpful (does it run through there?).

To build it, change arch="x86_64" back to arch="all" in aports/main/qt5-qtwebengine/APKBUILD, then run either:

• pmbootstrap build --arch=aarch64 qt5-qtwebengine
• pmbootstrap build --arch=armhf qt5-qtwebengine

Created by: ollieparanoid

I've tried to build for aarch64 again, and it hangs again. The hanging process is:

/usr/bin/qemu-aarch64-static /home/pmos/build/src/build/src/3rdparty/chromium/tools/gn/out/Release/gn

It is open 9 times, and each of them hangs at FUTEXWAIT_PRIVATE as strace reveals. So it's a deadlock. Maybe caused by a bug in Qemu.

It might make sense to package gn as standalone (example script). From what I've read chromium supports building it with a gn provided by the system. Then it might be easier to debug what is going wrong, we could try a newer version (maybe this is already fixed), and we could force -j1 as workaround.

By * postmarketOS Bot on 2017-12-16T04:31:58

I've tried building it natively on my Raspberry Pi 3 with ArchLinuxARM but cc1plus just gets killed after some time. (g++: internal compiler error: Killed (program cc1plus)). Probably RAM related or something. I've made a 4GB swap file and am trying again but I don't expect it to work...

By Luca Weiss on 2017-12-16T22:23:50

Result over the night: Xorg got killed while running but it got to about 8000/18000 instead of the only 4000/18000 prior to activating swap. So natively compiling is not an option on the RPi3 (at least for qtwebengine)

By Luca Weiss on 2017-12-17T10:12:32

Created by: ollieparanoid

Thanks for trying

I have another idea: Cross-compiling from the native chroot (just like we do it for the kernel). That would require the target arch buildroot to be mounted inside the native chroot (similar to #665). All in all I think it's quite some work, but since compiling does work for x86_64, I guess cross-compiling would work then as well. See also.

By * postmarketOS Bot on 2017-12-17T18:56:27

And I can reproduce the armhf issue with cross-compilation from x86_64, so it's not because of your computer

ar cqs Release/libqtwebenginecoreapi.a .obj/qtwebenginecoreglobal.o .obj/qwebenginecookiestore.o .obj/qwebenginehttprequest.o .obj/qwebengineurlrequestinfo.o .obj/qwebengineurlrequestjob.o .obj/qwebengineurlschemehandler.o .obj/qtbug-60565.o .obj/qtbug-61521.o .obj/moc_qwebenginecookiestore.o .obj/moc_qwebengineurlrequestinterceptor.o .obj/moc_qwebengineurlrequestjob.o .obj/moc_qwebengineurlschemehandler.o
make[3]: Leaving directory '/home/pmos/build/src/build/src/core/api'
( test -e Makefile.core_module || /usr/bin/qmake-qt5 -o Makefile.core_module /home/pmos/build/src/qtwebengine-opensource-src-5.9.3/src/core/core_module.pro WEBENGINE_CONFIG+=use_system_ffmpeg ) && /usr/bin/make -f Makefile.core_module 
make[3]: Entering directory '/home/pmos/build/src/build/src/core'
creating linker version script QtWebEngineCore.version
rm -f libQt5WebEngineCore.so.5.9.3 libQt5WebEngineCore.so libQt5WebEngineCore.so.5 libQt5WebEngineCore.so.5.9
linking ../../lib/libQt5WebEngineCore.so.5.9.3
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
compilation terminated.
distcc[3560] ERROR: compile (null) on localhost failed
make[3]: *** [Makefile.core_module:77: ../../lib/libQt5WebEngineCore.so.5.9.3] Error 1
make[3]: Leaving directory '/home/pmos/build/src/build/src/core'
make[2]: *** [Makefile:122: sub-core_module-pro-make_first] Error 2
make[2]: Leaving directory '/home/pmos/build/src/build/src/core'
make[1]: *** [Makefile:79: sub-core-make_first] Error 2
make[1]: Leaving directory '/home/pmos/build/src/build/src'
make: *** [Makefile:47: sub-src-make_first] Error 2
>>> ERROR: qt5-qtwebengine: all failed

By Luca Weiss on 2017-12-17T20:34:24

I was also able to reproduce both results, unfortunately.

@z3ntu, I noticed ld dumped core, could you try to examine the coredump with gdb to maybe see what the cause of the segfault was? Hopefully it will be something easier to fix than cross-compiling qtwebengine. I would check mine, but I ran a zap before noticing the coredump :(

By Alex Roth on 2017-12-17T20:38:28

/home/pmos/build/src/build/src/core # /usr/lib/gcc/armv6-alpine-linux-musleabihf/6.4.0/../../../../armv6-alpine-linux-musleabihf/bin/ld @/tmp/cchaahmL
Segmentation fault (core dumped)
/home/pmos/build/src/build/src/core # gdb --args /usr/lib/gcc/armv6-alpine-linux-musleabihf/6.4.0/../../../../armv6-alpine-linux-musleabihf/bin/ld @/tmp/cchaahmL
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "armv6-alpine-linux-musleabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/gcc/armv6-alpine-linux-musleabihf/6.4.0/../../../../armv6-alpine-linux-musleabihf/bin/ld...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/armv6-alpine-linux-musleabihf/bin/ld @/tmp/cchaahmL
qemu: Unsupported syscall: 26
Could not trace the inferior process.
Error: ptrace: Function not implementedDuring startup program exited with code 127.

Apparently you can't use gdb directly inside a qemu chroot but you need to use gdbserver or something (see https://tinkering-is-fun.blogspot.co.at/2009/12/debugging-non-native-programs-with-qemu.html)

By Luca Weiss on 2017-12-18T10:05:59

@z3ntu I think having the coredump already will mean you don't need to run the program within gdb, but you can still analyze what happened.

What happens if you run:

• ulimit -c unlimited to enable coredump file output
• The ld command you created to reproduce this
• gdb /path/to/ld /path/to/coredump/core

You should now be in a gdb shell as if the program had already been run and crashed.

The coredump should be in the directory you ran everything from, in a file called core. I believe that contains enough information for gdb to recreate the state of the program without needing ptrace or other hardware debug features.

By Alex Roth on 2017-12-18T12:36:44

(because I use systemd and systemd likes systemd-coredump, you have to change it:

cat /proc/sys/kernel/core_pattern # in chroot
sudo sysctl -w kernel.core_pattern=/var/crash/core.%u.%e.%p # on host
cat /proc/sys/kernel/core_pattern # in chroot

then

/home/pmos/build/src/build/src/core # gdb /usr/lib/gcc/armv6-alpine-linux-musleabihf/6.4.0/../../../../armv6-alpine-linux-musleabihf/bin/ld /var/crash/core.0.ld.28721
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "armv6-alpine-linux-musleabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/gcc/armv6-alpine-linux-musleabihf/6.4.0/../../../../armv6-alpine-linux-musleabihf/bin/ld...(no debugging symbols found)...done.
"/var/crash/core.0.ld.28721" is not a core dump: File format not recognized
(gdb) 

any idea? the file contains binary stuff

By Luca Weiss on 2017-12-18T13:07:43

Created by: ollieparanoid

Thanks for confirming that it's not my computer

I have looked into cross-compiling QT from the native chroot. The big problem is, that we would need to call QT's configure to prepare everything for cross-compilation. And the configure script is only shipped in the full QT source, but not in each module's source code. This means one would need to compile all of QT in one go, and not the way Alpine does it, by compiling each module in a single package (which makes more sense from a packager's point of view).

As alternative it might be possible to modify some *.pro files in the source tree, like done here?

Anyway, I put my WIP changes in the feature/native-cross-compile-qt branch if someone wants to play with it (based on @zhuowei's work in #665, see the commit description of e0f90e2f for details).

By * postmarketOS Bot on 2017-12-18T18:35:22

I was doing some digging trying to figure out why this might be happening, and stumbled upon this bug.

It's been resolved, but it pointed out that when qemu is doing 32bit binfmt emulation, the emulated utilities won't be able to use more than 2G of ram - which makes sense, even though it's a 64 bit system, the utilities are still 32bit.

That may be the cause of the segfault - I bet that final ld call will use a lot more than 2G of ram, and a swapfile won't help.

E: This bug mentions that Chromium suggests using a 64 bit machine even to generate 32 bit executables, and that the linker is likely to run out of memory on a 32bit system. This bug agrees that it's not generally possible to build a 32bit executable on a 32bit system because the linker runs out of memory, even with gold linker

By Alex Roth on 2017-12-19T02:00:19

Created by: zhuowei

@ollieparanoid if you need to check the correct command line arguments to pass to qmake for cross compiling, try looking at OpenEmbedded's qtwebengine recipe (which LuneOS uses). https://github.com/meta-qt5/meta-qt5/blob/master/recipes-qt/qt5/qtwebengine_git.bb and the generic qmake5 base recipe (which the qtwebchannel recipe uses). https://github.com/meta-qt5/meta-qt5/blob/master/classes/qmake5_base.bbclass

Seems they just write a temp qt.conf into the local directory directly when building instead of running qt's config script to set up paths.

By * postmarketOS Bot on 2017-12-19T04:36:44

Interesting fact: ArchLinuxARM has qt5-base for arm (v5 I think), armv6h, armv7h and aarch64 but qt5-webengine (equivalent to qt5-qtwebengine here) only for armv7h and aarch64...

By Luca Weiss on 2017-12-19T07:37:04

Also in ArchLinuxARM chromium is only available for armv7h and aarch64 and on Alpine it's only available for x86_64.

By Luca Weiss on 2017-12-19T07:40:00

If I'm not mistaken, qtwebengine is available for armhf in Debian. Maybe you can copy a few more patches from there: https://anonscm.debian.org/git/pkg-kde/qt/qtwebengine.git

Some time ago I read that the Debian developers packaging qtwebengine where near to giving up, but somehow they managed to package it.

By Jonah Brüchert on 2017-12-19T07:54:42

@JBBgameich I didn't find anything about arm* in the debian patches but the rules file could be interesting for us. They specify that for some architectures that qmake generates different makefiles. See line 29 to 33. e.g. armhf doesn't use the gold but the bfd linker

By Luca Weiss on 2017-12-19T08:57:04

Well, I was able to build for armhf without cross compiling!

The trick was using gold as the linker, not normal ld. It needs far less ram when linking large c++ projects

To do so, I just added binutils-gold to the makedepends, and passed CONFIG+="use_gold_linker", and WEBENGINE_CONFIG+="use_system_ffmpeg use_gold_linker" to qmake.

7 hours later, and I have a working qtwebengine for armhf! It loads webpages fine. It's a little slow, but I believe that's because of the lack of hardware accelleration on my device.

By Alex Roth on 2017-12-19T12:54:27

Interesting. Because Debian actually explicitely disables the ld.gold linker for armhf... But good job! Now only aarch64 is missing ^^

By Luca Weiss on 2017-12-19T13:06:16

Created by: ollieparanoid

Amazing!

Well, I was able to build for armhf without cross compiling!

Do you mean, that you compiled it on an arm device? Or that you did not use the "native" cross-compilation method I was trying to get working for qt5-qtwebengine? I'm asking, because even when not using that "native" method, we're still cross-compiling with pmbootstrap.

More information: https://wiki.postmarketos.org/wiki/Build_internals#Cross-compile_types

By * postmarketOS Bot on 2017-12-19T17:22:47

Sorry, yes, it's using the "distcc" type as it originally was, and my package was built with pmbootstrap.

I meant that I didn't need the "native" method for cross compiling.

By Alex Roth on 2017-12-19T17:27:47

Created by: ollieparanoid

Cool! Sounds like you made all edits in an APKBUILD already - if that is the case and you have time, could you make a PR (and possibly enable "armhf" for all luna packages, where it is disabled because of this issue)?

By * postmarketOS Bot on 2017-12-19T17:31:38

Created by: ollieparanoid

I have good news: the aarch64 version doesn't stop at LINK gn anymore with the gold linker patch we have. I'm doing a full compile of qt5-qtwebengine right now, that will not land in the binary repo, and when that ran through we can finally set arch="all" for all these packages again, and compile these packages for the repo.

By * postmarketOS Bot on 2017-12-22T18:22:59

Created by: ollieparanoid

It turns out, the LINK gn hanging does not appear every time. I have built the entire package for aarch64 as a test, as I wrote it above, and it went through. Then I changed the relevant APKBUILDs to arch="all" again and started building it for the repo.

The second time it hanged again. I've stopped it and started building a third time, and it also hung. But when stopped and started another time, it finally worked and there's a binary aarch64 package in the repo now! It seems like once it is through compiling/linking gn, it's safe.

So this means the bug is still there, but it does not appear every time and at least with some effort it is possible to compile the package. We could look into this more, but I doubt that it is much use at this point - there's already the next version of qt out, to which Alpine will migrate at some point in the future, and maybe the bug is fixed there already. My plan is to keep this status for now and look into it more when this becomes a real blocker.

By * postmarketOS Bot on 2017-12-23T18:16:16

A random thought I've just had: maybe the hanging does not appear if we would use ninja instead of make. Right now we don't built qt5-qtwebengine anymore, but this might change.

By Oliver Smith on 2018-09-19T04:44:30