Skip to content

enhancement: speed up builds by launching native cross compilers directly in the emulated chroot

This is an evolution/inversion of my proposal from #659.

I think we can speed up compiling packages by mounting the native chroot inside an emulated chroot, so we can use the native cross compilers directly from the emulated chroot.

i.e:

root@carl:~/.local/var/pmbootstrap# mount -o bind chroot_native chroot_buildroot_aarch64/mnt/native
...
zhuowei@carl:~/pmbootstrap$ ./pmbootstrap.py chroot -b aarch64
[07:52:54] (buildroot_aarch64) % sh -i
/ # cd
~ # ln -s /mnt/native/lib/ld-musl-x86_64.so.1 /lib/ld-musl-x86_64.so.1
~ # LD_LIBRARY_PATH=/mnt/native/lib:/mnt/native/usr/lib time /mnt/native/usr/bin/aarch64-alpine-linux-musl-g++ --sysroot=/ test.cpp
real	0m 0.56s
user	0m 0.49s
sys	0m 0.07s
~ # ./a.out 
hello world
~ # file a.out 
a.out: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, with debug_info, not stripped
~ #

Why would this be useful?

Because some tasks may require building large packages such as browsers, and anybody working on those tasks would greatly benefit from faster compilation speed.

Granted, as noted in the previous proposal, compile speed matters less now, since we have a binary repository, but the binary repo may not help packagers porting a new UI environment where almost everything needs to be built from scratch.

Have this been tested with actual packages?

Not yet. Just wanted to share this idea to see if has any merit before prototyping it.

Why would this be faster than just using distcc?

Because distcc still runs the preprocessor under emulation.

Currently pmbootstrap speeds up builds of emulated architectures with distcc. For example, if I'm building for aarch64 on a x86_64 computer:

  • pmbootstrap launches a chroot with qemu's aarch64 user emulation
  • code is compiled with distcc
  • distcc first uses qemu to run gcc -E, to preprocess the source using headers from the aarch64 chroot
  • distcc then sends the preprocessed source to the native chroot and compiles it

However, while this offloads the slowest part - the compilation - to the native cross compiler, the gcc -E preprocessing step itself takes a significant amount of time:

~ # cat test.cpp
#include <iostream>

int main() {
    std::cout << "hello world\n";
}
~ # time g++ -E test.cpp -o /tmp/emulated.cpp
real	0m 0.88s
user	0m 0.84s
sys	0m 0.04s

Compare this to the native speed of g++ -E:

~ # LD_LIBRARY_PATH=/mnt/native/lib:/mnt/native/usr/lib time /mnt/native/usr/bin/aarch64-alpine-linux-musl-g++ --sysroot=/ -E test.cpp -o /tmp/cross.cpp
real	0m 0.08s
user	0m 0.06s
sys	0m 0.01s

That's a 10x slowdown just in the preprocessor. On a large package, this preprocessing time adds up.

Why not just use distcc's pump mode?

distcc's pump mode offloads the preprocessing. Unfortunately, it doesn't work with ccache.

Why use this silly inverted cross compile setup instead of a normal cross compile?

Because this should work for all packages, not just those that support cross compilation. Also, it may be easier to implement in pmbootstrap.

My previous research in #659 aimed to build supported packages entirely inside the native chroot, using native makedepends such as make or cmake, and using headers bind-mounted from the aarch64 chroot.

However, this requires packages to support cross compilation in their APKBUILDs and build systems. Not all packages do, and even those that do support it requires hand holding to work well in cross complation, as I found out.

In this setup, the packages' own build scripts are still executed under emulation, and all the headers/libraries are present in their normal paths. From the packages' perspective, this should be no different than building with a native GCC.

This would be slower than a full cross compile, but would likely be faster than the current setup.

Also, this should require less modification to pmbootstrap than my previous proposal, since pmbootstrap doesn't need to handle makedepends for the native chroot.

How hard would this be to implement?

I haven't tested this with pmbootstrap yet. I'm guessing the steps would be:

  • make and package wrapper scripts for gcc/g++/ld/as that runs the native chroot cross compilers
  • modify pmbootstrap to install the wrappers to non-native build chroots
  • modify pmbootstrap to make the symlink and the bind mount when entering build chroots
  • benchmark the result against the current setup and see how much time it saves
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information