pmbootstrap broken on kernels older than 3.17 (e.g. Debian 8) (libressl not working in chroot)

changed the description

By silver on 2018-09-10T22:57:18

Well, what does pmbootstrap log say?

Please paste or attach the output of what you see in the terminal where you run pmbootstrap init as well as the output of pmbootstrap log. Thanks!

By Oliver Smith on 2018-09-11T04:45:27

changed title from pmbootstrap broken by last commit to pmbootstrap broken by pmaports split

By Oliver Smith on 2018-09-11T04:43:55

silver@log:~/apexq2/pmbootstrap$ ./pmbootstrap.py init
[16:28:57] Location of the 'work' path. Multiple chroots (native, device arch, device rootfs) will be created in there.
[16:28:57] Work path [/home/silver/.local/var/pmbootstrap]: 
[16:29:00] pmbootstrap does everything in Alpine Linux chroots, so your host system does not get modified. In order to work with these chroots, pmbootstrap calls 'sudo' internally. To see the commands it runs, you can run 'pmbootstrap log' in a second terminal.
[16:29:00] Setting up the native chroot and cloning the package build recipies (pmaports)...
[16:29:01] Update package index for x86_64 (4 file(s))
[sudo] password for silver: 
[16:29:08] (native) git clone https://gitlab.com/postmarketOS/pmaports.git
[16:29:09] NOTE: The failed command's output is above the ^^^ line in the log file: /home/silver/.local/var/pmbootstrap/log.txt
[16:29:09] ERROR: Command failed: (native) % cd /home/pmos/git/; busybox su pmos -c git clone https://gitlab.com/postmarketOS/pmaports.git pmaports.temp
[16:29:09] See also: <https://postmarketos.org/troubleshooting>
Run 'pmbootstrap log' for details.

(006067) [16:28:57] Location of the 'work' path. Multiple chroots (native, device arch, device rootfs) will be created in there.
[16:28:57] Work path [/home/silver/.local/var/pmbootstrap] /home/silver/.local/var/pmbootstrap
(006067) [16:29:00] Save config: /home/silver/.config/pmbootstrap.cfg
(006067) [16:29:00] pmbootstrap does everything in Alpine Linux chroots, so your host system does not get modified. In order to work with these chroots, pmbootstrap calls 'sudo' internally. To see the commands it runs, you can run 'pmbootstrap log' in a second terminal.
(006067) [16:29:00] Setting up the native chroot and cloning the package build recipies (pmaports)...
(006067) [16:29:00] (native) calculate depends of git (pmbootstrap -v for details)
(006067) [16:29:01] APKINDEX outdated (older than 4h): http://postmarketos.brixit.nl/x86_64/APKINDEX.tar.gz
(006067) [16:29:01] APKINDEX outdated (older than 4h): http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
(006067) [16:29:01] APKINDEX outdated (older than 4h): http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(006067) [16:29:01] APKINDEX outdated (older than 4h): http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
(006067) [16:29:01] Update package index for x86_64 (4 file(s))
(006067) [16:29:01] % rm /home/silver/.local/var/pmbootstrap/cache_http/APKINDEX_c5a7619e05ff97b647811ea60530885797e5b8cbd6285f0ec2cdc7d7b9b1acb9
(006067) [16:29:01] Download http://postmarketos.brixit.nl/x86_64/APKINDEX.tar.gz
(006067) [16:29:02] % sudo cp /home/silver/.local/var/pmbootstrap/cache_http/APKINDEX_c5a7619e05ff97b647811ea60530885797e5b8cbd6285f0ec2cdc7d7b9b1acb9 /home/silver/.local/var/pmbootstrap/cache_apk_x86_64/APKINDEX.8b865e19.tar.gz
(006067) [16:29:05] % rm /home/silver/.local/var/pmbootstrap/cache_http/APKINDEX_0999dbfe3755729bd8aa3997d03dbd36a87187a19b61ce5cea2e01671a6305d6
(006067) [16:29:05] Download http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(006067) [16:29:05] % sudo cp /home/silver/.local/var/pmbootstrap/cache_http/APKINDEX_0999dbfe3755729bd8aa3997d03dbd36a87187a19b61ce5cea2e01671a6305d6 /home/silver/.local/var/pmbootstrap/cache_apk_x86_64/APKINDEX.b53994b4.tar.gz
(006067) [16:29:05] % rm /home/silver/.local/var/pmbootstrap/cache_http/APKINDEX_b36af8b1d7f948f0d2fcae5bd60c1b876620e69b987bc066e7a90c810687e76f
(006067) [16:29:05] Download http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
(006067) [16:29:06] % sudo cp /home/silver/.local/var/pmbootstrap/cache_http/APKINDEX_b36af8b1d7f948f0d2fcae5bd60c1b876620e69b987bc066e7a90c810687e76f /home/silver/.local/var/pmbootstrap/cache_apk_x86_64/APKINDEX.066df28d.tar.gz
(006067) [16:29:06] % rm /home/silver/.local/var/pmbootstrap/cache_http/APKINDEX_6ccac8825a1621fb204ad4b3e645b24ffd8fa9acfe97edc265646cafd18d3e4c
(006067) [16:29:06] Download http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
(006067) [16:29:07] % sudo cp /home/silver/.local/var/pmbootstrap/cache_http/APKINDEX_6ccac8825a1621fb204ad4b3e645b24ffd8fa9acfe97edc265646cafd18d3e4c /home/silver/.local/var/pmbootstrap/cache_apk_x86_64/APKINDEX.30e6f5af.tar.gz
(006067) [16:29:08] (native) git clone https://gitlab.com/postmarketOS/pmaports.git
(006067) [16:29:08] (native) % cd /home/pmos/git/; busybox su pmos -c git clone https://gitlab.com/postmarketOS/pmaports.git pmaports.temp
Cloning into 'pmaports.temp'...
(006067) [16:29:09] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(006067) [16:29:09] NOTE: The failed command's output is above the ^^^ line in the log file: /home/silver/.local/var/pmbootstrap/log.txt
(006067) [16:29:09] ERROR: Command failed: (native) % cd /home/pmos/git/; busybox su pmos -c git clone https://gitlab.com/postmarketOS/pmaports.git pmaports.temp
(006067) [16:29:09] See also: <https://postmarketos.org/troubleshooting>
(006067) [16:29:09] Traceback (most recent call last):
  File "/home/silver/apexq2/pmbootstrap/pmb/__init__.py", line 49, in main
    return config_init.frontend(args)
  File "/home/silver/apexq2/pmbootstrap/pmb/config/init.py", line 327, in frontend
    pmb.config.pmaports.init(args)
  File "/home/silver/apexq2/pmbootstrap/pmb/config/pmaports.py", line 134, in init
    clone(args)
  File "/home/silver/apexq2/pmbootstrap/pmb/config/pmaports.py", line 52, in clone
    pmb.helpers.git.clone(args, "pmaports", False, True)
  File "/home/silver/apexq2/pmbootstrap/pmb/helpers/git.py", line 52, in clone
    working_dir="/home/pmos/git/")
  File "/home/silver/apexq2/pmbootstrap/pmb/chroot/user.py", line 40, in user
    output_return, check, {}, auto_init)
  File "/home/silver/apexq2/pmbootstrap/pmb/chroot/root.py", line 90, in root
    output_return, check, kill_as_root)
  File "/home/silver/apexq2/pmbootstrap/pmb/helpers/run_core.py", line 265, in core
    raise RuntimeError("Command failed: " + log_message)
RuntimeError: Command failed: (native) % cd /home/pmos/git/; busybox su pmos -c git clone https://gitlab.com/postmarketOS/pmaports.git pmaports.temp

(006089) [16:30:01] % tail -f /home/silver/.local/var/pmbootstrap/log.txt -n 60
(006089) [16:30:01] *** output passed to pmbootstrap stdout, not to this log ***
(006093) [16:30:25] % tail -f /home/silver/.local/var/pmbootstrap/log.txt -n 60
(006093) [16:30:25] *** output passed to pmbootstrap stdout, not to this log ***

By silver on 2018-09-12T06:30:05

Somehow git has failed without really throwing an error. Could you please also try these things?

Run pmbootstrap zap to delete the chroots, then try again. Does it work now?
If it does not, what's the content of your cache_git folder? (that's where it's supposed to clone the pmaports repository)

$ ls -R /home/silver/.local/var/pmbootstrap/cache_git

Thank you very much @5ilver!

By Oliver Smith on 2018-09-12T06:47:08

pmbootstrap zap doesn't fix, unfortunately. The cache_git directory is empty. Oddly I find this issue only on my laptop (running Debian 8.11) and not my desktop running (Ubuntu 18) despite both being on the latest commit

By rendeko on 2018-09-14T19:59:42

I can confirm there are issues on debian 8 (devuan) and no issues in ubuntu bionic. In IRC we found with strace that curl's TLS in the chroot was using a kernel feature introduced in kernel 3.17 and was crashing out silently. When curling from local env not in chroot SSL was used instead of TLS.

Perhaps the cypher selection done on the web server side triggered this, such as if tls 1.3 was recently enabled or raised in preference on their frontend.

By silver on 2018-09-14T20:33:52

Wow, thanks for tracing that down! I think we have two solutions for this now:

disable using that kernel feature in CURL if possible (maybe with an environment variable)
catch the failing git clone and display the full git clone command to the user, so they can execute it manually

By Oliver Smith on 2018-09-17T04:50:50

added type::bug + 1 deleted label and removed 1 deleted label

I did some research on that. There is no mention of KTLS in the source code of curl or openssl, so it didn't seem like we could simply set an environment variable or similar.

But I've started writing a patch that tries to use the host system's git binary if the one from the Alpine chroot failed to clone the folder correctly. It's working as proof of concept already and only needs some polishing, so I'll make a merge request with it tomorrow.

EDIT: I don't think the feature we are talking about is KTLS on second thought, this was added fairly recently. But still, the patch I've worked on seems like a good solution to me.

By Oliver Smith on 2018-09-17T07:53:04

mentioned in merge request !1695 (closed)

By Oliver Smith on 2018-09-18T04:33:37

@5ilver, @rendeko: I've created a merge request, which makes pmbootstrap fall back to the native git binary in !1695 (closed). Could you try out if that fixes the problem for you?

By Oliver Smith on 2018-09-18T04:35:09

So the merge request !1695 (closed) did not help, it works around the initial clone from git. But whenever libressl gets used again (e.g. by wget when downloading sources), it fails again.

@MartijnBraam wrote in https://bugs.alpinelinux.org/issues/9419:

One of the recent updates of libressl on alpine edge seem to break a lot of tools that use ssl when running as a container on debian 8.

This is due to debian 8 using the 3.16 kernel but libressl seems to crash on not having the getrandom syscall introduced in linux 3.17.

I've so far run into this with curl and git.

I've set up a VM with Debian 8 and a 3.16 kernel, and I can reproduce this.

So I took a look at the code: http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.7.4.tar.gz (couldn't find an up-to-date location where that could be browsed online)

The file where it fails is crypto/compat/getentropy_linux.c:

...
int
getentropy(void *buf, size_t len)
{
    int ret = -1;

    if (len > 256) {
        errno = EIO;
        return (-1);
    }

#if defined(SYS_getrandom) && defined(GRND_NONBLOCK)
    /*
     * Try descriptor-less getrandom(), in non-blocking mode.
     *
     * The design of Linux getrandom is broken.  It has an
     * uninitialized phase coupled with blocking behaviour, which
     * is unacceptable from within a library at boot time without
     * possible recovery. See http://bugs.python.org/issue26839#msg267745
     */
    ret = getentropy_getrandom(buf, len);
    if (ret != -1)
        return (ret);
#endif

    /*
     * Try to get entropy with /dev/urandom
     *
     * This can fail if the process is inside a chroot or if file
     * descriptors are exhausted.
     */
    ret = getentropy_urandom(buf, len);
    if (ret != -1)
        return (ret);

#ifdef SYS__sysctl
    /*
     * Try to use sysctl CTL_KERN, KERN_RANDOM, RANDOM_UUID.
     * sysctl is a failsafe API, so it guarantees a result.  This
     * should work inside a chroot, or when file descriptors are
     * exhausted.
     *
     * However this can fail if the Linux kernel removes support
     * for sysctl.  Starting in 2007, there have been efforts to
     * deprecate the sysctl API/ABI, and push callers towards use
     * of the chroot-unavailable fd-using /proc mechanism --
     * essentially the same problems as /dev/urandom.
     *
     * Numerous setbacks have been encountered in their deprecation
     * schedule, so as of June 2014 the kernel ABI still exists on
     * most Linux architectures. The sysctl() stub in libc is missing
     * on some systems.  There are also reports that some kernels
     * spew messages to the console.
     */
    ret = getentropy_sysctl(buf, len);
    if (ret != -1)
        return (ret);
#endif /* SYS__sysctl */

    /*
     * Entropy collection via /dev/urandom and sysctl have failed.
     *
     * No other API exists for collecting entropy.  See the large
     * comment block above.
     *
     * We have very few options:
     *     - Even syslog_r is unsafe to call at this low level, so
     *   there is no way to alert the user or program.
     *     - Cannot call abort() because some systems have unsafe
     *   corefiles.
     *     - Could raise(SIGKILL) resulting in silent program termination.
     *     - Return EIO, to hint that arc4random's stir function
     *       should raise(SIGKILL)
     *     - Do the best under the circumstances....
     *
     * This code path exists to bring light to the issue that Linux
     * still does not provide a failsafe API for entropy collection.
     *
     * We hope this demonstrates that Linux should either retain their
     * sysctl ABI, or consider providing a new failsafe API which
     * works in a chroot or when file descriptors are exhausted.
     */
#undef FAIL_INSTEAD_OF_TRYING_FALLBACK
#ifdef FAIL_INSTEAD_OF_TRYING_FALLBACK
    raise(SIGKILL);
#endif
    ret = getentropy_fallback(buf, len);
    if (ret != -1)
        return (ret);

    errno = EIO;
    return (ret);
}
...

By Oliver Smith on 2018-09-21T07:43:02

The easiest workaround for people on debian 8 is probably to install the 4.9 kernel from jessie-backports (https://packages.debian.org/jessie-backports/linux-image-amd64)

By Martijn Braam on 2018-09-21T09:25:49

Upgrading the kernel is not an option for users of shared systems, or systems which are locked to a vendor kernel. In my case, debian 8 is a containerized virtual private server and does not have access to the kernel.

Would it be possible to pin libressl to an older package version if kernel 3.17 is not available, or to use the existing system version?

By silver on 2018-09-21T16:41:33

some thoughts on this:

the stable version of Alpine (3.8) has libressl 2.6 (edge has 2.7), so this shouldn't be a problem with stable alpine. I'd like to get postmarketOS based on stable Alpine ASAP (pmaports#5 (closed)), so we shouldn't have to deal with breakage like that as often as we have now (and to clarify, this is not Alpine's fault - they have simply upgraded the libressl version here - but Alpine stable doesn't do version updates as often so we only have to migrate to new versions when updating the Alpine version more or less).
ncopa mentioned today that he will be working on using openssl in Alpine edge again. Maybe this problem doesn't appear with OpenSSL (from reading the comments in the source code above, I can imagine that)
we can not easily "pin to a specific libressl version" (fork the aport), because too many programs link against libressl, and we would need to fork all the aports of those as well and rebuild them for postmarketOS.
Docker is somewhat commercially backing Alpine, but it's unlikely that Docker users will have this problem since Docker requires fairly recent kernels.

@5ilver: thanks for the good idea though, and I'm sorry that right now I don't really have a solution. Maybe we can make it work soon again with either the first or second bullet point.

By Oliver Smith on 2018-09-24T21:03:17

changed title from pmbootstrap broken by pmaports split to pmbootstrap broken on kernels older than 3.17 kernel (e.g. Debian 8) (libressl not working in chroot)

By Oliver Smith on 2018-09-24T21:33:58

changed title from pmbootstrap broken on kernels older than 3.17 kernel (e.g. Debian 8) (libressl not working in chroot) to pmbootstrap broken on kernels older than 3.17 (e.g. Debian 8) (libressl not working in chroot)

By Oliver Smith on 2018-09-24T21:34:14

The alternative to fixing the problem is enabling the user to fix the problem. Perhaps a check for a sufficient kernel version or a working curl with a friendly error message could be added to pmb/init.py

By silver on 2018-09-24T22:33:27

mentioned in merge request !1696 (merged)

By Oliver Smith on 2018-09-25T07:19:26

Good point. Merge request !1696 (merged) will print the following when the clone fails, created that along with a wiki page.

NOTE: cloning from git is known to fail when the host linux kernel is older than 3.17: https://postmarketos.org/oldkernel

By Oliver Smith on 2018-09-25T07:24:44

mentioned in commit wiki@c193e6c2

By postmarketOS Wiki bot on 2018-09-25T07:42:29

mentioned in issue pmaports#107 (closed)

By Michael Hamann on 2018-10-05T18:42:27

@MartijnBraam: you had reported this to Alpine as this issue: https://gitlab.alpinelinux.org/alpine/aports/issues/9419

ncopa said, that it's probably fixed now, since Alpine switched back to openssl.

Can you check if it is fixed?

I'm asking, because then we can remove this from the pmbootstrap code (and we can close this issue):

        if not os.path.exists(args.work + "/cache_git/" + name_temp):
            logging.info("NOTE: cloning from git is known to fail when the"
                         " host linux kernel is older than 3.17:"
                         " <https://postmarketos.org/oldkernel>")

By Oliver Smith on 2019-12-22T08:35:55

On second thought: nevermind. I'm rewriting the function to use git from the host system, instead of git from the chroot (because then it will work properly, even if pmaports is not inside the work dir). So this should not matter. This is relevant for #1858 (closed).

Closing, because I'm assuming that this was fixed by switching back to openssl. If this is still an issue, please open a new issue with new logs, as this issue is pretty old by now.

By Oliver Smith on 2019-12-22T08:40:23

closed

By Oliver Smith on 2019-12-22T08:39:50

pmbootstrap broken on kernels older than 3.17 (e.g. Debian 8) (libressl not working in chroot)

Child items ...

Activity