Kill the child processes spawned by a run command

The timeout feature should kill all of the processes that are created by the command. The run command can (and often does) spawn multiple processes which can remain running after the process.pid is killed.

Reproducing the problem:

• Use device qemu-amd64
• Zap x86_64 apk cache (or delete the cached linux kernel apks then run pmbootstrap index)
• Run pmbootstrap install with a short timeout. A timeout shorter than it will take to download the linux kernel package on your connection.
  e.g. pmbootstrap.py -t 15 install --no-fde

Observe the processes while the packages are downloading/installing. Notice that the chroot command has spawned 3 processes.

$ps xao pid,ppid,pgid,cmd
31738 23247 31738 python3 ./pmbootstrap.py -t 15 install --no-fde
31746 31738 31738 sudo env -i /bin/sh -c CHARSET=UTF-8 HISTFILE='~/.ash_history' PATH=/usr/lib/ccache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SHELL=/bin/ash TERM=xterm /usr/sbin/chroot /ubuntu_home/home/r/.local/var/pmbootstrap/chroot_rootfs_qemu-amd64 /bin/sh -c 'cd /;apk --no-progress add -u --virtual .pmbootstrap postmarketos-base ttf-droid device-qemu-amd64 device-qemu-amd64-kernel-stable postmarketos-ui-weston mesa-dri-virtio dbus chrony cryptsetup eudev haveged htop iw nano openssh postmarketos-mkinitfs sudo tmux util-linux wpa_supplicant shadow networkmanager fontconfig linux-postmarketos-stable weston-clients weston weston-shell-desktop weston-backend-fbdev weston-backend-drm weston-backend-x11 weston-terminal weston-xwayland xorg-server-xwayland postmarketos-demos postmarketos-artwork-wallpapers llvm5-libs libdrm libelf expat libgcc mesa-glapi libstdc++ dbus-libs libcap cryptsetup-libs popt libuuid udev-init-scripts eudev-libs libblkid kmod ncurses-libs libnl3 libmagic openssh-client openssh-sftp-server openssh-server busybox-extras lddtree multipath-tools postmarketos-splash device-mapper parted e2fsprogs e2fsprogs-extra osk-sdl charging-sdl triggerhappy ncurses-terminfo libevent findmnt libfdisk libmount libsmartcols pcsc-lite-libs linux-pam dhcpcd iptables libcurl dbus-glib glib libintl jansson libmm-glib libndp newt nspr nss polkit readline libteam freetype mesa-egl mesa-gles cairo libjpeg-turbo pango pixman libpng wayland-libs-client wayland-libs-cursor wayland-libs-egl libwebp libxkbcommon libinput-libs libunwind wayland-libs-server libweston libweston-desktop mesa-gbm libva mesa-dri-swrast libx11 libxcb libxcursor font-misc-misc font-cursor-misc xkeyboard-config xkbcomp xinit mesa-gl libxau libxdmcp libxfont2 libepoxy libxshmfence gtk+3.0 libffi libpciaccess argon2-libs device-mapper-libs json-c xz-libs ncurses-terminfo-base openssh-keygen openssh-server-common libaio userspace-rcu ttf-dejavu py3-pillow lvm2-libs libcom_err e2fsprogs-libs sdl2 sdl2_ttf directfb libmnl libnftnl-libs ca-certificates nghttp2-libs libssh2 pcre slang sqlite-libs libdaemon libnl3-cli libbz2 libxext libxrender libxft fribidi harfbuzz libevdev mtdev libxfixes encodings font-alias mkfontscale mkfontdir util-macros libxkbfile xauth mcookie xmodmap xrdb libxdamage libxxf86vm libbsd libfontenc shared-mime-info gtk-update-icon-cache libxcomposite libxi libxinerama libxrandr atk at-spi2-atk cairo-gobject cups-libs gdk-pixbuf py3-olefile lcms2 openjpeg python3 tiff device-mapper-event-libs tslib graphite2 libxmu mcpp libxml2 hicolor-icon-theme at-spi2-core avahi-libs gnutls gdbm libxt mcpp-libs libxtst gmp nettle p11-kit libtasn1 libunistring libice libsm'
31747 31746 31738 /bin/sh -c CHARSET=UTF-8 HISTFILE='~/.ash_history' PATH=/usr/lib/ccache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SHELL=/bin/ash TERM=xterm /usr/sbin/chroot /ubuntu_home/home/r/.local/var/pmbootstrap/chroot_rootfs_qemu-amd64 /bin/sh -c 'cd /;apk --no-progress add -u --virtual .pmbootstrap postmarketos-base ttf-droid device-qemu-amd64 device-qemu-amd64-kernel-stable postmarketos-ui-weston mesa-dri-virtio dbus chrony cryptsetup eudev haveged htop iw nano openssh postmarketos-mkinitfs sudo tmux util-linux wpa_supplicant shadow networkmanager fontconfig linux-postmarketos-stable weston-clients weston weston-shell-desktop weston-backend-fbdev weston-backend-drm weston-backend-x11 weston-terminal weston-xwayland xorg-server-xwayland postmarketos-demos postmarketos-artwork-wallpapers llvm5-libs libdrm libelf expat libgcc mesa-glapi libstdc++ dbus-libs libcap cryptsetup-libs popt libuuid udev-init-scripts eudev-libs libblkid kmod ncurses-libs libnl3 libmagic openssh-client openssh-sftp-server openssh-server busybox-extras lddtree multipath-tools postmarketos-splash device-mapper parted e2fsprogs e2fsprogs-extra osk-sdl charging-sdl triggerhappy ncurses-terminfo libevent findmnt libfdisk libmount libsmartcols pcsc-lite-libs linux-pam dhcpcd iptables libcurl dbus-glib glib libintl jansson libmm-glib libndp newt nspr nss polkit readline libteam freetype mesa-egl mesa-gles cairo libjpeg-turbo pango pixman libpng wayland-libs-client wayland-libs-cursor wayland-libs-egl libwebp libxkbcommon libinput-libs libunwind wayland-libs-server libweston libweston-desktop mesa-gbm libva mesa-dri-swrast libx11 libxcb libxcursor font-misc-misc font-cursor-misc xkeyboard-config xkbcomp xinit mesa-gl libxau libxdmcp libxfont2 libepoxy libxshmfence gtk+3.0 libffi libpciaccess argon2-libs device-mapper-libs json-c xz-libs ncurses-terminfo-base openssh-keygen openssh-server-common libaio userspace-rcu ttf-dejavu py3-pillow lvm2-libs libcom_err e2fsprogs-libs sdl2 sdl2_ttf directfb libmnl libnftnl-libs ca-certificates nghttp2-libs libssh2 pcre slang sqlite-libs libdaemon libnl3-cli libbz2 libxext libxrender libxft fribidi harfbuzz libevdev mtdev libxfixes encodings font-alias mkfontscale mkfontdir util-macros libxkbfile xauth mcookie xmodmap xrdb libxdamage libxxf86vm libbsd libfontenc shared-mime-info gtk-update-icon-cache libxcomposite libxi libxinerama libxrandr atk at-spi2-atk cairo-gobject cups-libs gdk-pixbuf py3-olefile lcms2 openjpeg python3 tiff device-mapper-event-libs tslib graphite2 libxmu mcpp libxml2 hicolor-icon-theme at-spi2-core avahi-libs gnutls gdbm libxt mcpp-libs libxtst gmp nettle p11-kit libtasn1 libunistring libice libsm'
31748 31747 31738 apk --no-progress add -u --virtual .pmbootstrap postmarketos-base ttf-droid device-qemu-amd64 device-qemu-amd64-kernel-stable postmarketos-ui-weston mesa-dri-virtio dbus chrony cryptsetup eudev haveged htop iw nano openssh postmarketos-mkinitfs sudo tmux util-linux wpa_supplicant shadow networkmanager fontconfig linux-postmarketos-stable weston-clients weston weston-shell-desktop weston-backend-fbdev weston-backend-drm weston-backend-x11 weston-terminal weston-xwayland xorg-server-xwayland postmarketos-demos postmarketos-artwork-wallpapers llvm5-libs libdrm libelf expat libgcc mesa-glapi libstdc++ dbus-libs libcap cryptsetup-libs popt libuuid udev-init-scripts eudev-libs libblkid kmod ncurses-libs libnl3 libmagic openssh-client openssh-sftp-server openssh-server busybox-extras lddtree multipath-tools postmarketos-splash device-mapper parted e2fsprogs e2fsprogs-extra osk-sdl charging-sdl triggerhappy ncurses-terminfo libevent findmnt libfdisk libmount libsmartcols pcsc-lite-libs linux-pam dhcpcd iptables libcurl dbus-glib glib libintl jansson libmm-glib libndp newt nspr nss polkit readline libteam freetype mesa-egl mesa-gles cairo libjpeg-turbo pango pixman libpng wayland-libs-client wayland-libs-cursor wayland-libs-egl libwebp libxkbcommon libinput-libs libunwind wayland-libs-server libweston libweston-desktop mesa-gbm libva mesa-dri-swrast libx11 libxcb libxcursor font-misc-misc font-cursor-misc xkeyboard-config xkbcomp xinit mesa-gl libxau libxdmcp libxfont2 libepoxy libxshmfence gtk+3.0 libffi libpciaccess argon2-libs device-mapper-libs json-c xz-libs ncurses-terminfo-base openssh-keygen openssh-server-common libaio userspace-rcu ttf-dejavu py3-pillow lvm2-libs libcom_err e2fsprogs-libs sdl2 sdl2_ttf directfb libmnl libnftnl-libs ca-certificates nghttp2-libs libssh2 pcre slang sqlite-libs libdaemon libnl3-cli libbz2 libxext libxrender libxft fribidi harfbuzz libevdev mtdev libxfixes encodings font-alias mkfontscale mkfontdir util-macros libxkbfile xauth mcookie xmodmap xrdb libxdamage libxxf86vm libbsd libfontenc shared-mime-info gtk-update-icon-cache libxcomposite libxi libxinerama libxrandr atk at-spi2-atk cairo-gobject cups-libs gdk-pixbuf py3-olefile lcms2 openjpeg python3 tiff device-mapper-event-libs tslib graphite2 libxmu mcpp libxml2 hicolor-icon-theme at-spi2-core avahi-libs gnutls gdbm libxt mcpp-libs libxtst gmp nettle p11-kit libtasn1 libunistring libice libsm

Wait for the timeout as it is downloading a large package such as the linux kernel. Notice that it only kills 1/3 processes.

(031738) [17:16:32] Process did not write any output for 15.0 seconds. Killing it.
(031738) [17:16:32] NOTE: The timeout can be increased with 'pmbootstrap -t'.
(031738) [17:16:32] % sudo kill -9 31746

$ps xao pid,ppid,pgid,cmd
31747 31746 31738 /bin/sh -c CHARSET=UTF-8 HISTFILE='~/.ash_history' PATH=/usr/lib/ccache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SHELL=/bin/ash TERM=xterm /usr/sbin/chroot /ubuntu_home/home/r/.local/var/pmbootstrap/chroot_rootfs_qemu-amd64 /bin/sh -c 'cd /;apk --no-progress add -u --virtual .pmbootstrap postmarketos-base ttf-droid device-qemu-amd64 device-qemu-amd64-kernel-stable postmarketos-ui-weston mesa-dri-virtio dbus chrony cryptsetup eudev haveged htop iw nano openssh postmarketos-mkinitfs sudo tmux util-linux wpa_supplicant shadow networkmanager fontconfig linux-postmarketos-stable weston-clients weston weston-shell-desktop weston-backend-fbdev weston-backend-drm weston-backend-x11 weston-terminal weston-xwayland xorg-server-xwayland postmarketos-demos postmarketos-artwork-wallpapers llvm5-libs libdrm libelf expat libgcc mesa-glapi libstdc++ dbus-libs libcap cryptsetup-libs popt libuuid udev-init-scripts eudev-libs libblkid kmod ncurses-libs libnl3 libmagic openssh-client openssh-sftp-server openssh-server busybox-extras lddtree multipath-tools postmarketos-splash device-mapper parted e2fsprogs e2fsprogs-extra osk-sdl charging-sdl triggerhappy ncurses-terminfo libevent findmnt libfdisk libmount libsmartcols pcsc-lite-libs linux-pam dhcpcd iptables libcurl dbus-glib glib libintl jansson libmm-glib libndp newt nspr nss polkit readline libteam freetype mesa-egl mesa-gles cairo libjpeg-turbo pango pixman libpng wayland-libs-client wayland-libs-cursor wayland-libs-egl libwebp libxkbcommon libinput-libs libunwind wayland-libs-server libweston libweston-desktop mesa-gbm libva mesa-dri-swrast libx11 libxcb libxcursor font-misc-misc font-cursor-misc xkeyboard-config xkbcomp xinit mesa-gl libxau libxdmcp libxfont2 libepoxy libxshmfence gtk+3.0 libffi libpciaccess argon2-libs device-mapper-libs json-c xz-libs ncurses-terminfo-base openssh-keygen openssh-server-common libaio userspace-rcu ttf-dejavu py3-pillow lvm2-libs libcom_err e2fsprogs-libs sdl2 sdl2_ttf directfb libmnl libnftnl-libs ca-certificates nghttp2-libs libssh2 pcre slang sqlite-libs libdaemon libnl3-cli libbz2 libxext libxrender libxft fribidi harfbuzz libevdev mtdev libxfixes encodings font-alias mkfontscale mkfontdir util-macros libxkbfile xauth mcookie xmodmap xrdb libxdamage libxxf86vm libbsd libfontenc shared-mime-info gtk-update-icon-cache libxcomposite libxi libxinerama libxrandr atk at-spi2-atk cairo-gobject cups-libs gdk-pixbuf py3-olefile lcms2 openjpeg python3 tiff device-mapper-event-libs tslib graphite2 libxmu mcpp libxml2 hicolor-icon-theme at-spi2-core avahi-libs gnutls gdbm libxt mcpp-libs libxtst gmp nettle p11-kit libtasn1 libunistring libice libsm'
31748 31747 31738 apk --no-progress add -u --virtual .pmbootstrap postmarketos-base ttf-droid device-qemu-amd64 device-qemu-amd64-kernel-stable postmarketos-ui-weston mesa-dri-virtio dbus chrony cryptsetup eudev haveged htop iw nano openssh postmarketos-mkinitfs sudo tmux util-linux wpa_supplicant shadow networkmanager fontconfig linux-postmarketos-stable weston-clients weston weston-shell-desktop weston-backend-fbdev weston-backend-drm weston-backend-x11 weston-terminal weston-xwayland xorg-server-xwayland postmarketos-demos postmarketos-artwork-wallpapers llvm5-libs libdrm libelf expat libgcc mesa-glapi libstdc++ dbus-libs libcap cryptsetup-libs popt libuuid udev-init-scripts eudev-libs libblkid kmod ncurses-libs libnl3 libmagic openssh-client openssh-sftp-server openssh-server busybox-extras lddtree multipath-tools postmarketos-splash device-mapper parted e2fsprogs e2fsprogs-extra osk-sdl charging-sdl triggerhappy ncurses-terminfo libevent findmnt libfdisk libmount libsmartcols pcsc-lite-libs linux-pam dhcpcd iptables libcurl dbus-glib glib libintl jansson libmm-glib libndp newt nspr nss polkit readline libteam freetype mesa-egl mesa-gles cairo libjpeg-turbo pango pixman libpng wayland-libs-client wayland-libs-cursor wayland-libs-egl libwebp libxkbcommon libinput-libs libunwind wayland-libs-server libweston libweston-desktop mesa-gbm libva mesa-dri-swrast libx11 libxcb libxcursor font-misc-misc font-cursor-misc xkeyboard-config xkbcomp xinit mesa-gl libxau libxdmcp libxfont2 libepoxy libxshmfence gtk+3.0 libffi libpciaccess argon2-libs device-mapper-libs json-c xz-libs ncurses-terminfo-base openssh-keygen openssh-server-common libaio userspace-rcu ttf-dejavu py3-pillow lvm2-libs libcom_err e2fsprogs-libs sdl2 sdl2_ttf directfb libmnl libnftnl-libs ca-certificates nghttp2-libs libssh2 pcre slang sqlite-libs libdaemon libnl3-cli libbz2 libxext libxrender libxft fribidi harfbuzz libevdev mtdev libxfixes encodings font-alias mkfontscale mkfontdir util-macros libxkbfile xauth mcookie xmodmap xrdb libxdamage libxxf86vm libbsd libfontenc shared-mime-info gtk-update-icon-cache libxcomposite libxi libxinerama libxrandr atk at-spi2-atk cairo-gobject cups-libs gdk-pixbuf py3-olefile lcms2 openjpeg python3 tiff device-mapper-event-libs tslib graphite2 libxmu mcpp libxml2 hicolor-icon-theme at-spi2-core avahi-libs gnutls gdbm libxt mcpp-libs libxtst gmp nettle p11-kit libtasn1 libunistring libice libsm

This patch recursively kills the PID of the command and its child processes.

There is a simpler method to kill processes by group ID but that doesn't seem to be usable for this particular case. The subprocess.Popen has a start_new_session[1] argument which will run the command under a new session by calling setsid[2]. The command will have its own group ID which can be used to kill all of the child processes at once. But by using setsid the process has no controlling terminal. It looks like this is something we don't want because the command should be a subprocess of pmbootstrap.py otherwise the terminal interaction will break.

[1] https://docs.python.org/3.4/library/subprocess.html?highlight=subprocess#popen-constructor
[2] https://linux.die.net/man/2/setsid

changed the description

By ryang on 2018-09-25T14:35:36

added 1 commit

• a631f8d3 - Kill the child processes spawned by a run command

Compare with previous version

By ryang on 2018-09-25T15:35:39

Busybox' ps doesn't have these fancy flags, so when running on Alpine one would need to install procps. Would you like to add that to the README.md (best would probably be after rebaseing, as I've just modified the README with another MR)?

I'm running into this timout-does-not-kill-all-child-processes problem every now and then on the build machine, so I highly appreciate this fix. Also this is one of the most high quality merge requests I have seen for pmbootstrap, thank you so much for this @ryang2678!

By Oliver Smith on 2018-09-26T07:28:41

Idea for follow up patches: how about we also kill all processes on...

• pressing ^C (all child processes of this pmbootstrap instance)
• pmbootstrap shutdown (before umounting everything etc., all processes running inside the chroot, but I am not sure what to use as parent then, we might need different logic for that)

By Oliver Smith on 2018-09-26T07:31:09

added 2 commits

• f68a7a6b - 1 commit from branch postmarketOS:master
• 0c81a6d9 - Kill the child processes spawned by a run command

Compare with previous version

By ryang on 2018-09-26T20:08:25

added 1 commit

• 155c57d7 - Kill the child processes spawned by a run command

Compare with previous version

By ryang on 2018-09-26T20:17:38

added 1 commit

• a7e1faf8 - Kill the child processes spawned by a run command

Compare with previous version

By ryang on 2018-09-26T20:19:25

pressing ^C (all child processes of this pmbootstrap instance)

According to my searches pressing ^C sends SIGINT to the entire process group[1]. For the example I've used above in which apk add processes are still running. ^C will cause all the child processes to exit.

That brings up the question of whether pmbootstrap should allow processes to exit cleanly. i.e. Send SIGTERM. Wait. Then send SIGKILL to kill hung processes.

pmbootstrap shutdown (before umounting everything etc., all processes running inside the chroot, but I am not sure what to use as parent then, we might need different logic for that)

/proc/pid/root points to the chroot directory. Compare that against the work directory.

[1] https://stackoverflow.com/a/6804155

By ryang on 2018-09-26T21:47:01

resolved all discussions

By ryang on 2018-10-01T16:40:11

mentioned in issue #1673 (closed)

By Oliver Smith on 2018-10-02T07:05:26

Excellent work, thanks again @ryang2678!

I've created #1673 (closed) to track the idea of killing processes on pmbootstrap shutdown.

By Oliver Smith on 2018-10-02T07:06:32

approved this merge request

By Oliver Smith on 2018-10-02T07:06:43

added 2 commits

• 3e7c95e8 - 1 commit from branch postmarketOS:master
• 277854e8 - Kill the child processes spawned by a run command

Compare with previous version

By Oliver Smith on 2018-10-02T07:06:53

enabled an automatic merge when the pipeline for 277854e8 succeeds

By Oliver Smith on 2018-10-02T07:07:11

(PS: if you didn't see it, I'm a bit cought up in pmaports#103 (closed), hence the long answering delay.)

EDIT: disregard that, you've contributed to that issue as well

By Oliver Smith on 2018-10-02T07:20:50

merged

By Oliver Smith on 2018-10-02T07:11:48