pmbootstrap/pmOS should offer a 'debug mode'

This debug mode would produce a pmOS install that

• disables splash screens on boot
• prints kernel and openrc messages to screen (pass PMOS_NO_OUTPUT_REDIRECT kernel cmdline)
• disables fde
• adds the usb hook to initramfs.
• pmbootstrap presents a big fat warning that pmos is running in 'debug mode' and should not be used for production
• modify /etc/os-release to indicate that it's pmOS-debug or something
• This would be a (non-default) parameter to pmbootstrap install, like --debug-mode or -d for short (assuming d isn't already taken.. I didn't look)
• do not install any UI
• add some kernel cmdline to have a more verbose kernel log
• install all possible -dbg subpackages?
• Install some extra tools to help with debugging:
  • evtest
  • fftest
  • gdb-server (not sure it works for armhf)
  • strace
  • sysstat (performance monitoring tools)
  • procps
  • nmap

Anything else that would be useful? Any additional tools that could be installed automatically when this is enabled to assist folks with porting/debugging their device?

@craftyguy nice idea, here some ideas that can be added to the list:

• do not install any UI
• add some kernel cmdline to have a more verbose kernel log
• install all possible -dbg subpackages?
• possible extra tools:
  • gdb-server (not sure it works for armhf)
  • strace
  • sysstat (performance monitoring tools)
  • procps
  • nmap

By Daniele Debernardi on 2017-10-17T16:27:26

@drebrez great suggestions! I have combined your list with the original one in my comment to make it easier to parse out the 'specification' for this request!

By clayton craft on 2017-10-17T16:30:31

Created by: ollieparanoid

Cool idea!

prints kernel and openrc messages to screen

I've tested that about half a year ago with my lg-mako and it was basically unusable. it took 5 or so? minutes until it printed anything to the screen (I randomly left it powered on but I thought the image didn't even boot). So I think we can't enable that for all devices, at least not with the standard stuff from the kernel. But that may also be lg-mako specific and it may work fine with other Androids.

By * postmarketOS Bot on 2017-10-17T16:53:31

@ollieparanoid

prints kernel and openrc messages to screen

Yea I was thinking this might actually just be setting the PMOS_NO_OUTPUT_REDIRECT, since that allows for quite a few useful messages to be printed to screen. Is that what you did in your test with Mako, or did you set things like debug on the kernel cmdline? That would slow down all but the beefiest systems.

Oh, maybe adding earlyprintk would be useful too!

By clayton craft on 2017-10-17T16:58:13

Created by: ollieparanoid

What I did was enabling the print of console messages to the screen at all. By default all Android devices have that deactivated. I didn't specify debug on purpose, but it might have been there by default, I am not sure (6 months ago pmbootstrap was a bit different, PMOS_NO_OUTPUT_REDIRECT had not been invented yet :p)

By * postmarketOS Bot on 2017-10-17T17:26:17

@craftyguy in chat we discussed about some tool to debug the framebuffer (#1188 (closed)) and adding them to the debug-shell. We can consider adding also some other tools that we mentioned here, like strace, evtest, ...

By Daniele Debernardi on 2018-02-01T11:13:37

@z3ntu in chat suggested the fftest tool used for testing force feedback (=vibration) devices.

By Daniele Debernardi on 2018-03-21T22:25:19

Created by: ollieparanoid

The debug-shell hook fulfills most of these ideas now - it displays a splash, adds lots of debugging tools, starts the telnet server etc.

Is this close enough, or do we still need to do something here? (@craftyguy and everyone else)

By * postmarketOS Bot on 2018-03-26T18:19:18

One thing that might be worth to consider to be added to the debug-shell hook (at least for devices with a keyboard attached) is to open a shell on a different vt, like the "systemd.debug-shell=1" opens one at tty9.

Probably something like this:

/bin/busybox openvt -c 9 /bin/sh

By Bernhard on 2018-07-23T19:01:04

Interesting, what's the advantage of that?

By Oliver Smith on 2018-07-23T22:28:29

The advantage would be to be able to inspect e.g. dmesg, mounts or /pmOS_init.log. Also checking if the network interface would be up and which ip is assigned would be possible.

PMOS_NO_OUTPUT_REDIRECT opens a shell but as far as I see one has to leave that to continue and if the problem is later in the boot process e.g. mounting boot partition I did not knew some way to find out what is going on.

In my case the boot partition could not be mounted because of missing filesystem drivers. Without debug-shell I just got the splash with "initramfs-extra not found". With debug-shell I also got just the splash finding the network interface not yet initialized. The mount failed with "No such file ..." but device node and mount point looked ok. But /proc/filesystems missed the ext entries and modprobe ext4 gave a missing fscrypto.ko.gz. Inspecting that interactively did help here much. And just adding filesystem drivers in deviceinfo got me further.

Creating such a shell at really early stages would miss the device therefore this should be more complete:

/bin/busybox mknod /dev/tty9 c 4 9
/bin/busybox openvt -c 9 /bin/sh

Such a shell did stop working when booted up into the weston demo (could not find any file anymore). So if that should work when booted up it needs probably to be restarted after switching root.

By Bernhard on 2018-07-24T09:05:08

One small addition: When adding the debug-shell it appears to be stuck in the boot showing the splash and a blinking cursor. After entering "exit" boot continued. So debug-shell should probably disable the splash?

By Bernhard on 2018-07-24T09:54:45

Thank you very much for the detailed explanation. That makes a lot of sense indeed.

Such a shell did stop working when booted up into the weston demo (could not find any file anymore).

Yeah, we run switch_root at the end of the init shell script, which deletes everything that the initramfs loads into RAM, and mounts the real filesystem.

When adding the debug-shell it appears to be stuck in the boot showing the splash and a blinking cursor. After entering "exit" boot continued. So debug-shell should probably disable the splash?

Well, most devices don't have a TTY enabled. For them it's nice to have the splash screen, so they know that the debug-shell is active.

So from a user's perspective, I think it would be nice if the debug-shell hook checked if the device has a keyboard (we could read that from deviceinfo), and in case it has one:

• disable the splash screens
• disable redirect of all output to a log file

I think we could implement this, if we supported to load hooks at the very beginning of the init script (#1607 (closed)), before we would redirect the log.

Then we could override the setup_log() function.

If you are fine with spending some time on this specific issue, would you like to give it a shot?

By Oliver Smith on 2018-07-25T20:31:24

mentioned in merge request pmaports!1197 (merged)

By Daniele Debernardi on 2020-04-30T01:18:16