Skip to content
Snippets Groups Projects
New look: Off

google-sargo: Device lockup after some usage

    • Closed Imported Issue created by Administrator @root

    Describe your issue

    The entire Pixel 3a appears to lockup and stop responding after some amount of usage. I can't reliably figure out a way to trigger the lock up, but I have observed it to happen sometimes coming back from suspend (after about a few seconds where it works), on a fresh boot after a few minutes, launching apps, etc.

    Of note is that this appears to have started happening in the past two weeks on edge. I don't experience this issue on v24.06. It makes me suspect some package update recently may have caused it.

    I could only so far trigger this with Plasma Mobile, I am not sure if it's a Plasma issue or it's exacerbated by Plasma's heavier use of the GPU over other DEs (UPDATE: I just experienced this issue while in gdm, not logged into any session so it's probably not Plasma specific). Of note is that I only encounter this lockup issue on the Pixel 3a specifically, and not on my OnePlus 6.

    What's the expected behaviour?

    The device does not lockup.

    What's the current behaviour?

    After some usage, the entire device locks up, with even SSH being unresponsive. I can't switch to the TTY either.

    How to reproduce your issue?

    Use the device for a bit with Plasma Mobile, and you will probably experience it.

    What device are you using?

    google-sargo

    On what postmarketOS version did you encounter the issue?

    • edge (master branch)

    • v24.06

    • v23.12 (supported until 2024-07-16)

    • I confirm that the issue still is present after running sudo apk upgrade -a

    On what environment did you encounter the issue?

    Environments

    • GNOME Shell on Mobile
    • Phosh
    • Plasma Mobile
    • Sxmo (Wayland/Sway) Please post the output of sxmo_version.sh
    • Other: Please fill out

    How did you get postmarketOS image?

    What's the build date of the image? (in yyyy-mm-dd format)

    2024-07-25

    Additional information

    Please let me know what sort of logs I can get to help debug this issue.

    6 of 12 checklist items completed · Edited
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    • Administrator marked the checklist item GNOME Shell on Mobile as completed · Imported

      marked the checklist item GNOME Shell on Mobile as completed

      By espidev on 2024-07-27T03:39:12

    • Administrator added release-edge type::support ui-gnome-mobile labels · Imported

      added release-edge type::support ui-gnome-mobile labels

      • Administrator
        Administrator @root · Imported
        Author Owner

        I followed logs with logread, and here are the final logs before the system freezes:

        [Jul 27 00:28:17] daemon supervise-daemon[1019]: /usr/bin/hexagonrpcd, pid 7911, exited with return code 4 
[Jul 27 00:28:17] daemon supervise-daemon[7913]: Child command line: /usr/libexec/iio-sensor-proxy  
[Jul 27 00:28:17] daemon supervise-daemon[2734]: /usr/libexec/iio-sensor-proxy, pid 7913, exited with return code 0 
[Jul 27 00:28:18] user pulseaudio[5269]: [pulseaudio] alsa-ucm.c: Failed to set verb HiFi 
[Jul 27 00:28:18] user pulseaudio[5269]: [pulseaudio] udev-util.c: Failed to get card object. 
[Jul 27 00:28:18] user pulseaudio[5269]: [pulseaudio] module-alsa-card.c: Failed to find a working profile. 
[Jul 27 00:28:18] user pulseaudio[5269]: [pulseaudio] module.c: Failed to load module "module-alsa-card" (argument: "device_id="0" name="platform-sound" card_name="alsa_card.platform-sound" namereg_fail=false tsched=yes fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes avoid_resampling=no card_properties="module-udev-detect.discovered=1""): initialization failed. 
[Jul 27 00:28:19] daemon supervise-daemon[7924]: Child command line: /usr/bin/hexagonrpcd -f /dev/fastrpc-adsp -d adsp -s -R /usr/share/qcom/sdm670/Google/sargo  
[Jul 27 00:28:19] daemon supervise-daemon[1019]: /usr/bin/hexagonrpcd, pid 7924, exited with return code 4 
[Jul 27 00:28:19] daemon supervise-daemon[2734]: /usr/libexec/iio-sensor-proxy, pid 7944, terminated by signal 11 <<<<<<<<

        Perhaps it has something to do with the sensors that got enabled recently? Since iio-sensor-proxy segfaults

        By espidev on 2024-07-27T04:36:35

        Edited by Administrator
      • Administrator
        Administrator @root · Imported
        Author Owner

        I spoke too soon, it's probably not related to iio-sensor-proxy since I disabled the service and still get system lockup a few seconds after waking from suspend with these logs:

        [Jul 27 00:35:22] daemon dbus-daemon[1796]: [system] Activating service name='org.freedesktop.nm_dispatcher' requested by ':1.3' (uid=0 pid=2006 comm="/usr/sbin/NetworkManager -n") (using servicehelper) 
[Jul 27 00:35:22] daemon dbus-daemon[1796]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' 
[Jul 27 00:35:23] daemon supervise-daemon[4990]: Child command line: /usr/bin/hexagonrpcd -f /dev/fastrpc-adsp -d adsp -s -R /usr/share/qcom/sdm670/Google/sargo  
[Jul 27 00:35:23] daemon supervise-daemon[1020]: /usr/bin/hexagonrpcd, pid 4990, exited with return code 4 
[Jul 27 00:35:25] user pulseaudio[3804]: [pulseaudio] alsa-ucm.c: Failed to set verb HiFi 
[Jul 27 00:35:25] user pulseaudio[3804]: [pulseaudio] udev-util.c: Failed to get card object. 
[Jul 27 00:35:25] user pulseaudio[3804]: [pulseaudio] module-alsa-card.c: Failed to find a working profile. 
[Jul 27 00:35:25] user pulseaudio[3804]: [pulseaudio] module.c: Failed to load module "module-alsa-card" (argument: "device_id="0" name="platform-sound" card_name="alsa_card.platform-sound" namereg_fail=false tsched=yes fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes avoid_resampling=no card_properties="module-udev-detect.discovered=1""): initialization failed. 
[Jul 27 00:35:25] daemon supervise-daemon[5003]: Child command line: /usr/bin/hexagonrpcd -f /dev/fastrpc-adsp -d adsp -s -R /usr/share/qcom/sdm670/Google/sargo

        By espidev on 2024-07-27T04:36:26

      • Administrator
        Administrator @root · Imported
        Author Owner

        It's because of the service for FastRPC. The FastRPC service is used to serve some proprietary files to the DSP, and was enabled to support sensors.

        Workaround:

        # rc-service hexagonrpcd-adsp-sensorspd stop
# rc-update del hexagonrpcd-adsp-sensorspd

        We may need to stop enabling it by default and add an entry on the wiki to give directions on how to enable/disable sensor support. Beyond that, suspend with sensors seems like a long-term issue.

        Offending code: !5329 (diffs)

        Relevant dmesg -w lines for reference:

        [  153.638374] PM: suspend exit
[  153.677671] qcom_q6v5_pas 62400000.remoteproc: fatal error received: err_qdi.c:450:EX:sensor_process:1:frpc_dsp:c4:PC=b2086e14
[  153.677711] remoteproc remoteproc1: crash detected in 62400000.remoteproc: type fatal error
[  153.677732] remoteproc remoteproc1: handling crash #1 in 62400000.remoteproc
[  153.677738] remoteproc remoteproc1: recovering 62400000.remoteproc
[  153.792857] clk_unregister: unregistering prepared clock: LPASS_CLK_ID_INT_MCLK_0
[  153.873831] remoteproc remoteproc1: stopped remote processor 62400000.remoteproc
[  154.090725] remoteproc remoteproc1: remote processor 62400000.remoteproc is now up
[  154.095195] qcom,fastrpc 62400000.remoteproc:glink-edge.fastrpcglink-apps-dsp.-1.-1: no reserved DMA memory for FASTRPC
[  154.096606] qcom,fastrpc-cb 62400000.remoteproc:glink-edge:fastrpc:compute-cb@3: Adding to iommu group 13
[  154.098069] qcom,fastrpc-cb 62400000.remoteproc:glink-edge:fastrpc:compute-cb@4: Adding to iommu group 14
[  154.099723] qcom,fastrpc-cb 62400000.remoteproc:glink-edge:fastrpc:compute-cb@5: Adding to iommu group 15
[  154.102265] qcom,fastrpc-cb 62400000.remoteproc:glink-edge:fastrpc:compute-cb@6: Adding to iommu group 16
[  154.138497] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:9
[  154.139853] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:a
[  154.140698] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:b
[  154.142516] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:3
[  154.142796] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:4
[  154.150728] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:7
[  154.152064] q6asm-dai 62400000.remoteproc:glink-edge:apr:service@7:dais: Adding to iommu group 17
[  154.154188] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:8
[  154.170249] qcom,pm8916-wcd-spmi-codec c440000.spmi:pmic@3:audio-codec@f000: PMIC REV: 0	CODEC Version: 4
[  154.580902] qcom-q6asm aprsvc:service:4:7: DSP returned error[1]
[  154.580931] q6asm-dai 62400000.remoteproc:glink-edge:apr:service@7:dais: Memory_map_regions failed
[  154.580941] q6asm-dai 62400000.remoteproc:glink-edge:apr:service@7:dais: Audio Start: Buffer Allocation failed rc = -22
[  154.580950] q6asm-dai 62400000.remoteproc:glink-edge:apr:service@7:dais: ASoC: error at snd_soc_pcm_component_prepare on 62400000.remoteproc:glink-edge:apr:service@7:dais: -12
[  154.580962]  MultiMedia1: ASoC: error at __soc_pcm_prepare on MultiMedia1: -12
[  154.580974]  MultiMedia1: ASoC: error at dpcm_fe_dai_prepare on MultiMedia1: -12
[  194.955440] qcom_q6v5_pas 62400000.remoteproc: watchdog received: SFR Init: wdog or kernel error suspected.
[  194.955498] remoteproc remoteproc1: crash detected in 62400000.remoteproc: type watchdog
[  194.955669] remoteproc remoteproc1: handling crash #2 in 62400000.remoteproc
[  194.955680] remoteproc remoteproc1: recovering 62400000.remoteproc
[  199.580262] qcom-q6asm aprsvc:service:4:7: CMD 10d92 timeout
[  199.580296] q6asm-dai 62400000.remoteproc:glink-edge:apr:service@7:dais: Memory_map_regions failed
[  199.580309] q6asm-dai 62400000.remoteproc:glink-edge:apr:service@7:dais: Audio Start: Buffer Allocation failed rc = -110
[  199.580323] q6asm-dai 62400000.remoteproc:glink-edge:apr:service@7:dais: ASoC: error at snd_soc_pcm_component_prepare on 62400000.remoteproc:glink-edge:apr:service@7:dais: -12
[  199.580340]  MultiMedia2: ASoC: error at __soc_pcm_prepare on MultiMedia2: -12
[  199.580357]  MultiMedia2: ASoC: error at dpcm_fe_dai_prepare on MultiMedia2: -12
[  199.633997] qcom-q6afe aprsvc:service:4:4: packet not transmitted (-5)
[  199.634025] qcom-q6afe aprsvc:service:4:4: AFE close failed -22
[  199.634041] q6afe-dai 62400000.remoteproc:glink-edge:apr:service@4:dais: fail to close AFE port (-22)
[  199.634121] qcom-q6afe aprsvc:service:4:4: packet not transmitted (-5)
[  199.634131] qcom-q6afe aprsvc:service:4:4: AFE set params failed -22
[  199.634141] q6afe-dai 62400000.remoteproc:glink-edge:apr:service@4:dais: ASoC: error at snd_soc_dai_set_sysclk on INT0_MI2S_RX: -22
[  199.728690] remoteproc remoteproc1: stopped remote processor 62400000.remoteproc
[  199.956269] remoteproc remoteproc1: remote processor 62400000.remoteproc is now up
[  199.959885] qcom,fastrpc 62400000.remoteproc:glink-edge.fastrpcglink-apps-dsp.-1.-1: no reserved DMA memory for FASTRPC
[  199.961864] qcom,fastrpc-cb 62400000.remoteproc:glink-edge:fastrpc:compute-cb@3: Adding to iommu group 13
[  199.963353] qcom,fastrpc-cb 62400000.remoteproc:glink-edge:fastrpc:compute-cb@4: Adding to iommu group 14
[  199.964685] qcom,fastrpc-cb 62400000.remoteproc:glink-edge:fastrpc:compute-cb@5: Adding to iommu group 15
[  199.965960] qcom,fastrpc-cb 62400000.remoteproc:glink-edge:fastrpc:compute-cb@6: Adding to iommu group 16
[  199.999717] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:9
[  200.000896] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:a
[  200.001247] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:b
[  200.002113] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:3
[  200.002356] qcom,apr 62400000.remoteproc:glink-edge.apr_audio_svc.-1.-1: Adding APR/GPR dev: aprsvc:service:4:4
[  200.007691] Unable to handle kernel write to read-only memory at virtual address ffffc64510b3af34

        By Richard Acayan on 2024-07-27T18:06:01

      • Administrator
        Administrator @root · Imported
        Author Owner

        Stopping the service indeed does appear to fix it

        By espidev on 2024-07-27T20:45:21

      • Administrator
        Administrator @root · Imported
        Author Owner

        Beyond that, suspend with sensors seems like a long-term issue.

        Maybe we should instead ship a config to disable suspend by default? This was done in other devices, and given it has issues with remote processors in most Android device, this might be best? Enabling it is always a couple touches away?

        By Pablo Correa Gomez on 2024-07-29T19:49:40

      • Administrator
        Administrator @root · Imported
        Author Owner

        That sounds like a good idea. I think we still need to remove the post-upgrade script, just so advanced users can keep working suspend (regressions aren't good) without having to keep disabling the service.

        By Richard Acayan on 2024-07-29T20:50:56

      • Please register or sign in to reply
    • Administrator marked the checklist item Phosh as completed · Imported

      marked the checklist item Phosh as completed

      By espidev on 2024-07-27T04:41:04

    • Administrator
      Administrator @root · Imported
      Author Owner

      I've been able to now reproduce it in Phosh too pretty reliably:

      1. Use the phone for a bit
      2. Run sudo loginctl suspend over ssh
      3. Wake the phone and use it for a few seconds, then it locks up

      I have plasma mobile also installed on the drive during this test as well, I will try again tomorrow with a fresh flash to check if this issue occurs.

      By espidev on 2024-07-27T04:42:50

      Edited by Administrator
    • Administrator
      Administrator @root · Imported
      Author Owner

      @flamingradian could you look into this?

      By Robert Eckelmann on 2024-07-27T04:53:33

    • Administrator
      Administrator @root · Imported
      Author Owner

      I have seen the same thing on my Pixel 3a. I so far have only see it with Plasma Mobile

      By Alistair Francis on 2024-07-27T07:33:09

    • Administrator mentioned in merge request !5400 (merged) · Imported

      mentioned in merge request !5400 (merged)

      By Richard Acayan on 2024-07-29T21:19:32

    • Administrator
      Administrator @root · Imported
      Author Owner

      Now I'm curious about the underlying bug. Does dmesg report a different remoteproc crash (e.g. SLPI - Sensor Low Power Island) on the OnePlus 6?

      The lack of a system-wide hang doesn't imply that there is no remoteproc crash. I tried adding CDSP and running HexagonRPCD on it, and CDSP crashed, but the device kept working.

      By Richard Acayan on 2024-07-29T23:56:42

    • Administrator
      Administrator @root · Imported
      Author Owner

      I'm considering this issue fixed as of !5400 (merged) Let me know if you disagree and I should re-open

      By Pablo Correa Gomez on 2024-07-30T22:37:03

    • Administrator closed · Imported

      closed

    • Administrator removed ui-gnome-mobile label · Imported

      removed ui-gnome-mobile label

    • Administrator added category::device device-category::testing type::bug labels and removed type::support label · Imported

      added category::device device-category::testing type::bug labels and removed type::support label

    • Administrator added type::support label and removed category::device device-category::testing type::bug labels · Imported

      added type::support label and removed category::device device-category::testing type::bug labels

    • Administrator added category::device device-category::testing type::bug labels and removed type::support label · Imported

      added category::device device-category::testing type::bug labels and removed type::support label

    Please register or sign in to reply