"QUALCOMM CrashDump Mode" on oneplus-enchilada

assigned to @calebccff

By Newbyte on 2024-04-18T18:28:34

[y] I built it using pmbootstrap

Did you enable the iwd wifi backend or are you using the default (wpa_supplicant)?

By clayton craft on 2024-04-19T01:24:17

I used whatever was the default (and recommended) in pmbootstrap. So, if it is the default, I used wpa_supplicant.

On another note, I am way more confident that it has to do with the modem. I wasn't able to start the device yesterday, and had to remove the sim to get it to boot. It does work with the sim sometimes though, so there is still the slight possibility that this is coincidence.

By Julian Groß on 2024-04-19T01:24:16

I just had this too on a oneplus 6t with a fresh install. I can confirm that after switching from iwd to wpa_supplicant (created a new image with pmbootstrap), it didn't boot into crashdump mode anymore and works fine now.

The kernel also works fine on my op6 with wpa_supplicant.

By Oliver Smith on 2024-04-19T17:17:52

this will be fixed with the next kernel upgrade (and iwd will work!)

By Caleb Connolly on 2024-04-19T18:06:56

You experienced this: #2440 (closed)

But below @JulianGro said they were using wpa_supplicant... so maybe what they reported here is a new issue?

Or maybe it's a case of something else besides iwd trying to load/use that faulty module that caused #2440 (closed)?

By clayton craft on 2024-04-19T17:45:39

oh yeah there is (probably multiple) other causes for the random crashdumps. but the specific one due to qcrypto will be fixed

By Caleb Connolly on 2024-04-19T18:06:22

well any crashes caused by qcrypto will be fixed cuz it will be disabled. Not sure if anything else makes uses of it

By Caleb Connolly on 2024-04-19T18:09:10

When trying Renegade Project, I would get the Qualcomm Crash Dump error consistently at boot if I installed a specific newer WiFi driver. No problems with the older/default WiFi driver though as long as I didn't need to reload the driver (disable and re-enable would BSoD).

I don't know what Windows is doing for the network stack or what the newer driver does, or if this is really related, but maybe debugging tools on Windows might show something undocumented? The newer crashing driver is "Qualcomm Technolgies, Incorporated - SYSTEM - 7/1/2009 1.0.105.40.0" and was in the Windows Update optional updates section on Win10 21H2.

By Sean Rhone on 2024-04-20T02:11:34

added device-oneplus-enchilada status::confirmed labels

I had another similar experience on a device recently where the modem just crashes in a loop repeatedly. It's possible there is some strange configuration issue going on. I'd suggest installing the latest stock Android images (instructions for this are on the wiki), inserting a SIM and checking everything works, and then reflashing postmarketOS.

By Caleb Connolly on 2024-05-26T03:07:29

got an OP6 256GB, was not able to edl older android onto it (it was always bricked). ended up with oxygen android 11.1.1, flashed pmos onto it and had a lot of crash dump screens out of nothing, thought that there is a hardware failure but each reboot ended in crash dump mode. so i took the time to figure out how to downgrade to android 9.0.8 on both slots (see 'Notes' section of the timeconsuming process in wiki) Now every reboot works! so far no crash dump screens at all! somehow there is something hidden which we still can not understand, but i put my hand into the fire that it is mandatory to first downgrade the os before flashing postmarketOS!!! having 2 op6, one with a blocked sim tray... there seems to be no trouble with or without sim card edit: Still random crashes, but way less.. will make dumps once a computer is near

By magdesign on 2024-06-02T13:56:38

it is mandatory to first downgrade the os before flashing postmarketOS!!!

no, and this is bad advice. Every instance of this bug I've seen points to a weird edgecase in the rmtfs daemon (or one of its friends). But without logs this is impossible to narrow down.

If you ever run into this again (repeated crashdumps that is) then it would be extremely useful if you could just run the edl command with no arguments. This will collect a full ram dump from the device which we can use to start narrowing down the cause.

I suspect that the relevant state is in the efs partition, this is where the modem NVRAM lives. We need a utility for postmarketOS to access this (it's encrypted at rest and decrypted at runtime by the modem, we can only read it via some QMI service).

Downgrading is generally a bad idea because there are a lot of variations this can introduce, which only make it harder for us to deal with bug reports.

By Caleb Connolly on 2024-06-01T06:07:11

I wonder if downgrading to Android 9, forcing it to both slots for consistent firmware, and then OTA updating twice (both slots) to the latest 11.1.2.2 would help?

I noticed when I msmdownloadtools'd to OOS 5.1.5 (Android 8?), wifi nor cellular worked, but it worked when I OTA'd to Android 9. That implies that something isn't upgraded or downgraded when using that tool. There's a SMT mode apparently to thoroughly flash stuff but I've heard it also wipes IMEI and that needs backed up and restored later.

I also wonder if there's something that can be done from official OOS or Android to refresh everything modem/radio related? Like wipe all nvram settings and let it re-populate stuff from Android and the carrier? Apparently there's something called AT!RMARESET=1 that can do that from qmcli?

By Sean Rhone on 2024-05-31T23:27:58

I've tested a few different combinations of version and UI to see what crashes - I can't see an obvious pattern although some setups are more crashy than others. Each of these was installed using pmbootstrap on up-to-date Arch linux and with no FDE. Between each different installation I used fastboot to format system and userdata as I was wondering if carry over from an old install had been causing the crashes - I don't think so now but I continued doing the formatting for consistency. In case it's relevant, I previously downgraded the original Android to the version suggested to try to get GPS working (no luck though).

In the list below any reboots or shutdowns were done from the UI, except if I've specified otherwise. I have memory dumps taken with edl for each place I've noted edl memdump - please let me know if/where you want any of these uploaded.

I'm not using this device for anything at the moment and can retest/test other combinations if needed.

Plasma mobile edge
- 6.13.0-rc2
- Firstboot - OK - configured WiFi and mobile data
- Reboot - crashdump
- Hard reboot (power button) - crashdump
- Boot into fastboot then select start - OK
- Shutdown then start - OK
- Reboot - crashdump - edl memdump1 - immediately booted OK after crashdump completed
- Shutdown then start - OK
- Shutdown then start - OK
- Hard reboot (hold power button) - crashdump
- Fastboot then start - crashdump
- Fastboot then start - crashdump - edl memdump2 - did not immediately boot cleanly (either didn't reboot or booted to crashdump)
- Fastboot then start - OK
Gnome mobile edge
- Firstboot OK - configured wifi and mobile data
- Reboot - OK
- Shutdown then start - OK
- Hard reboot (power button) - crashdump - edl memdump3 - immediately rebooted cleanly
- Reboot - OK
Plasma mobile v24.06
- 6.9.0
- Firstboot - crashdump - edl memdump4 - immediate reboot OK
- Shutdown then start - crashdump
- Hard reboot into fastboot then start - OK
- Reboot - crashdump
- Clean install - firstboot with no SIM - OK - configured wifi
- Reboot - crashdump - edl memdump5 - immediately rebooted cleanly
- Shutdown then start - OK
- Shutdown then start with SIM in - crashdump
Gnome mobile v24.06
- Firstboot - OK - configured wifi and mobile data
- Reboot - OK
- Shutdown then start - OK
- Hard reboot (power button) - OK
Fbkeyboard edge
- Firstboot - crashdump - edl memdump6 - rebooted OK but no touchscreen.
- Hard reboot into fastboot then start - booted but no keyboard.
- Clean install - no SIM - booted but no keyboard
- Rebooted via usb network/SSH - OK but no keyboard

I only seem to encounter this issue on my enchilada when a SIM card is present. But, reading Caleb's writings here has me questioning what it is I think I really know about, well, anything

Anyway, here's a crashdump I took today.

Is it relevant that there's two different issues in sdm845-mainline tracker that seem to implicate qcrypto?

I'm going to try building 6.13-rc2 without that module and see if I have any success..

Booting a 6.13.0-rc2 kernel with CONFIG_CRYPTO_DEV_QCE disabled did NOT fix this issue for me, still hit crashdump mode quite consistently when booting with a SIM card inserted.

mentioned in issue #3456 (closed)

As noted earlier, I was reliably reproducing this issue with a SIM card inserted - 5 out of 5 attempted boots all landed in crashdump mode.

I then re-flashed OxygenOS 11.1.2.2 from this link (by the way, is it just me or are all of the OxygenOS downloads on the official oneplus website dead?). Made absolutely sure that cellular was active and working, and then re-flashed pmOS. Now I've booted (rebooted + cold boot) a dozen times in a row without experiencing a single crashdump.

What I did differently this time was ensure the SIM card was inserted during the OxygenOS onboarding/first-run app, and I did notice that it spent a non-trivial amount of time on the "Installing" step after that, which I guess might support Caleb's earlier theory about something stateful happening to the modem's NVRAM...

I installed the latest OxygenOS (already had it downloaded) and ran through the basic setup, ensured I had mobile internet working, and then reinstalled 24.06 with Plasma Mobile and it went straight into crashdump mode. edge seems stable but I've not yet tested extensively. I notice that link is for the 6T - did you flash a 6T build and it work?

v24.06 is not supported any more, please use v24.12.

v24.12 boots OK but goes into crashdump on most reboots. Hard reboot after that sometimes goes into crashdump but mostly boots OK.

After testing v24.14 a freshly pmbootstraped edge went into crashdump multiple times before booting OK.

"QUALCOMM CrashDump Mode" on oneplus-enchilada

Describe your issue

What's the expected behaviour?

What's the current behaviour?

How to reproduce your issue?

What device are you using?

On what postmarketOS version did you encounter the issue?

On what environment did you encounter the issue?

Environments

How did you get postmarketOS image?

What's the build date of the image? (in yyyy-mm-dd format)

Additional information

Child items ...

Activity