dnsmasq 2.90 hangs at 100% cpu, dns lookup fail

changed the description

By laurentl on 2024-02-14T09:25:03

Thanks for the report, I have the very same setup and similar symptoms:

Telegram is working but anything related to dns fail
dnsmask 100% cpu
ping 8.8.8.8 is working
ping google.com is not

The only difference is that for me, even the namesever 8.8.8.8 in resolv.conf does not fix anything. @lolgzs Did you do anything else to get a working environnement ? Is this line the only one in your `resolv.conf` ? I tried but failed to setup networkmanager ... I am not sure editing this file is the right way :( I am becoming mad

By Charles Gueunet on 2024-02-14T10:05:43

nameserver , not namesever ;)

By laurentl on 2024-02-14T10:52:45

wow I made a bad typo. Thanks ^^

I stay available to help with any further debugging on the current issue

By Charles Gueunet on 2024-02-14T10:52:45

Hello, this issue is also present in the shiftmq-axolotl since yesterday, when I ran apk upgrade -a. I'm also on edge. Although I do not see any issues with CPU consumption, DNS is not working. Editing /etc/resolv.conf also solves the issue temporarily for me. Using logread -d 0, I can see the following lines:

[0] daemon NetworkManager[4635]: <warn>  [1707912409.6025] device (qrtr0): retrieving IP configuration failed: modem IP method unsupported
[0] daemon dnsmasq[6234]: reading /etc/resolv.conf
[0] daemon dnsmasq[6234]: using nameserver 127.0.0.1#53
[0] daemon NetworkManager[4635]: <warn>  [1707912409.6090] platform-linux: do-change-link[7]: failure 22 (Invalid argument), setting MTU to requested size is not possible
[0] daemon NetworkManager[4635]: <warn>  [1707912409.6090] device (qrtr0): mtu: failure to set MTU. Did you configure the MTU correctly?
[0] user nm-dns-filter[10522]: ip6 gateway:
[0] user nm-dns-filter[10523]: disabling IPv4 filter
[0] user nm-dns-filter[10524]: setting A filter to 'false'
[0] daemon NetworkManager[4635]: <warn>  [1707912414.6071] dns-mgr: update-pending changed: DNS plugin did not become ready again. Assume something is wrong
[0] daemon NetworkManager[4635]: <info>  [1707912414.6071] device (qrtr0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
[0] daemon NetworkManager[4635]: <info>  [1707912414.6106] device (qrtr0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
[0] daemon NetworkManager[4635]: <info>  [1707912414.6109] device (qrtr0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
[0] daemon NetworkManager[4635]: <info>  [1707912414.6116] device (qrtr0): Activation: successful, device activated.
[0] daemon NetworkManager[4635]: <warn>  [1707912429.6049] dnsmasq[dd33c8fa346e4fc1]: dnsmasq update failed: Timeout was reached
[0] daemon dbus-daemon[4366]: [system] Activating service name='org.freedesktop.Accounts' requested by ':1.85' (uid=10000 pid=6064 comm="/usr/bin/gnome-software --gapplication-service") (using servicehelper)
[0] daemon accounts-daemon[10800]: started daemon version 23.13.9
[0] daemon dbus-daemon[4366]: [system] Successfully activated service 'org.freedesktop.Accounts'
[0] daemon busctl[10525]: Call failed: Operation timed out
[0] daemon nm-dispatcher: req:2 'down' [qrtr0], "/usr/lib/NetworkManager/dispatcher.d/50-dns-filter.sh": complete: failed with Script '/usr/lib/NetworkManager/dispatcher.d/50-dns-filter.sh' exited with status 1.
[0] daemon NetworkManager[4635]: <warn>  [1707912436.6867] dispatcher: (14) /usr/lib/NetworkManager/dispatcher.d/50-dns-filter.sh failed (failed): Script '/usr/lib/NetworkManager/dispatcher.d/50-dns-filter.sh' exited with status 1.

By antoniomaciej on 2024-02-14T12:08:47

Same symptoms,

pine64-pinebookpro on edge, v23.12
pine64-pinephone on v23.12

Looks like this commit? I would've updated my system in the morning before this was pushed. Then again today afterwards. https://gitlab.alpinelinux.org/alpine/aports/-/commit/72ab6c34cd91684f87630c5dbe1180b18360e4b3

No issues on two of my devices running standard Alpine Linux edge and v3.19. Will have to look into my DNS configuration for these a little more closely after work, but neither of them run Network Manager.

Edit: Should also clarify, I'm not seeing this 100% CPU usage. Forgot to mention that. But DNS is failing.

By Quade Curry on 2024-02-14T14:36:45

Could you try downgrading dnsmasq to 2.89-r6 to see if it works around the problem?

By Newbyte on 2024-02-14T13:39:23

Only able to test on pine64-pinebookpro on v23.12 right now, but yep. Seems to fix it right up.

By Quade Curry on 2024-02-14T14:20:27

Thanks for testing. I created an issue in Alpine as well: https://gitlab.alpinelinux.org/alpine/aports/-/issues/15788

By Newbyte on 2024-02-14T14:29:19

No problem. I'll get NetworkManager installed on a normal Alpine system later to see if that's relevant, if no one's figured it out by then.

By Quade Curry on 2024-02-14T14:30:57

Same issue on pinephone 23.12 stable, but the workaround with dnsmasq downgrade didn't work with: ERROR: Unable to select packages: dnsmasq-dnssec-dbus-2.90-r0: breaks: world[dnsmasq=2.89-r6] satisfies: postmarketos-base-ui-networkmanager-15-r0[dnsmasq-dnssec-dbus>=2.89-r2]; networkmanager-dnsmasq-1.44.2-r1[dnsmasq-dnssec-dbus]; postmarketos-base-ui-gnome-3-r0[dnsmasq]

By Marco A on 2024-02-14T15:46:08

@crowdtier Unfortunately Alpine doesn't keep old package versions, so pinning a package doesn't work if that package has already been updated. i.e. it can't select it because it can't find it in the repos, since it's not there.

You'll have to clone aports, checkout a commit before dnsmasq was updated, build it, and install it yourself if you want to downgrade.

Sorry.

By Quade Curry on 2024-02-14T15:50:33

added release-edge status::reported type::bug labels

added release-v23.12 label

It seems that the 100% CPU only occurs when a SIM card is inserted in the phone during boot ... If I boot the phone without SIM and activate wifi, dnsmasq CPU usage is normal.

DNS lookup still not working

By laurentl on 2024-02-14T15:33:03

marked #2599 (closed) as a duplicate of this issue

By Newbyte on 2024-02-14T17:57:24

marked this issue as related to #2599 (closed)

By Newbyte on 2024-02-14T17:57:24

mentioned in issue #2599 (closed)

By Newbyte on 2024-02-14T17:57:26

I spent some time looking into this, here are my notes:

this is triggered by the NM dispatcher script that configures DNS filtering in dnsmasq over dbus. I don't think that patch is broken, but maybe related to interacting with dnsmasq over dbus "too often" or ??
dnsmasq is responsive over dbus initially, but will eventually hang at 100% cpu usage
I haven't been able to bisect this in dnsmasq... 2.89 in Alpine includes the dbus filter toggle patch and it works. 2.89 from git + that patch hangs eventually at 100% cpu usage. Maybe this suggests something external is the cause (in NM or ?)
I have NOT been able to reproduce this by running dnsmasq 2.90 manually, and quickly applying filtering options over dbus

$ doas dnsmasq -p 5353 --enable-dbus=uk.org.thekelleys.dnsmasq
$ while true; do doas busctl call uk.org.thekelleys.dnsmasq /uk/org/thekelleys/dnsmasq uk.org.thekelleys.dnsmasq SetFilterA b false; done
# this doesn't fail...

If I repeat the above but target dnsmasq as run by NM, sometimes it hangs and sometimes it does not...

$ while true; do doas busctl call org.freedesktop.NetworkManager.dnsmasq /uk/org/thekelleys/dnsmasq org.freedesktop.NetworkManager.dnsmasq SetFilterA b false; done

I added sleep all over the dispatcher script thinking maybe it was a timing thing, it didn't help
according to gdb (let it hang, then C-c randomly to see where execution is happening), dnsmasq seems to be spending a lot of time in rrfilter. Seems like a LOT has changed in rrfilter stuff between 2.89..2.90
I've not been able to reproduce this on my aarch64 laptop

By clayton craft on 2024-02-19T20:18:57

I merged a workaround to edge and 23.12 that disables this dns filtering stuff for now until we can figure out what broke...

This workaround will re-expose folks to #1430

By clayton craft on 2024-02-14T20:22:26

changed title from OnePlus 6 / Edge : dnsmasq 100% cpu, dns lookup fail to dnsmasq 2.90 hangs at 100% cpu, dns lookup fail

By clayton craft on 2024-02-14T20:00:51

mentioned in commit cd1f3015

By clayton craft on 2024-02-14T20:05:26

mentioned in merge request !4824 (closed)

By clayton craft on 2024-02-14T20:06:08

mentioned in commit 776ca83c

By clayton craft on 2024-02-14T20:09:33

mentioned in commit c391a3fb

By clayton craft on 2024-02-14T20:12:08

mentioned in commit 4dac8961

By clayton craft on 2024-02-14T20:18:09

mentioned in commit e09952c8

By clayton craft on 2024-02-14T20:20:30

mentioned in issue #1430

By clayton craft on 2024-02-14T20:23:38

After the workaround, everything is working again. Thanks for the fix! I would say that in terms of network, it works better than when #1430 was fixed with dnsmasq and this issue was not present. Before (let's say for instance 4 days ago), the modem was crashing constantly and every hour or so I had to reintroduce the PIN of my SIM card. If this was my primary phone, I could have lost some calls. After the workaround, I have not noticed any crashes. Also, I think today my battery life has been greatly improved. I have no data to back this up, thought, it's only my impression. Maybe other people have noticed something similar and can share their own impression.

By antoniomaciej on 2024-02-15T18:09:36

I don't think dnsmasq, at least how we use it, has anything to do with the modem crashing. NetworkManager and ModemManager interact with the modem more directly than dnsmasq would.

The workaround I merged still uses dnsmasq, it just skips filtering out A and/or AAAA records based on which protocol is available/routable over NM's "primary connection". We need that filtering to fix #1430 (unless someone can come up with a better solution, but let's discuss that in the issue if you have ideas)

By clayton craft on 2024-02-15T18:09:36

Ok, thanks for the response. Yes, maybe I am just jumping to conclusions because my phone has had a good day :).

By antoniomaciej on 2024-02-15T18:27:19

no worries, I like good phone days. if you run into problems, try to collect logs if you can. logread can be used to dump syslog for the current boot, there are also options to it to display logs from previous boots too (in the future I hope we can greatly improve collecting debug info like this for users...) If it looks unrelated based on the logs, or if you can't tell, just file a new issue so it's not lost in some existing, possibly unrelated issue like this one :)

By clayton craft on 2024-02-15T21:46:44

changed the description

By clayton craft on 2024-02-19T21:32:33

mentioned in commit cbab162e

By clayton craft on 2024-02-20T00:06:10

mentioned in merge request !4846 (closed)

By clayton craft on 2024-02-20T00:06:42

mentioned in issue #2753

By lm2lm2 on 2024-04-18T12:38:42

mentioned in commit 99bd7b61

By clayton craft on 2024-06-04T16:41:44

mentioned in commit 45dec6cc

By clayton craft on 2024-06-04T16:43:14

closed

dnsmasq 2.90 hangs at 100% cpu, dns lookup fail

Describe your issue

What's the expected behaviour?

What's the current behaviour?

How to reproduce your issue?

What device are you using?

On what postmarketOS version did you encounter the issue?

On what environment did you encounter the issue?

Environments

How did you get postmarketOS image?

What's the build date of the image? (in yyyy-mm-dd format)

Additional information

Child items ...

Activity