mobile data network still used even when connected to wifi/ethernet

There are some common situations where network traffic still goes over lower priority networks, like mobile data / wwan, when it should be going over some other preferred network, like wifi.

I assume that this is a very common expectation that people have for how network traffic on a mobile device is handled. I also expect this behavior on laptops and other systems with multiple/mixed network interface types.

For example:

1. connected to the following networks:

• WiFi: ipv4-only
• wwan/mobile data: ipv4 & ipv6

3. DNS nameservers are configured for both connections e.g., nmcli connection show "<connection name>" |grep DNS

4. perform a lookup e.g., dig postmarketos.org

5. if a AAAA query was submitted and the response was the one chosen, then any connections to the ipv6 IP in the response will go over the wwan connection, breaking the expectations above

Things to consider:

• MMS and VVM (visual voicemail) are mostly(?) US things, but we have to support them. These apps must have DNS lookups and network connections routed through wwan. VoLTE will probably(?) require this special handling too, so we cannot ignore it even if I wanted to (and I'm in the US, using MMS, so I can't ignore it for personal reasons 😅) So if these provider servers are only accessible over v6, then we cannot just disable v6 over wwan entirely.

• Since IPv6 as a whole is enabled at the OS level (and cannot be disabled because of the above point), getaddrinfo etc will send out DNS requests for AAAA records. This will be the case regardless of any route metrics set over wlan0 and wwan0, so just setting a lower metric (higher priority) on wlan0 over wwan0 is not a solution by itself.

• If an application gets both an A record and an AAAA record, it will likely prefer to make an IPv6 connection, or make both and then prefer the IPv6 connection if it succeeds (Happy Eyeballs).

Ideas:

1. custom DNS resolver. I've already tried building a custom resolver that scrubbed responses from DNS lookups e.g., removes AAAA records when only ipv4 on pri. conn. It kinda worked, but it seems like a bad idea to continue because 1) I don't want to maintain this, 2) doing DNS stuff seems like CVE minefield, further justifying the first reason!

2. dnsmasq DNS record filtering. dnsmasq supports setting A and AAAA record filters via dbus, so use a NetworkManager dispatcher script to set this filtering conditionally based on which protocols are routable through NM's primary connection. This was tried in !3823 (closed).

   • dnsmasq is fragile, upgrade to 2.90 introduced 2 problems: one confirmed to be a regression, the other is pending

3. Use a special network NS and a nftables rule to tag / route packets directly to the wwan interface. Apps (MMS, etc) ran in this network NS.

   • I tried this... very fragile to set up and maintain at runtime
   • is appealing because apps themselves wouldn't have to be aware of any special routing that needs to happen for traffic to/from them
   • maybe this idea could be improved / fixed?

4. Block all ipv6 connections not routable over the primary connection (say via a blackhole route), and add exceptions (routes with lower metric) as needed at runtime.

   • By default, routes will be added to make the wwan DNS servers be reachable.
   • When the device receives a new SMS indicating that there is a new MMS, the device would add routes (if necessary) to the MMS server. Unclear how the MMS server would be known, but it's probably either from mobile-broadband-provider-info or part of the SMS itself.
   • Similar thing happens for VVM?
   • Unsure about VoLTE.
   • Problem 1: This requires vvmd / mmsd / chatty / whatever to not act upon the MM notification of an incoming SMS until our script is done setting up the routes. So it would require some coordination with / code change in those programs, to either wait for the routes to be set up, or to set up the routes themselves.
   • Problem 2: This handles the case of adding routes in response to incoming notifications, but it doesn't handle the case of user-initiated requests. For example this will probably break the initial VVM registration.

5. Create a new network namespace, move wwan0 into it, and then run mmsd / vvmd / etc in that netns. Since wwan0 is no longer part of the global netns, other applications will have no choice but to use wlan0. This step of moving wwan0 in and out of the netns would need to be done every time the primary connection changes, and mmsd / vvmd / etc will have to be restarted to run against the new netns.

   • This will require NM etc to be able to work with interfaces in multiple namespaces.

6. Create a new "physical" network namespace, and move both wlan0 and wwan0 into it. Create a veth pair with one end in the global netns and the other in the physical netns. mmsd / vvmd / etc can run in the physical netns and use wwan0 directly. Everything else will only see the veth pair, and we can use a script to set up nftables rules to control whether the veth pair in the physical netns can talk to wlan0 or to wwan0 (depending on whether wlan0 is connected or not). This avoids the problem of the previous idea where wwan0 would need to be moved in and out of netns's constantly and require mmsd / vvmd / etc to be restarted each time.

   • This will require NM etc to be able to work with interfaces in multiple namespaces.

7. Use systemd-resolved somehow

   • the manpage implies(?) that DNS routing priority set per interface
   • priority doesn't matter if the nameserver(s) still respond with AAAA records