configuring wifi mac is race-y under systemd

I did a lot of poking around when trying to get bootmac to work properly under systemd, filing a new issue about this to collect some of my notes/observations in case they're useful later. This focuses entirely on bootmac for wifi, bootmac for bluetooth is hopefully "solved" by this point (see !10 (merged))

bootmac exists to provide a persistent MAC addr for wifi devices that boot with missing or random MAC addresses. The ath10k kernel driver will assign a random MAC when it loads, if it detects that the wifi dev is missing one, other drivers may behave differently. In any case, the goal is that when wifi devices are "handed over" to NM and iwd/wpa_sup for managing, they have a persistent MAC addr just like you'd expect any off-the-shelf wifi card to have. NM, iwd, wpa_sup (and users using macchanger) are free to modify the MAC further if they want.

This means that bootmac has to run right after the driver loads, but not so early that the driver hasn't finished initializing the hardware. It also must run before sw higher in the stack (NM, iwd, wpa_supp) has started using it.

There are at least two scenarios where bootmac has to be used to do this:

1. On bootup

2. When the wifi driver is reloaded (rmmod && modprobe, or something similar)

It's possible, though still somewhat unreliable, to get bootmac to load on bootup by creating a systemd service in network-pre.target, that loads before NM/iwd/wpa_sup. This doesn't help with the second scenario: systemd isn't going to restart the service if the module is reloaded, so bootmac never runs to reconfigure the MAC before sw higher in the stack start to use the device. Another approach along these lines was to add a NetworkManager.service.d/override.conf that basically has ExecStartPre=bootmac --wlan, but this still doesn't address the second scenario (and @DylanVanAssche reported that this didn't work correctly for him, though we didn't debug it further)

Using a udev rule, as we do with openrc, to cover the second scenario (driver reload) doesn't work either, triggering on net and KERNEL=wlan[0-9*] is often too late, sw higher in the stack has access to it by now and may be using the wrong/random mac from the driver. I even found that sometimes multiple instances of bootmac might be run as more than one wlan* device might be created (but all except 1 are brought down at the end of dev init). The rule could trigger on something earlier, like SUBSYSTEM=="ieee80211", and set the mac in /sys/class/ieee80211/phy0 but unfortunately it looks like /sys/class/ieee80211/phy0/macaddress` is readonly.

Lastly, I looked at using a udev rule to trigger a systemd unit that started bootmac, using the SYSTEMD_WANTS="bootmac-wifi.service" udev option, but it was often too late and NM/iwd were already using the device.

I believe this is still one big race on openrc installs, but that udev(?) on openrc is slow enough that it seems to win the race on bootup. I think that the other problem wrt driver reloads is still present though even on openrc, it seems likely that bootmac run via udev will still run too late and iwd/NM might use a random (in the case of ath10k) MAC regardless of what NM config says.

For now, the recommendation is to not ship the bootmac-related udev rules for wlan on systemd OSes.