move kconfig checks out of pmbootstrap's config/__init__.py to pmaports.git

In the last team meeting we decided it makes more sense to move the kconfig checks to pmaports.git. This avoids the problem we currently have, that when we require a new option in edge, the kconfig check will fail on stable unless we also change the kernels there.

As format we figured a json file from which we strip comments before parsing in python (as python's json parser doesn't handle comments), would make sense. Then it looks almost the same as the current format.

This file should also be added to v22.06, so kconfig check still works there. For earlier releases pmbootstrap kconfig check will break, but that's fine, these are EOL anyway. Let's print an error that the file is missing on that branch in pmaports in that case.

added type::feature label

assigned to @z3ntu

By Oliver Smith on 2022-09-05T05:36:19

mentioned in issue #2166 (closed)

By Oliver Smith on 2022-09-05T05:41:31

After this patchset is merged, this should be quite feasible :)

@z3ntu: still interested in implementing this, or would you prefer if somebody else takes over? Do you want to review the patchset?

By Oliver Smith on 2023-03-12T15:29:54

Some more notes:

• currently we have pmbootstrap kconfig check --waydroid --nftables --iwd etc. arguments, one for each check component. this syntax needs to be reworked when moving the kconfig checks to pmaports since we only know the components after parsing the json file, not yet at the time when we parse the command-line arguments. we can probably change it to pmbootstrap kconfig check -c waydroid,iwd,nftables etc.
• I'd name the file kconfig_checks.json and put it in the root of pmaports.git
• besides the checks themselves, the json should have this information:
  • which components the "community" component implies
  • the enforce_check parameter, that is currently not set for devices in testing. TBH I think this was a mistake now, sorry for not realizing this earlier. After moving the kernel config checks to pmaports, let's make it so that we can enable the enforce_check paramter to address pmaports#1799 (closed) and not have print pmbootstrap kconfig check print misleading errors by default. For devices that don't pass it, we can either remove pmb:kconfig-check-community from the APKBUILD options, or we fix them up.
  • note that if we get #2190 done, it should be quite feasible to change a lot of kernels at once

By Oliver Smith on 2023-03-12T15:40:36

mentioned in issue pmaports#1799 (closed)

By Oliver Smith on 2023-03-12T15:39:59

When doing this, we should add a check that runs pmbootstrap kconfig check across all kernels when modifying kconfig_checks.json in pmaports.

By Oliver Smith on 2023-03-21T07:22:53

assigned to @ollieparanoid and unassigned @z3ntu

By Oliver Smith on 2023-03-21T07:24:53

I've played around with the format.

JSON with comments doesn't seem to be that suitable, because:

• In pure json you can't let the last item of an object end with a comma: {example: "value",}
• This leads to unclean diffs as you always need to modify the line above the last one when adding a new line to add a comma there
• And it's error prone because it's not intuitive that this is not allowed
• (and of course comments are not allowed in json, but stripping them is feasible)
• If we did something like json, we would basically need json5. But that would need an extra library for python, and pmbootstrap so far doesn't require any python libraries. So I don't think it's worth it.

I've also looked at toml:

• an improved version of ini that python can do natively since 3.11. (currently we support all non-EOL versions of python3, which means >= 3.9, before that it needs a library)
• but the format itself doesn't seem to be so suitable for our use case: it would make sense to have different sections for each category-versions-architectures combination. But object levels are split with dots, whereas our versions have dots (>=0.0.0), so it wouldn't split it right / need a workaround for that.

So I've arrived at the good old configparser library that python speaks natively, and which we already use in pmbootstrap code and in pmaports with pmaports.cfg, channels.cfg. It's like ini and has comments, and editors typically provide syntax highlighting for it (neovim and mediawiki highlight the sections in a different colors too, unfortunately gitlab doesn't do it).

Sections look like this: [<category> <versions> <arches>]

• category: "nftables", "containers" etc.
• versions: in the format ">=0.0.0" or "<5.17" or ">=3.13.0,<5.17"
• arches: in the format "all" or "x86_64" or "x86_64,aarch64"

Examples:

# this is a comment

[nftables >=3.13.0 all]
NETFILTER = true
NF_CONNTRACK = true
[nftables >=3.13.0,<5.17 all]
NFT_COUNTER = true

[containers >=0.0.0 all]
NAMESPACES = true
NET_NS = true
PID_NS = true
IPC_NS = true
UTS_NS = true
CGROUPS = true
[containers >=3.6 x86,x86_64]
HUGETLB_PAGE = true
CGROUP_HUGETLB = true
[containers >=3.6,<6.1_rc1 all]
MEMCG_SWAP = true

Fully converted file of current pmbootstrap kconfig opts (+ Luca's patch to add usb options):

# this is a comment
[postmarketOS >=0.0.0 all]
ANDROID_PARANOID_NETWORK = true
BLK_DEV_INITRD = true
CGROUPS = true
CRYPTO_AES = true
CRYPTO_XTS = true
DEVTMPFS = true
DM_CRYPT = true
INPUT_EVDEV = true
EXT4_FS = true
KINETO_GAN = true
PFT = true
SEC_RESTRICT_ROOTING = true
SYSVIPC = true
TMPFS_POSIX_ACL = true
USE_VFB = true
VT = true
[postmarketOS >=2.6.0 all]
BINFMT_ELF = true
[postmarketOS >=3.10.0 all]
BINFMT_SCRIPT = true
[postmarketOS >=4.0.0 all]
UEVENT_HELPER = true
USER_NS = true
[postmarketOS <4.7.0 all]
DEVPTS_MULTIPLE_INSTANCES = true
[postmarketOS <4.14.0 all]
SAMSUNG_TUI = true
TZDEV = true
[postmarketOS <5.2.0 armhf,armv7,x86]
LBDAF = true

[waydroid >=0.0.0 all]
ANDROID_BINDERFS = true
ANDROID_BINDER_DEVICES = ['binder', 'hwbinder', 'vndbinder']
ANDROID_BINDER_IPC = true
ANDROID_BINDER_IPC_SELFTEST = true
BLK_DEV_LOOP = true
BPF_SYSCALL = true
BRIDGE = true
BRIDGE_VLAN_FILTERING = true
CGROUP_BPF = true
FUSE_FS = true
IP_NF_MANGLE = true
NETFILTER_XTABLES = true
NETFILTER_XT_MATCH_COMMENT = true
PSI = true
PSI_DEFAULT_DISABLED = true
SQUASHFS = true
SQUASHFS_XATTR = true
SQUASHFS_XZ = true
TMPFS_XATTR = true
TUN = true
VETH = true
VLAN_8021Q = true
[waydroid >=3.5 all]
CROSS_MEMORY_ATTACH = true
[waydroid >=4.20.0 all]
PSI = true
PSI_DEFAULT_DISABLED = true
[waydroid <5.18 all]
ASHMEM = true

[iwd >=0.0.0 all]
ASYMMETRIC_KEY_TYPE = true
ASYMMETRIC_PUBLIC_KEY_SUBTYPE = true
CRYPTO_AES = true
CRYPTO_CBC = true
CRYPTO_CMAC = true
CRYPTO_DES = true
CRYPTO_ECB = true
CRYPTO_HMAC = true
CRYPTO_MD5 = true
CRYPTO_SHA1 = true
CRYPTO_SHA256 = true
CRYPTO_SHA512 = true
CRYPTO_USER_API_HASH = true
CRYPTO_USER_API_SKCIPHER = true
KEYS = true
KEY_DH_OPERATIONS = true
PKCS7_MESSAGE_PARSER = true
PKCS8_PRIVATE_KEY_PARSER = true
X509_CERTIFICATE_PARSER = true

[nftables >=3.13.0 all]
NETFILTER = true
NF_CONNTRACK = true
NF_TABLES = true
NF_TABLES_INET = true
NFT_CT = true
NFT_LOG = true
NFT_LIMIT = true
NFT_MASQ = true
NFT_NAT = true
NFT_REJECT = true
NF_TABLES_IPV4 = true
NF_REJECT_IPV4 = true
IP_NF_IPTABLES = true
IP_NF_FILTER = true
IP_NF_TARGET_REJECT = true
IP_NF_NAT = true
NF_TABLES_IPV6 = true
NF_REJECT_IPV6 = true
IP6_NF_IPTABLES = true
IP6_NF_FILTER = true
IP6_NF_TARGET_REJECT = true
IP6_NF_NAT = true
[nftables >=3.13.0,<5.17 all]
NFT_COUNTER = true

[containers >=0.0.0 all]
NAMESPACES = true
NET_NS = true
PID_NS = true
IPC_NS = true
UTS_NS = true
CGROUPS = true
CGROUP_CPUACCT = true
CGROUP_DEVICE = true
CGROUP_FREEZER = true
CGROUP_SCHED = true
CPUSETS = true
KEYS = true
VETH = true
BRIDGE = true
BRIDGE_NETFILTER = true
IP_NF_FILTER = true
IP_NF_TARGET_MASQUERADE = true
NETFILTER_XT_MATCH_ADDRTYPE = true
NETFILTER_XT_MATCH_CONNTRACK = true
NETFILTER_XT_MATCH_IPVS = true
NETFILTER_XT_MARK = true
NETFILTER_XT_TARGET_CHECKSUM = true
IP_NF_NAT = true
NF_NAT = true
POSIX_MQUEUE = true
BLK_DEV_DM = true
DUMMY = true
BLK_CGROUP = true
BLK_DEV_THROTTLING = true
CGROUP_PERF = true
NET_CLS_CGROUP = true
FAIR_GROUP_SCHED = true
RT_GROUP_SCHED = true
IP_NF_TARGET_REDIRECT = true
IP_VS = true
IP_VS_NFCT = true
IP_VS_PROTO_TCP = true
IP_VS_PROTO_UDP = true
IP_VS_RR = true
EXT4_FS_POSIX_ACL = true
EXT4_FS_SECURITY = true
[containers >=3.2 all]
CFS_BANDWIDTH = true
[containers >=3.3 all]
CHECKPOINT_RESTORE = true
[containers >=3.6 all]
MEMCG = true
DM_THIN_PROVISIONING = true
SWAP = true
[containers >=3.6 x86,x86_64]
HUGETLB_PAGE = true
CGROUP_HUGETLB = true
[containers >=3.6,<6.1_rc1 all]
MEMCG_SWAP = true
[containers >=3.7,<5.0 all]
NF_NAT_IPV4 = true
NF_NAT_IPV6 = true
[containers >=3.7 all]
VXLAN = true
IP6_NF_TARGET_MASQUERADE = true
[containers >=3.9 all]
BRIDGE_VLAN_FILTERING = true
MACVLAN = true
[containers >=3.14 all]
CGROUP_NET_PRIO = true
[containers >=3.18 all]
OVERLAY_FS = true
[containers >=3.19 all]
IPVLAN = true
SECCOMP = true
[containers >=4.4 all]
CGROUP_PIDS = true

[zram >=3.14.0 all]
ZRAM = true
ZSMALLOC = true
CRYPTO_LZ4 = true
LZ4_COMPRESS = true
SWAP = true

[netboot >=0.0.0 all]
BLK_DEV_NBD = true

[wireguard >=5.6_rc1 all]
WIREGUARD = true
IP_ADVANCED_ROUTER = true
IP_MULTIPLE_TABLES = true
IPV6_MULTIPLE_TABLES = true
NF_TABLES = true
NF_TABLES_IPV4 = true
NF_TABLES_IPV6 = true
NFT_CT = true
NFT_FIB = true
NFT_FIB_IPV4 = true
NFT_FIB_IPV6 = true
NF_CONNTRACK_MARK = true

[filesystems >=0.0.0 all]
BTRFS_FS = true
EXFAT_FS = true
EXT4_FS = true
F2FS_FS = true

[usb_gadgets >=0.0.0 all]
USB_ETH = true
USB_FUNCTIONFS = true
USB_MASS_STORAGE = true
USB_G_SERIAL = true
USB_CONFIGFS_RNDIS = true

[community >=0.0.0 all]
INPUT_UINPUT = true
LEDS_TRIGGER_TIMER = true
NETFILTER_XT_MATCH_TCPMSS = true
NETFILTER_XT_TARGET_TCPMSS = true

[uefi >=0.0.0 all]
EFI_STUB = true
EFI = true
DMI = true
EFI_ESRT = true
EFI_VARS_PSTORE = true
EFI_PARAMS_FROM_FDT = true
EFI_RUNTIME_WRAPPERS = true
EFI_GENERIC_STUB = true

I've converted it with the following script:

#!/usr/bin/env python3
import sys

sys.path.append("/home/user/code/pmbootstrap/")
import pmb.config
import pmb.parse.kconfig

def print_component(name, opts):
    for version in opts.keys():
        for arch in opts[version].keys():
            section = name
            section += f" {version.replace(' ', ',')}"
            section += f" {arch.replace(' ', ',')}"

            print(f"[{section}]")

            for key, val in opts[version][arch].items():
                if isinstance(val, list):
                    val = str(val)
                elif isinstance(val, str):
                    val = str
                elif val in [True, False]:
                    val = "true" if True else "false"
                else:
                    raise RuntimeError(f"{key}: can't handle type of {val}")

                print(f"{key} = {val}")



def print_components():
    # NOTE: the postmarketOS component will be renamed to be more obvious, it has
    # the options that are required for all devices, no matter in which device category
    components_list = ["postmarketOS"] + pmb.parse.kconfig.get_all_component_names()
    
    for name in components_list:
        if name == "postmarketOS":
            pmb_config_var = "kconfig_options"
        else:
            pmb_config_var = f"kconfig_options_{name}"
    
        opts = getattr(pmb.config, pmb_config_var, None)
        assert opts, f"invalid kconfig component name: {name}"
        print_component(name, opts)
        print("")

print_components()

I'm not 100% sure if storing arrays like the following is a good idea, that will become more obvious when adjusting the code to parse the ini file:

[waydroid >=0.0.0 all]
ANDROID_BINDER_DEVICES = ['binder', 'hwbinder', 'vndbinder']

But besides that I'm pretty happy with it.

By Oliver Smith on 2023-04-26T06:40:09

but the format itself doesn't seem to be so suitable for our use case: it would make sense to have different sections for each category-versions-architectures combination. But object levels are split with dots, whereas our versions have dots (>=0.0.0), so it wouldn't split it right / need a workaround for that.

Tables in Toml can use dots if you quote them. See https://toml.io/en/v1.0.0#table and https://toml.io/en/v1.0.0#keys

I would personally prefer Toml over "ini" since it's well-defined with a clear standard unlike ini which is different depending on what implementation you use.

By Newbyte on 2023-04-26T10:33:09

ah good point, I didn't see that. I guess with minor changes we can turn the proposed format above into valid toml and then use that.

By Oliver Smith on 2023-04-26T06:43:37

As valid toml it would look like the following.

[containers.">=3.6".x86 x86_64]                        
HUGETLB_PAGE = true       
CGROUP_HUGETLB = true                                  
[containers.">=3.6 <6.1_rc1".all]
MEMCG_SWAP = true  
[containers.">=3.7 <5.0".all]                                                                                  
NF_NAT_IPV4 = true
NF_NAT_IPV6 = true
[containers.">=3.7".all]
VXLAN = true
IP6_NF_TARGET_MASQUERADE = true
[containers.">=3.9".all]
BRIDGE_VLAN_FILTERING = true
MACVLAN = true

By Oliver Smith on 2023-04-27T14:28:33

toml is really nice, the parser understands (for our purpose relevant) booleans, strings, lists and complains if the value isn't any of those :)

By Oliver Smith on 2024-06-23T21:49:00

added discussion label

mentioned in merge request !2333 (merged)

By Oliver Smith on 2024-06-23T21:18:19

implemented the pmbootstrap changes, this was the hard part

TODO:

• pmbootstrap: get this merged: !2333 (merged)
• pmaports: add copy of kconfigcheck.toml and adjust CI to use pmbootstrap master: pmaports!5301

By Oliver Smith on 2024-08-11T12:58:49

closed