Adjust pmbootstrap to have minimal kernel configs in pmaports.git
@bshah wrote:
Btw
Our defconfigs in all of repos are saved wrong way
They're mostly copy of .config copied to the aports
But this is wrong way to save defconfigs
And you should instead use make savedefconfig
So that it saves just delta to default defconfig
And not all of defconfig as is.
I went ahead and created a patch for it (see below). However, as I've tested it, our "kconfig check" code complained right away after using it for the first time, that some kernel configs were not set. So... nice idea, but doesn't work for us 
I've created this issue for future reference, in case the question comes up again.
If I'm overlooking something and there is a way to use this, without breaking "kconfig check" or requiring to download all the kernel tarballs (which is not feasible, "kconfig check" should not take forever...), please let me know.
FWIW, the feature was introduced to linux in 2010: https://lwn.net/Articles/397363/
---
pmb/build/menuconfig.py | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/pmb/build/menuconfig.py b/pmb/build/menuconfig.py
index 5c632419..a5fdbc82 100644
--- a/pmb/build/menuconfig.py
+++ b/pmb/build/menuconfig.py
@@ -138,13 +138,25 @@ def menuconfig(args, pkgname):
# Run make menuconfig
outputdir = get_outputdir(args, pkgname)
+ arch_kernel = pmb.parse.arch.alpine_to_kernel(arch)
logging.info("(native) make " + kopt)
pmb.chroot.user(args, ["make", kopt], "native",
outputdir, output="tui",
- env={"ARCH": pmb.parse.arch.alpine_to_kernel(arch),
+ env={"ARCH": arch_kernel,
"DISPLAY": os.environ.get("DISPLAY"),
"XAUTHORITY": "/home/pmos/.Xauthority"})
+ # Get a delta to the default config
+ try:
+ pmb.chroot.user(args, ["make", "savedefconfig"], "native", outputdir,
+ env={"ARCH": arch_kernel})
+ pmb.chroot.user(args, ["mv", "defconfig", ".config"], "native",
+ outputdir)
+ except:
+ logging.warning("WARNING: failed to run 'make savedefconfig' to"
+ " minimize your kernel config. Maybe your kernel is so"
+ " old, that it does not have this feature?")
+
# Find the updated config
source = args.work + "/chroot_native" + outputdir + "/.config"
if not os.path.exists(source):
--
2.17.2Activity
added help wanted type::feature labels
added discussion label
One way would be to avoid the kconfig check entirely by using shared "kernel config fragments", that enforce that all the required options are set. The idea is that you have a file that lists the config options that should be set/not set, and apply that on top of another kernel config.
There was a nice talk about it last year: "Managing Linux Kernel Configs With Config Fragments" (slides, video) that also explains the problems when storing "raw" or "full" configs like in postmarketOS (especially: large, unreadable diffs but also maintainability).
I'm actually using a similar setup for
linux-postmarketos-qcom-msm8916(https://gitlab.com/postmarketOS/pmaports/merge_requests/582) but I reverted it for the postmarketOS package shortly before opening the MR because I wanted to discuss this separately.My reason for using kernel config fragments was that I wanted to make a custom (minimal) kernel configuration because the aarch64 defconfig is huge. But eventually I want to use the msm8916 kernel in other projects as well, so I want to re-use the same base defconfig for all of them. So I split the config into two files:
-
msm8916_defconfig: base defconfig that configures all needed device drivers and some standard kernel options -
pmos.config: the postmarketOS specific options, more or less translated as-is from the kconfig check
A slightly enhanced
pmos.configcould be shared across all kernels, and changes would only need to be made in one file rather than touching all the kernel configurations. I'm not sure if this is a great idea for downstream kernels, but at least for mainline it should work quite well.Nevertheless, there are a few open problems when using kernel config fragments:
- Editing: There is menuconfig for normal kernel configurations, but it cannot really be used to edit fragments. It's more easy to manually edit them with a text editor, but care must be taken to also list dependencies that are required for a listed config options.
-
Envkernel: The reason I have the
pmos.configpart directly in the kernel repository at the moment is that envkernel does not provide a nice way to initialize the kernel configuration. We would need some easy way to apply kernel configuration + postmarketOS fragment to a local Linux repository.
What do you think?
By Minecrell on 2019-09-04T17:47:34
-
-
@Minecrell
-
Envkernel: The reason I have the
pmos.configpart directly in the kernel repository at the moment is that envkernel does not provide a nice way to initialize the kernel configuration. We would need some easy way to apply kernel configuration + postmarketOS fragment to a local Linux repository.
For that, you can use the new option
pmbootstrap kconfig check --file .configto check a kernel config file in your filesystem without it being in a pmaports package.By Luca Weiss on 2019-09-05T15:43:01
Edited by Ghost User -
Envkernel: The reason I have the
-
I suppose this is useful to check if the final generated configuration is really valid. But what I meant here that it is quite difficult to initialize the kernel configuration for envkernel because you cannot copy a single file anymore. I.e. the envkernel guide suggests that you:
Copy the defconfig from the kernel aport to the kernel source directory.
$ cp /code/pmboostrap/aports/device/linux-oneplus-bacon/config-oneplus-bacon.armhf arch/arm/configs/oneplus_bacon_defconfigBut this isn't as easy anymore if you need to combine multiple config fragments from different locations. So it would help to have some helper that initializes the kernel configuration for a given postmarketOS kernel package.
By Minecrell on 2019-09-05T15:43:01
-
@Minecrell, thanks for the detailed explanation of the kernel config fragments idea. This sounds to me like it would be great to have, especially in code review!
However, here are the biggest problems I see with using kernel config fragments:
- kconfig check checks different options depending on the architecture and kernel version (see
__init__.py), we would need to reflect this in the fragments approach (multiple fragments? is it still an improvement then?) - How can we make sure, that all kernels build after we updated the shared config parts? (Right now this works nicely, we change requirements in "kconfig check", let it run across all configs, it complains about the configs we need to adjust. Then we adjust them, bump the pkgrel, test if the kernels still build, and make the merge request.)
So maybe one can figure them out, and also find good solutions for the "kconfig edit" and envkernel problems that you outlined above, the latter two can probably be solved with some python code that merges and splits the config as needed.
All in all, I don't think it is worth the effort - but if somebody has clever solutions for the above problems and would like to take a shot at it, please explain here what you're about to do and then go for it.
By Oliver Smith on 2019-09-07T17:52:23
Edited by Ghost User - kconfig check checks different options depending on the architecture and kernel version (see
-
kconfig check checks different options depending on the architecture and kernel version (see
__init__.py), we would need to reflect this in the fragments approach (multiple fragments? is it still an improvement then?)Android's kernel fragments are split in multiple files for example (one common, then one per arch if necessary). I suppose that works for the few architecture-specific options we have. Otherwise, it would be theoretically possible to generate the fragments on-the-fly from a Python script or similar, using the config you linked. But that sounds a bit odd.
How can we make sure, that all kernels build after we updated the shared config parts? (Right now this works nicely, we change requirements in "kconfig check", let it run across all configs, it complains about the configs we need to adjust. Then we adjust them, bump the pkgrel, test if the kernels still build, and make the merge request.)
I don't quite understand this problem. I'm not entirely sure what's the best way to integrate a shared kernel config fragment into the postmarketOS build scripts, but I suppose it would need to be integrated into the kernel APKBUILDs. So when you bump the pkgrel, it would also use the newer kernel config fragments the next time you build the package. Therefore, the process isn't much different from before, except that you skip the step where you have to update the kernel configuration manually.
By Minecrell on 2019-09-07T17:52:23
-
So when you bump the pkgrel, it would also use the newer kernel config fragments the next time you build the package. Therefore, the process isn't much different from before, except that you skip the step where you have to update the kernel configuration manually.
The difference is, that we won't know which kernels need a pkgrel bump. This can introduce silent breakage.
By Oliver Smith on 2019-09-11T16:29:24
-
The difference is, that we won't know which kernels need a pkgrel bump. This can introduce silent breakage.
But why do you not know which kernels need a bump? If there is only a common fragment plus an architecture-specific fragment (e.g. for ARM), you need to bump either all, or only the kernels with the architecture that was changed.
That doesn't mean that there will be necessarily a change in the final kernel image - a new option that is added to a fragment could be already enabled in the base config ("defconfig") for a particular kernel. In that case, there will not be any difference in the kernel that is built, but that shouldn't hurt too much.
By Minecrell on 2019-09-11T16:48:38
-
Fair enough, if we do rebuild all kernels that could potentially be affected, then we avoid the silent breakage. This could work.
There are still downsides to that:
- re-compiling takes time
- users need to download the new kernel although possibly nothing changed
- harder to find regressions with commits that bump the pkgrel without possibly changing anything
But IMHO they are an acceptable trade-off for the fragments feature.
By Oliver Smith on 2019-09-14T01:08:06
-
@z3ntu
So the advantage of using this is smaller diffs? I wouldn't say that's a good trade-off then for all the things you listed above.
(Sorry for the long comment...)
We are discussing two separate things here:
- using defconfig style kernel configurations (i.e. those produced by
savedefconfig) vs "full" kernel configurations - forcing certain options using kernel config fragments vs editing manually with assistance of "kconfig check"
The trade-offs listed above are for (2) but the "smaller diffs" advantage is one of (1).
The two points are not strictly related to each other:
-
(1 ↛ 2): defconfigs do not necessarily require the use of kernel config fragments: this issue was opened because defconfigs break the
kconfig check.But a very simple solution to continue using the
kconfig checksetup with defconfigs is to simply look at the configuration of the built kernel (package?). e.g. Alpine kernels store the "full", resulting configuration at/boot/config-<flavor>, which can be inspected bykconfig checknormally.Then you can also use continue using the existing
kconfig editetc tools, you just need to callmake savedefconfigand copy the defconfig instead of the "full" configuration. -
(1 ↚ 2): Kernel config fragments can be also applied on top of "full" kernel configurations. In other words, existing configurations do not need to be converted to defconfigs to use them with kernel config fragments.
Maybe it would be better to discuss (1) and (2) in separate issues.
For:
-
(defconfig vs "full" config): The unreadable diffs of "full" kernel configurations are a major problem in terms of maintainability in my opinion. Nobody looks at them. So you don't notice when options you really care about are suddenly no longer enabled because of missing dependencies or other problems.
Example: Did anyone notice that
CONFIG_PREEMPT=ygot lost forlinux-postmarketos-allwinnerduring the upgrade to 5.3-rc1? (See https://gitlab.com/postmarketOS/pmaports/commit/9fde750935cf3702e0812962072882307e5934d5#45cc45fcb1bf1089fc0765c8e0408d9f64c4d6cc_72_77, manually scroll to the line with# CONFIG_PREEMPT_NONE is not set) There was a bug in 5.3-rc1 that converted old configurations incorrectly. (I had the same problem forlinux-postmarketos-qcom-msm8916and noticed it later only by a coincidence... Now I checked the allwinner kernel and seems like the same happened there.)In this case, a defconfig would have produced the correct configuration again after upgrading to the fixed kernel version. Or at least it would have produced a readable diff that shows that an option you care about is no longer applied. The "full" configuration carries such changes forever, usually unnoticed, until they are manually fixed.
-
(kernel config fragments) have the primary advantage that it's easier to stay up-to-date with a kernel config that is continuously updated upstream. You can use an upstream config as a base (either "full" or defconfig) and only apply a few postmarketOS overrides on top of it.
e.g. original base config:
- pine64_defconfig
- msm8916_defconfig
- LineageOS config
- ...
plus postmarketOS fragment would result in the postmarketOS kernel configuration.
Updating the kernel source/version would automatically also apply upstream changes to the kernel configuration. This avoids having to check which config options were changed upstream and finally having to update all of them manually.
By Minecrell on 2019-09-16T14:20:00
Edited by Administrator - using defconfig style kernel configurations (i.e. those produced by
-
(defconfig vs "full" config): The unreadable diffs of "full" kernel configurations are a major problem in terms of maintainability in my opinion. Nobody looks at them. So you don't notice when options you really care about are suddenly no longer enabled because of missing dependencies or other problems.
This is huge. There are examples of issues right now in gitlab(https://gitlab.com/postmarketOS/pmaports/-/merge_requests/1575 and https://gitlab.com/postmarketOS/pmaports/-/issues/776) that are resolved by making changes to the kernel config, and I'm really not confident that we won't introduce regressions when the kernel + config is updated in the future. Furthermore, any regressions would be very hard to spot in review since the config is long, and we don't always know (or remember) which config options need to be switched on for any particular platform.
I don't have a strong opinion about which option we choose to solve this, but I do have a strong opinion that this is a problem that needs to be solved one way or another.
By clayton craft on 2020-09-14T06:26:45
Edited by Administrator
mentioned in issue pmaports#431 (closed)
By Minecrell on 2020-01-18T19:28:47
mentioned in issue postmarketos#24 (closed)
By Minecrell on 2020-03-29T11:49:12
mentioned in merge request !2006 (closed)
By HenriDellal on 2020-12-14T14:52:13
-
@HenriDellal, @minlexx, @calebccff, @craftyguy, @dylanvanassche, @z3ntu, @MartijnBraam and I had a call about how to move this forward.
Here are the notes: https://wiki.postmarketos.org/wiki/User:Ollieparanoid/kconfig-fragments-2021-10-05
By Oliver Smith on 2021-10-05T20:04:03
