porting to new device: pmbootstrap kconfig edit Error

Mmm... Sounds like there is a mismatch between the folder you are trying to cd into and the folders created when extracting your kernel... It would be interesting to see your apkbuild, especially the prepare() part.

You can also pmbootstrap chroot in after it failed, and manually look around /home/pmos/build, then change the path in the APKBUILD to the correct one.

By Mayeul C. on 2018-06-11T08:52:24

@MayeulC APKBUILD: https://pastebin.com/YWJeCrYB

By Moocow9m on 2018-06-11T13:41:35

Created by: ollieparanoid

Welcome to postmarketOS @Moocow9m and thanks for opening this issue!

This is possibly a regression from #1509. I will look into this tomorrow.

In the meantime, you could revert that PR (git revert 30df072) and add this menuconfig() function to your kernel's APKBUILD. In case this really was caused by a regression, then the menuconfig should appear with pmbootstrap menuconfig samsung-i927.

menuconfig() {
	cd "$srcdir"/build
	make ARCH="$_carch" menuconfig
	cp .config "$startdir"/$_config
}

In case it does not help, then there's something different with your kernel repository compared to the other kernel repos (consider trying another kernel).

Sorry for the inconvenience, we'll figure this out

By * postmarketOS Bot on 2018-06-11T22:51:14

@ollieparanoid Same error. I am undoing the revert now. It seems to not be making the /home/pmos/build/src/build folder.

By Moocow9m on 2018-06-12T03:05:19

Created by: ollieparanoid

Thanks for trying that out, this made it easier to figure out the problem! I've reproduced your error now, and found a solution. When changing the last part of prepare() to the following, it works as expected:

    # Prepare kernel config ('yes ""' for kernels lacking olddefconfig)
    mkdir -p "$srcdir"/build
    cp "$srcdir"/$_config "$builddir"/.config
    yes "" | make O="$srcdir"/build ARCH="$_carch" HOSTCC="$HOSTCC" oldconfig
}

As I understand it, your kernel has a different default value for KBUILD_OUTPUT defined somewhere (although I could not find where exactly).

PR #1556 will add this by default to the newly generated templates.

(Please close the issue if you can confirm that this works and feel free to open a new one if you have further issues.)

By * postmarketOS Bot on 2018-06-12T17:59:16

@ollieparanoid thank you, this has indeed resolved my problem.

By Moocow9m on 2018-06-13T20:20:14

I've reopened this issue because there are a lot of old linux-* packages that are broken and the solution #1556 partially solves the problem.

Right now for the new packages the prepare function will update the .config file (with make ... oldconfig) and with the new O="$srcdir"/build parameter, the .config is saved in the $WORK/chroot_native/home/pmos/build/src/build/.config.

The problem is that in the build function we actually compile the kernel in the builddir folder defined in the APKBUILD, which still contains the original .config file and not the updated one.

I'm thinking/trying a different solution...

By Daniele Debernardi on 2018-06-14T22:33:41

Things to consider:

prepare will update the -config file with make ... oldconfig
build should use the updated -config file
most linux-* packages use the builddir (except very old APKBUILD)
pmbootstrap kconfig edit should know the .config file location in order to copy it back

By Daniele Debernardi on 2018-06-14T22:57:18

Created by: ollieparanoid

Thanks for reopening!

How about we change all old APKBUILDs to use builddir as well, and add a test case that makes sure it's always the case (if someone is continuing an old port or something)?

By * postmarketOS Bot on 2018-06-14T23:32:41

@ollieparanoid I agree that we have to update all the old APKBUILDs to use the builddir

A possible solution for the issue:

revert #1556 and remove the O=... from the recently added APKBUILDs
update the python call like this:

diff --git a/pmb/build/menuconfig.py b/pmb/build/menuconfig.py
index a70d1a58..014049a3 100644
--- a/pmb/build/menuconfig.py
+++ b/pmb/build/menuconfig.py
@@ -102,14 +102,16 @@ def menuconfig(args, pkgname):
     # Run make menuconfig
     srcdir = "/home/pmos/build/src"
     logging.info("(native) make " + kopt)
+    builddir = pmb.chroot.user(args, ["sh", "-c", "source APKBUILD; echo $builddir"],
+                               "native", "/home/pmos/build", return_stdout=True).rstrip()
     pmb.chroot.user(args, ["make", kopt], "native",
-                    srcdir + "/build", log=False,
+                    srcdir + builddir, log=False,
                     env={"ARCH": pmb.parse.arch.alpine_to_kernel(arch),
                          "DISPLAY": os.environ.get("DISPLAY"),
                          "XAUTHORITY": "/home/pmos/.Xauthority"})

     # Find the updated config
-    source = args.work + "/chroot_native" + srcdir + "/build/.config"
+    source = args.work + "/chroot_native" + srcdir + builddir + "/.config"
     if not os.path.exists(source):
         raise RuntimeError("No kernel config generated: " + source)

In other words, run make menuconfig directly inside the builddir

By Daniele Debernardi on 2018-06-14T23:35:44

@ollieparanoid in the next hours I have some time, I can test a bit the changes I suggested while updating the old APKBUILDs to use the builddir.

these needs to be updated: comm -3 <(grep -ir 'builddir' aports/device/linux* | cut -d":" -f1 | sort -u) <(find aports/device/linux* -name APKBUILD | sort -u)

my idea was to add the downstreamkernel_prepare function to the devicepkg-dev like discussed in #688 (closed)

edit: proof of concept => https://github.com/postmarketOS/pmbootstrap/commit/421f96405edd8f66b3c2a019a75793e67f619b10 @ollieparanoid can you take a look at that commit and let me know what you think?

edit2: I've created an actual PR so you can leave comments there #1570

By Daniele Debernardi on 2018-06-15T19:51:43

While this might seem nice, some/many phones need customized build / packaging functions so it's not possible everywhere.

By Luca Weiss on 2018-06-15T19:52:04

Created by: ollieparanoid

@drebrez: I've been reading your earlier message in https://github.com/postmarketOS/pmbootstrap/issues/1551#issuecomment-397457314 over and over, and I always come to a different conclusion.

What am I missing?
Could you provide example commands to trigger the problem?

I've reopened this issue because there are a lot of old linux-* packages that are broken and the solution #1556 partially solves the problem.

My understanding is that linux packages are broken, which fulfill both points:

don't have O=... specified in prepare()
have a different KBUILD_OUTPUT specified in the kernel source

Therefore the new menuconfig code won't work for them because we expect the config file to be in $srcdir/build/.config.

=> my conclusion: update these old kernels to use O=... in prepare()

Right now for the new packages the prepare function will update the .config file (with make ... oldconfig) and with the new O="$srcdir"/build parameter, the .config is saved in the $WORK/chroot_native/home/pmos/build/src/build/.config.

My understanding:

prepare() updates .config
it selects the default values for the kernel config
saves the output in $srcdir/build/.config

The problem is that in the build function we actually compile the kernel in the builddir folder defined in the APKBUILD, which still contains the original .config file and not the updated one.

My understanding:

$srcdir/build/.config does not get used
the unmodified config gets used instead
this makes the config update from prepare() useless
the build does not run through non-interactively, because not all kernel config options are answered (with y/n/m)

=> my conclusion: always building in $srcdir/build instead. (using O=... in build() too)

By * postmarketOS Bot on 2018-06-16T05:56:51

@ollieparanoid Thanks for deciphering my message written in a hurry, you identified the issues I was trying to explain.

I don't really agree with your conclusions because there was already a discussion some time ago if we want to build the kernel in a dedicated folder (e.x: $srcdir/build) or directly where the source files are (cant find the issue number).

As for changing and building in $srcdir/build will mean to update the template and something like ~60 linux packages instead of keep building like we are doing now (builddir to the extracted folder,...) and just adapt the kconfig edit command to read the $builddir variable and run menuconfig there (see https://github.com/postmarketOS/pmbootstrap/blob/379dd0975efce4be3a7875d223eb306d40115412/pmb/build/menuconfig.py#L102-L111) and update something like 10-15 linux packages which are very old and don't use the builddir like we have in the template right now.

By Daniele Debernardi on 2018-06-16T09:58:53

Created by: ollieparanoid

Thanks, I'll look into this tomorrow

By * postmarketOS Bot on 2018-06-16T21:49:00

Created by: ollieparanoid

I don't really agree with your conclusions because there was already a discussion some time ago if we want to build the kernel in a dedicated folder (e.x: $srcdir/build) or directly where the source files are (cant find the issue number).

You are right! Looks like the discussion was partially done in #821 ("pmbootstrap init: Wizard for new port device- and linux-packages") and then followed up in the matrix. That PR started with not building in $builddir and then used it when it was ready to be merged.

As for changing and building in $srcdir/build will mean to update the template and something like ~60 linux packages instead of keep building like we are doing now (builddir to the extracted folder,...)

Right again, this is not a good solution, and $srcdir/build does not have any advantage over building in $builddir. Quite the opposite, the name builddir suggests, that we build inside that, so it would become less intuitive. Let's not change that back.

and just adapt the kconfig edit command to read the $builddir variable and run menuconfig there (see [...]) and update something like 10-15 linux packages which are very old and don't use the builddir like we have in the template right now.

I agree that it would be good to do this in the future. However, changing these linux APKBUILDs seems to me like a lot of effort as well, so I have another idea for a faster solution:

Revert the template change
Change the kconfig edit code to handle both the old style ($srcdir/build) as well as the state of art version ($builddir)
Print a note when using the old style that suggests to re-create the APKBUILD with the new style

I'm working on that now, so I can publish a PR that does that either today or tomorrow.

I take the full blame for this mess, because in my reviews of the kconfig edit PR I've made some edits in the end in which I intended to make it compatible with the old style APKBUILDs (looked at samsung-i9100). But I did not realize that this breaks compatibility with the newer APKBUILDs and that we agreed on using $builddir instead of $srcdir/build earlier.

You're constantly having an eye on the quality of the code base and make sure that we don't have such bugs. I can't thank you enough for that @drebrez!

By * postmarketOS Bot on 2018-06-17T15:30:05

I think that also old device packages should be put on a list to be modernized with devicepkg-dev...

By Luca Weiss on 2018-06-17T16:14:09

I agree that it would be good to do this in the future. However, changing these linux APKBUILDs seems to me like a lot of effort as well, so I have another idea for a faster solution

While this might seem like a lot of effort, we can distribute this quite easily, and would help resolving a part of the technical debt they have accumulated. I could have a stab at a few of them. Do we have a validator somewhere to check against the latest syntax?

By Mayeul C. on 2018-06-17T18:14:02

Created by: ollieparanoid

I think that also old device packages should be put on a list to be modernized with devicepkg-dev...

devicepkg-dev is able to generate udev rules that work for most, but not for all devices. To make sure that we don't have regressions, we would need to test each device after changing it. It's probably the easiest if people modernize the device packages of the devices they own in one PR instead of all at once.

I could have a stab at a few of them [linux aports]. Do we have a validator somewhere to check against the latest syntax?

If you can make sure, that the package still builds after the change (and you keep stuff that is specific to the kernel package, such as QCDT additions in build()), then it should still work on the device as well. Ideally you could test this on a device, but nobody has all the devices of course. It would be very nice if you could take a stab at some of them!

By * postmarketOS Bot on 2018-06-17T21:38:13

devicepkg-dev is able to generate udev rules that work for most, but not for all devices. To make sure that we don't have regressions, we would need to test each device after changing it. It's probably the easiest if people modernize the device packages of the devices they own in one PR instead of all at once.

Yeah sure, but I think the old-style devices packages should still be put on a list that they are tested with an updated device package and make a PR if there are no regressions.

By Luca Weiss on 2018-06-18T08:29:01

Created by: ollieparanoid

#1577 (closed)

By * postmarketOS Bot on 2018-06-18T19:12:48

porting to new device: pmbootstrap kconfig edit Error

Child items ...

Activity