Skip to content
Snippets Groups Projects

Support using initramfs compression setting from deviceinfo

Merged Imported Support using initramfs compression setting from deviceinfo
compression_formats into master
Merged Imported Administrator requested to merge compression_formats into master

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Administrator
    Administrator @root · Imported
    Author Owner

    cc @z3ntu @ollieparanoid

    By clayton craft on 2022-11-18T00:54:22

  • Administrator marked this merge request as draft · Imported

    marked this merge request as draft

    By clayton craft on 2022-11-18T01:20:49

  • Administrator mentioned in merge request !24 (closed) · Imported

    mentioned in merge request !24 (closed)

    By clayton craft on 2022-11-18T18:09:35

  • Administrator changed the description · Imported

    changed the description

    By clayton craft on 2022-11-19T07:49:16

  • Administrator added 5 commits · Imported

    added 5 commits

    • fe2a9d41 - archive: add CompressFormat type with initial constants
    • 9216d508 - archive: accept compression format parameter during instantiation
    • 3bc19e13 - archive: allow selecting different formats when writing compressed file
    • 6ecc1538 - archive: support using zstd
    • f4ff79c7 - archive: support using lzma

    Compare with previous version

    By clayton craft on 2022-11-19T08:01:23

    Toggle commit list
  • Administrator changed the description · Imported

    changed the description

    By clayton craft on 2022-11-19T08:03:32

  • Administrator
    Administrator @root · Imported
    Author Owner

    As discussed on Matrix, my proposal:

    • Use zstd for community/main devices, except..
    • Use lzma for community/main devices that have limited boot partition space
    • Use gzip (existing) for everything else.

    By Luca Weiss on 2022-12-02T16:44:10

  • Administrator marked this merge request as ready · Imported

    marked this merge request as ready

    By clayton craft on 2022-12-08T19:54:31

  • Administrator changed milestone to %v2.0 · Imported

    changed milestone to %v2.0

    • Administrator
      Administrator @root · Imported
      Author Owner
      Resolved by Administrator

      Just being curious: why use a native go implementation for these, instead of running external commands? If we did run external commands, one advantage would be that deviceinfo_initfs_compression could have the compression level too, like zstd -19 (just an example, never tried that level in particular) and use any compression command. We could also let the device package depend on the compression tool if they don't want the default that is gzip. And I could imagine the official zstd tool might be more optimized than a 3rd party go implementation, but I didn't benchmark this.

      Regarding Luca's proposal, we could change it initially and then have recommendations in the wiki about which algorithm to use (a good place would be the description of deviceinfo_initfs_compression). I'm not sure there's a big advantage in switching community/main over to zstd though.

      By Oliver Smith on 2023-01-23T07:05:07

  • Administrator
    Administrator @root · Imported
    Author Owner

    If we decide to not use shell commands for compression, then the patches look good to me. Thanks for implementing this!

    By Oliver Smith on 2023-01-23T07:06:31

  • Administrator marked this merge request as draft · Imported

    marked this merge request as draft

    By clayton craft on 2023-02-13T16:37:15

    • Administrator
      Administrator @root · Imported
      Author Owner
      Resolved by Administrator

      @ollieparanoid ok I reworked this a bit. As discussed in the chat, supporting external apps is tricky, it requires knowing what the cmdline parameters of the target exe are, which may not always be consistent if it's an app from busybox or gnu, or bsd, or ???. So for this first revision, I went with a simpler implementation that uses Go compression packages to do the work. The downside of this is that the compression Level is a bit weird to set, since most libraries discourage setting the numeric compression level directly.

      This is configured by setting deviceinfo_initfs_compression, the value it expects is a string in the form: FORMAT[:LEVEL], where [:LEVEL] is optional. Actually setting the variable at all is optional... if nothing is specified, or it can't parse the format/level from the string value, it defaults to using gzip with the "default" level for the package (which tries to mirror gzip's default, or something). It basically falls back to gzip with the "default" compression level, so that this change won't cause mkinitfs to fail if a user or device maintainer put garbage in deviceinfo_initfs_compression.

      The level can be one of default, fast, best.

      To configure gzip with the fastest compression (so, bigger size): deviceinfo_initfs_compression="gzip:fast"

      To configure zstd with the most compression: deviceinfo_initfs_compression="zstd:best"

      To configure zstd with default compression: deviceinfo_initfs_compression="zstd" (or deviceinfo_initfs_compression="zstd:default")

      In this case, gzip:default is assumed: deviceinfo_initfs_compression="bananas:mmmm"

      Is this implementation going to let a user eek out a few less bytes by tweaking compression levels (zstd has 22 as of today)? Nope, you only get 3 levels. But I'm not sure if the added complexity is worth it. So I think this is good enough for now until someone can prove that they need to be able to explicitly set compression level 17 with zstd, etc.

      Test script:

      #!/bin/sh


for format in gzip zstd lzma; do
        echo "# Format: $format"
        for level in default fast best; do
                echo "#### Level: $level"
                for i in $(seq 1 3); do
                        echo "######## Run #$i"
                        sed -i "s|deviceinfo_initfs_compression=.*|deviceinfo_initfs_compression=\"$format:$level\"|" /etc/deviceinfo
                        mkinitfs 2>&1|grep completed
                        ls -lah /boot/initramfs*
                done
        done

done

      Output of tests on qemu x86_64:

      / # ./test.sh
# Format: gzip
#### Level: default
######## Run #1
2023/02/21 13:42:29 initramfs completed in: 293.46213ms
2023/02/21 13:42:29 initramfs-extra completed in: 234.511708ms
2023/02/21 13:42:29 boot-deploy completed in: 104.899492ms
2023/02/21 13:42:29 mkinitfs completed in: 636.135643ms
-rw-r--r--    1 root     root        4.0M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.6M Feb 21 13:42 /boot/initramfs-extra
######## Run #2
2023/02/21 13:42:30 initramfs completed in: 302.187528ms
2023/02/21 13:42:30 initramfs-extra completed in: 240.505136ms
2023/02/21 13:42:30 boot-deploy completed in: 178.360375ms
2023/02/21 13:42:30 mkinitfs completed in: 725.379174ms
-rw-r--r--    1 root     root        4.0M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.6M Feb 21 13:42 /boot/initramfs-extra
######## Run #3
2023/02/21 13:42:30 initramfs completed in: 316.666873ms
2023/02/21 13:42:31 initramfs-extra completed in: 237.29465ms
2023/02/21 13:42:31 boot-deploy completed in: 185.648973ms
2023/02/21 13:42:31 mkinitfs completed in: 743.212436ms
-rw-r--r--    1 root     root        4.0M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.6M Feb 21 13:42 /boot/initramfs-extra
#### Level: fast
######## Run #1
2023/02/21 13:42:31 initramfs completed in: 199.890657ms
2023/02/21 13:42:31 initramfs-extra completed in: 97.472291ms
2023/02/21 13:42:31 boot-deploy completed in: 124.315671ms
2023/02/21 13:42:31 mkinitfs completed in: 426.636284ms
-rw-r--r--    1 root     root        4.3M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.9M Feb 21 13:42 /boot/initramfs-extra
######## Run #2
2023/02/21 13:42:31 initramfs completed in: 201.162864ms
2023/02/21 13:42:31 initramfs-extra completed in: 94.817054ms
2023/02/21 13:42:32 boot-deploy completed in: 151.51925ms
2023/02/21 13:42:32 mkinitfs completed in: 454.661603ms
-rw-r--r--    1 root     root        4.3M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.9M Feb 21 13:42 /boot/initramfs-extra
######## Run #3
2023/02/21 13:42:32 initramfs completed in: 152.142832ms
2023/02/21 13:42:32 initramfs-extra completed in: 130.954922ms
2023/02/21 13:42:32 boot-deploy completed in: 168.66531ms
2023/02/21 13:42:32 mkinitfs completed in: 455.888039ms
-rw-r--r--    1 root     root        4.3M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.9M Feb 21 13:42 /boot/initramfs-extra
#### Level: best
######## Run #1
2023/02/21 13:42:33 initramfs completed in: 982.101039ms
2023/02/21 13:42:34 initramfs-extra completed in: 639.560519ms
2023/02/21 13:42:34 boot-deploy completed in: 117.283639ms
2023/02/21 13:42:34 mkinitfs completed in: 1.742882952s
-rw-r--r--    1 root     root        4.0M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.6M Feb 21 13:42 /boot/initramfs-extra
######## Run #2
2023/02/21 13:42:35 initramfs completed in: 1.018930371s
2023/02/21 13:42:36 initramfs-extra completed in: 636.381478ms
2023/02/21 13:42:36 boot-deploy completed in: 95.831608ms
2023/02/21 13:42:36 mkinitfs completed in: 1.753305427s
-rw-r--r--    1 root     root        4.0M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.6M Feb 21 13:42 /boot/initramfs-extra
######## Run #3
2023/02/21 13:42:37 initramfs completed in: 1.011154723s
2023/02/21 13:42:37 initramfs-extra completed in: 640.679381ms
2023/02/21 13:42:37 boot-deploy completed in: 97.007414ms
2023/02/21 13:42:37 mkinitfs completed in: 1.751570725s
-rw-r--r--    1 root     root        4.0M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.6M Feb 21 13:42 /boot/initramfs-extra
# Format: zstd
#### Level: default
######## Run #1
2023/02/21 13:42:37 initramfs completed in: 104.657931ms
2023/02/21 13:42:38 initramfs-extra completed in: 53.416846ms
2023/02/21 13:42:38 boot-deploy completed in: 175.01312ms
2023/02/21 13:42:38 mkinitfs completed in: 336.6723ms
-rw-r--r--    1 root     root        4.1M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.7M Feb 21 13:42 /boot/initramfs-extra
######## Run #2
2023/02/21 13:42:38 initramfs completed in: 196.268255ms
2023/02/21 13:42:38 initramfs-extra completed in: 61.310147ms
2023/02/21 13:42:38 boot-deploy completed in: 171.930122ms
2023/02/21 13:42:38 mkinitfs completed in: 433.093903ms
-rw-r--r--    1 root     root        4.1M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.7M Feb 21 13:42 /boot/initramfs-extra
######## Run #3
2023/02/21 13:42:38 initramfs completed in: 191.988574ms
2023/02/21 13:42:38 initramfs-extra completed in: 91.559124ms
2023/02/21 13:42:39 boot-deploy completed in: 158.10204ms
2023/02/21 13:42:39 mkinitfs completed in: 444.944424ms
-rw-r--r--    1 root     root        4.1M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.7M Feb 21 13:42 /boot/initramfs-extra
#### Level: fast
######## Run #1
2023/02/21 13:42:39 initramfs completed in: 100.242451ms
2023/02/21 13:42:39 initramfs-extra completed in: 89.03388ms
2023/02/21 13:42:39 boot-deploy completed in: 128.363652ms
2023/02/21 13:42:39 mkinitfs completed in: 321.555507ms
-rw-r--r--    1 root     root        4.2M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.8M Feb 21 13:42 /boot/initramfs-extra
######## Run #2
2023/02/21 13:42:39 initramfs completed in: 151.436346ms
2023/02/21 13:42:39 initramfs-extra completed in: 62.213311ms
2023/02/21 13:42:39 boot-deploy completed in: 170.031066ms
2023/02/21 13:42:39 mkinitfs completed in: 389.061746ms
-rw-r--r--    1 root     root        4.2M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.8M Feb 21 13:42 /boot/initramfs-extra
######## Run #3
2023/02/21 13:42:39 initramfs completed in: 135.972162ms
2023/02/21 13:42:40 initramfs-extra completed in: 87.531301ms
2023/02/21 13:42:40 boot-deploy completed in: 114.676114ms
2023/02/21 13:42:40 mkinitfs completed in: 341.430064ms
-rw-r--r--    1 root     root        4.2M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.8M Feb 21 13:42 /boot/initramfs-extra
#### Level: best
######## Run #1
2023/02/21 13:42:40 initramfs completed in: 590.63998ms
2023/02/21 13:42:41 initramfs-extra completed in: 603.612583ms
2023/02/21 13:42:41 boot-deploy completed in: 113.364787ms
2023/02/21 13:42:41 mkinitfs completed in: 1.311574671s
-rw-r--r--    1 root     root        3.9M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.5M Feb 21 13:42 /boot/initramfs-extra
######## Run #2
2023/02/21 13:42:42 initramfs completed in: 563.518018ms
2023/02/21 13:42:42 initramfs-extra completed in: 604.4174ms
2023/02/21 13:42:42 boot-deploy completed in: 101.509467ms
2023/02/21 13:42:42 mkinitfs completed in: 1.272726233s
-rw-r--r--    1 root     root        3.9M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.5M Feb 21 13:42 /boot/initramfs-extra
######## Run #3
2023/02/21 13:42:43 initramfs completed in: 549.86713ms
2023/02/21 13:42:43 initramfs-extra completed in: 599.251332ms
2023/02/21 13:42:44 boot-deploy completed in: 187.673645ms
2023/02/21 13:42:44 mkinitfs completed in: 1.340223007s
-rw-r--r--    1 root     root        3.9M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.5M Feb 21 13:42 /boot/initramfs-extra
# Format: lzma
#### Level: default
######## Run #1
2023/02/21 13:42:46 initramfs completed in: 2.011374938s
2023/02/21 13:42:47 initramfs-extra completed in: 1.059582632s
2023/02/21 13:42:47 boot-deploy completed in: 145.551298ms
2023/02/21 13:42:47 mkinitfs completed in: 3.218283797s
-rw-r--r--    1 root     root        3.8M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.4M Feb 21 13:42 /boot/initramfs-extra
######## Run #2
2023/02/21 13:42:49 initramfs completed in: 1.961247723s
2023/02/21 13:42:50 initramfs-extra completed in: 1.076169159s
2023/02/21 13:42:50 boot-deploy completed in: 136.957797ms
2023/02/21 13:42:50 mkinitfs completed in: 3.178413969s
-rw-r--r--    1 root     root        3.8M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.4M Feb 21 13:42 /boot/initramfs-extra
######## Run #3
2023/02/21 13:42:52 initramfs completed in: 1.990789271s
2023/02/21 13:42:53 initramfs-extra completed in: 1.077546574s
2023/02/21 13:42:53 boot-deploy completed in: 137.182502ms
2023/02/21 13:42:53 mkinitfs completed in: 3.20858008s
-rw-r--r--    1 root     root        3.8M Feb 21 13:42 /boot/initramfs
-rw-r--r--    1 root     root        2.4M Feb 21 13:42 /boot/initramfs-extra


<rest of lzma results left off, since it only supports the "default" level>

      By clayton craft on 2023-02-22T06:53:19

      Edited by Administrator
  • Administrator added 84 commits · Imported

    added 84 commits

    • f4ff79c7...4074eada - 74 commits from branch master
    • 6bbc6f5e - archive: remove pgzip, use gzip from go std lib (MR 25)
    • a6f6bdbb - archive: add CompressFormat type with initial constants (MR 25)
    • be94823a - archive: accept compression format parameter during instantiation (MR 25)
    • c7fb3d8a - archive: allow selecting different formats when writing compressed file (MR 25)
    • c213494f - archive: support using zstd (MR 25)
    • ed5b5e00 - archive: add CompressLevel type and consts (MR 25)
    • 0464dbcf - archive: add ExtractFormatLevel function (MR 25)
    • d5eecdcb - archive.New: accept compression level (MR 25)
    • e587e882 - archive: Use compression level when generating archive (MR 25)
    • 8387b412 - cmd/mkinitfs: print info about compression format and level used (MR 25)

    Compare with previous version

    By clayton craft on 2023-02-21T21:29:13

    Toggle commit list
  • Administrator changed the description · Imported

    changed the description

    By clayton craft on 2023-02-21T21:29:35

  • Administrator marked this merge request as ready · Imported

    marked this merge request as ready

    By clayton craft on 2023-02-21T21:30:00

  • Administrator added 1 commit · Imported

    added 1 commit

    • bea3aaaf - archive: support using lzma (MR 25)

    Compare with previous version

    By clayton craft on 2023-02-21T21:40:50

  • Administrator added 1 commit · Imported

    added 1 commit

    • 6d41fc88 - archive: support using lzma (MR 25)

    Compare with previous version

    By clayton craft on 2023-02-21T21:41:06

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply