Implement bash tab completion

The code looks good and I've tested it and can confirm that it's working. I like that you've made the argcomplete dependency optional.

However, before merging this I think we should discuss first if it makes sense to ship a shell completion method, that works completely different from the ZSH autocompletion which we already have.

Let me quickly outline the differences:

zsh version completes package names after "build", "checksum", "pkgrel_bump"
zsh version does not support any command parameter switches yet
zsh version reads the actions ("build", "checksum", "install", ...) with grep and cut from the Python code (fast and accurate), but everything else needs to be manually adjusted in the autocompletion file
the bash version runs the Python interpreter and executes the argparser each time one presses the tab key (slower, but accurate and no extra maintenance effort)

Personally, I like to have my tab completions instant, and I'm using the package name completion often. But of course accuracy and not having to update the completions file whenever the arguments change are important as well.

See #1517 (closed) for an approach that combines the best of all worlds from my point of view, but would also be some effort to implement.

What do you think, folks?

Thank you very much for this merge request @GrantM11235!

By Oliver Smith on 2018-08-01T22:11:03

added 1 commit

da437cdc - Add package completion proof of concept (very bad)

Compare with previous version

By Grant Miller on 2018-08-01T23:19:16

marked as a Work In Progress

By Grant Miller on 2018-08-01T23:24:38

changed the description

By Grant Miller on 2018-08-01T23:24:38

I made a quick proof of concept for package completion, it is very bad. The aports folder is hardcoded to aports (relative), so it only works if you are in the pmbootstrap folder. Is there a way to find out what the aports folder is before running parser.parse_args()? Or some existing way to get a list of packages?

Also, do you know of a good way to test the delay of tab completion?

By Grant Miller on 2018-08-01T23:33:04

closed

By Grant Miller on 2018-08-01T23:33:05

Is there a way to find out what the aports folder is before running parser.parse_args()?

This is a bit of a chicken-egg-problem, because you can override the aports folder with the --aports parameter. The default path is stored here: pmb.config.defaults["aports"]

Maybe it makes sense to set choices= only when running in tab completion mode? Otherwise it will not allow to override the aports folder with one that has packages not present in the default folder.

Or some existing way to get a list of packages?

We don't have a function that gets a list of all packages yet (I'm not sure if we needed that twice or more in the code). In pmb/helpers/frontend.py:apkbuild_parse() it is done like this:

for apkbuild in glob.glob(args.aports + "/*/*/APKBUILD"):
    packages.append(os.path.basename(os.path.dirname(apkbuild)))

Also, do you know of a good way to test the delay of tab completion?

I coudldn't find a recommended way to do it anywhere, but here's an idea that should work in theory:

modify the bash shell completion script, to execute time before the actual completion command
redirect the output of time to a file
this works with GNU's time for example, on Arch Linux you can install it with pacman -S time, and then call it with /usr/bin/time -o /tmp/output -- sleep 1 (when you just write time, your shell's internal implementation of time will be used that most likely does not support the -o parameter).
now after each completion command, you have the time it took in /tmp/output
watch -n1 /tmp/output can be used in a second terminal to always display the last time you had

(I'm reopening this, because it does not sound like you meant to close it.)

By Oliver Smith on 2018-08-02T22:05:00

reopened

By Oliver Smith on 2018-08-02T22:05:01

added 1 commit

77533c5a - Improve package completion

Compare with previous version

By Grant Miller on 2018-08-03T17:05:27

marked the checklist item Actually implement package completion as completed

By Grant Miller on 2018-08-03T17:05:50

Alright, thanks to a custom argcomplete completer, package completion actually works. It should work whether you use the default aports directory, a different one in the configuration, or one specified with --aports. It even allows you to specify a package that doesn't exist, so the existing missing package handling is the same.

(I'm reopening this, because it does not sound like you meant to close it.)

You are correct, thank you

By Grant Miller on 2018-08-03T17:17:07

added 1 commit

c14e7f5a - Set always_complete_options to "long"

Compare with previous version

By Grant Miller on 2018-08-03T17:40:13

changed the description

By Grant Miller on 2018-08-03T17:42:28

added 2 commits

50706e3e - Add command to install completion script
112da4f3 - Speed up entry_points with fastentrypoints

Compare with previous version

By Grant Miller on 2018-08-05T18:56:55

I started doing some benchmarking, and I found that autocompletion with the pmbootstrap entry_point installed with setup.py is about three times slower that just using pmbootstrap.py (or a symlink to it). I tracked this down to two things:

It takes argcomplete's global completer a while to parse the entry_point shim to see if it points to an argcomplete enabled entry_point
- This can be solved by setting up a completer just for pmbootstrap. I added a command in pmbootstrap to do just that, register_bash_completion. Alternatively, it can be set up by putting eval "$(register-python-argcomplete pmbootstrap)" in .bashrc
The entry_point shim imports pkg_resources, which takes time. This is a known issue, and it affects pmbootstrap whenever it is run from the entry_point, but it is even more noticeable with argcomplete.
- The workaround for this is to use fastentrypoints

After this, both options are basically the same.

Benchmarks

Edit: Moved to Merge Request description

Methodology

Edit: Moved to Merge Request description

register_bash_completion

I wanted to name this something unambiguous, but it seems a bit long to type (especially without tab completion ). I am open to suggestions.

This command creates a file with the same name of the pmbootstrap that ran it (pmbootstrap or pmbootstrap.py). The default directory is ~/.local/share/bash-completion/completions/, but it can be changed with the --dest option. There is also an --uninstall option to remove the file.

Complications

Argcomplete added the ability for a script to generate it's own completion file in the latest version, 1.9.4. I added an error when that function is not found, with instructions on how to update with pip3. I could add a fallback that generates the file with register-python-argcomplete in a shell, but I think that might just over complicate things.

After I wrote register_bash_completion, I discovered that bash-completion only added user-specific completions version 2.2. This means that users with older distros like Ubuntu Xenial or Debain Stretch will need to do one of the following:

Manually move the file to /usr/share/bash-completion/completions/
Source the file in .bashrc
Include eval "$(register-python-argcomplete pmbootstrap)" in .bashrc
Stick with argcomplete's global completion

Setting --dest to /usr/share/bash-completion/completions/ wouldn't work, because that would require root, and pmbootstrap complains when it is run as root.

The more I think about register_bash_completion, the worse it seems. Would it be better to just tell everyone to put eval "$(register-python-argcomplete pmbootstrap)" in .bashrc? Brutal honesty welcome.

Conclusion

This comment is way too long.

By Grant Miller on 2018-08-05T23:07:50

changed the description

By Grant Miller on 2018-08-05T23:07:46

Excellent update, both the code and the detailed text comments above. This is turning into a really beautiful feature now, and it really shows how much work went into this already. I've tested the bash completion in the current version, also with the package name completion, and it's working as expected. Great work!

Also the information you've gathered in the benchmarks is very useful - typical 9 year old laptop taking 0.4 seconds to complete sounds very reasonable to me.

recommendations

After spending quite some time playing with this, here are some changes I would recommend:

scrap pmbootstrap register_bash_completion (because of the issues you mentioned, especially with older bash versions)
implement pmbootstrap autocomplete instead, which does some checks and displays the steps necessary for the user to get autocompletion working with both bash and zsh (see below)
remove the old zsh completion script (because now we would use the same code!)
remove fastentrypoints.py - it seems like an usesful hack, but if we do it as described below, we won't need it, so it would be one less script to carry around

pmbootstrap autocomplete

I would recommend to put the code into a new file: pmb/helpers/autocomplete.py
test if argcomplete is installed, abort with the pip message otherwise (like register_bash_completion does it). only difference is, that I would recommend to install with --user, so it does not touch the whole system.
create a pmbootstrap symlink, that directly points to the pmbootstrap.py, and is then the same for everyone. So the python code would execute something like this:

mkdir -p ~/.local/bin
ln -sf absolute/path/to/pmbootstrap.py ~/.local/bin/pmbootstrap

note: pmb.config.pmb_src + "/pmbootstrap.py" can be used here
now it should tell the user to append something to their .bashrc/.zshrc
- we can figure out the shell with an optional --shell parameter (auto, bash, zsh), and when it is set to auto, make use of the $SHELL environment variable
- print some header comment like # pmbootstrap autocompletion
- if ~/.local/bin is not in the $PATH environment variable, print: export PATH="~/.local/bin/pmbootstrap:$PATH"
- if it is zsh, then print (source):

autoload bashcompinit
bashcompinit

finally, for both shells, print:

eval "$(register-python-argcomplete pmbootstrap)"

epilogue

I'm pretty sure that the ZSH version should work that way - although as I've tested it with an empty ZSH config, I'm getting this error: bash: compdef: command not found.... My guess is, that I need to initialize my zsh config properly first, but I have a whole backlog of e-mails ahead of me, so I'm not trying it out more right now. Maybe you could take a look at that as well?

Looking forward to what you think and come up with based on this feedback, I hope this approach is a very convenient and reliable one for the users, that does not mean much maintenance effort from our part. Thanks again for your work on this, I highly appreciate it! \o/

By Oliver Smith on 2018-08-14T00:27:20

added 1 commit

9fb51cd7 - Revert "Add command to install completion script"

Compare with previous version

By Grant Miller on 2018-08-25T05:32:58

added 44 commits

9fb51cd7...de9e42ba - 33 commits from branch postmarketOS:master
2c39e1c0 - Implement basic bash tab completion
abc10aff - Add package completion proof of concept (very bad)
87ea057e - Improve package completion
9d1eb5a1 - Set always_complete_options to "long"
1605e9e9 - Add command to install completion script
148e511c - Speed up entry_points with fastentrypoints
aa973ecf - Revert "Add command to install completion script"
9b5eebfb - Rename extras_require feature to `completion`
7c1da359 - Remove old Zsh completion script
23f38714 - Replace installation instructions with link to wiki page
f8be32cd - Revert "Speed up entry_points with fastentrypoints"

Compare with previous version

By Grant Miller on 2018-08-25T19:51:28

	My Desktop	My Laptop
Age	6 years	9 years
CPU	i5 3570K	Turion 64 X2 Mobile
Storage	SSD	5400 RPM HDD
Latency	0.12s - 0.14s	0.30s - 0.40s

Implement bash tab completion

Benchmarks

Methodology

Todo

Activity

Benchmarks

Methodology

register_bash_completion

Complications

Conclusion

recommendations

pmbootstrap autocomplete

epilogue

Implement bash tab completion

Benchmarks

Methodology

Todo

Merge request reports

Activity

Benchmarks

Methodology

register_bash_completion

Complications

Conclusion

recommendations

pmbootstrap autocomplete

epilogue