Drop directfb

We're currently using the directfb backend of sdl2 to get osk-sdl running on downstream kernels, which only have framebuffer drivers. directfb upstream has been dead for a while now, it would be great if we could get rid of that component, but still support framebuffer. (In fact, directfb is only carried in Alpine, because we use it... it would be great to remove that piece of technical debt.)

Some ideas from the chat:

1. @minlexx and @IdanHoro have been working on directly implementing framebuffer into SDL2, but that is WIP: https://gitlab.com/IdanHoro/SDL/commits/video_fbdev. Apparently it would get accepted upstream if implemented properly (and that's what we would want, we don't want to fork sdl2 long-term obviously.) If going down this route, maybe we benefit from using tslib? directfb uses it too, but in contrary to directfb, tslib seems much more maintained.

2. @MartijnBraam proposed making my fbdev backend for wlroots work with cage, and then using that. While the backend works as proof of concept in my wlroots branch, there are quite some things that need to be resolved to make it work like that (IIRC it breaks when rebasing on master, it would need to be turned into an external module - see end of related PR - , cage would need to be adjusted to load that external module). A downside of this approach is, that we add more components to the initramfs, potentially making it start a bit slower and increasing the size.

3. Replace SDL2 dependency and custom keyboard with lvgl. It would look something like what NixOS Mobile has here. Some notes:

• lvgl is not packaged in alpine. It seems to be quite the feature-rich framework, a potential downside might be a long build time (not verified!)
• The linked PR is written against ruby bindings of lvgl, we would use C / C++ just like the existing osk-sdl codebase.
• We would have to replace our custom keyboard with the one from lvgl. I think our custom keyboard is pretty nice by now, and a few people have put a lot of work into it, so this wouldn't be an easy decision. If we do that, we should make sure that the lvgl keyboard is as least as good as our keyboard in every aspect.
  • The keyboard layout seen on the screenshot isn't mobile friendly, how do we customize that? (our keyboard has the same layout as the one from phosh by now, and typing in a keyboard is quite convenient with it)

Let's discuss here!

added discussion label

lvgl is not packaged in alpine. It seems to be quite the feature-rich framework, a potential downside might be a long build time (not verified!)

Most of the features in LVGL can be disabled, thus resulting in pretty fast compile times. For reference, https://gitlab.com/DolphinChips/charging-lvgl takes around 5 seconds to compile with make -j5

The keyboard layout seen on the screenshot isn't mobile friendly, how do we customize that? (our keyboard has the same layout as the one from phosh by now, and typing in a keyboard is quite convenient with it)

Keyboard widget in LVGL allows to set custom keymap, see https://docs.lvgl.io/latest/en/html/widgets/keyboard.html#new-keymap, plus everything from https://docs.lvgl.io/latest/en/html/widgets/btnmatrix.html should more or less apply to keyboard too.

By Dolphin von Chips on 2021-01-14T20:20:32

I would like to be sure that every feature of osk-sdl's current keyboard (as of today) can be implemented using lvgl and its keyboard. I don't think we should switch if it means giving up some of the visual enhancements and accessibility improvements made to it recently.

By clayton craft on 2021-01-14T20:20:32

IMHO the 'best' option there is #1:

• it would add the fewest new components to initramfs
• it has the best chance of working with the current osk-sdl project (it would just be another sdl backend to switch to)
• it would be easy to switch over to supporting in pmOS (and other distros)

but it requires someone picking up the fbdev branch that @minlexx found and upstreaming it to sdl2, which is probably non-trivial :P

By clayton craft on 2021-01-14T20:18:43

If going down this route, maybe we benefit from using tslib? directfb uses it too, but in contrary to directfb, tslib seems much more maintained.

I don't know details about tslib, but I'd try to use libinput first, which would not only solve touchscreen input, but keyboard input too (and potentially any other input), which is what we want to have in backend.

lvgl is not packaged in alpine

As far as I know lvgl is embeddable library, it is not supposed to be packaged. If we go lvgl way, It should be compiled as part of osk-lvgl/chargin-lvgl. So, lvgl source becomes part of osk/charging source (a subdirectory, or git submodule). And unused code is of course stripped from binary by liker, making resulting executable binary as small as possible without extra deps (like libsdl2).

I like options 1 and 3, but I don't know which one is better/easier/smaller/ ... Adding framebuffer video backend to sdl is not hard, and I did it initially (and then IdanHoro improved it even more), but adding input devices support into that backend adds more complexity.

I'm personally leaning towards the 3rd option. LittlevGL looks nice, it seems like it has whatever we need, with this we can throw away both DirectFB and SDL2 from initramfs entirely and reduce its size. We don't even have to use its built-in keyboard, maybe we can reuse what we already have in osk? Downsides are of course osk will need to be rewritten; not entirely, only drawing code paths, not the whole FS unlock logic. But still..

By Alexey Min on 2021-01-14T23:28:20

Downsides are of course osk will need to be rewritten; not entirely, only drawing code paths, not the whole FS unlock logic. But still..

the unlocking logic is a very small part of osk-sdl, so effectively this would be a new program :P

By clayton craft on 2021-01-15T03:59:58

Unfortunately maybe so. But it will be better program, with less code and simple logic and easier to maintain.

By Alexey Min on 2021-01-15T04:47:54

I spent about 2 hours looking at LVGL, and playing with a small test app (just running the 'widgets' demo on my laptop on /dev/fb0), and my conclusion is that LVGL has some serious input deficiencies:

1. the 'evdev' driver in LVGL doesn't support keyboards, only mouse/touch. The only way a keyboard appears to work is with the SDL(lol......) backend to LVGL. (https://github.com/lvgl/lvgl/issues/377#issuecomment-416053077)

2. neither the 'evdev' or 'libinput' drivers support any kind of device detection stuff, so we'd have to go back to hard-coding /dev/input/event*, or write our own detection scheme. (https://github.com/lvgl/lv_drivers/blob/master/lv_drv_conf_template.h#L372)

3. no support for tslib, so that's not an option unless we want to implement it in lvgl.. (https://github.com/lvgl/lvgl/issues/1355#issuecomment-574686363)

By clayton craft on 2021-01-29T11:34:53

This all looks fixable to me.

or write our own detection scheme.

given that lvgl sources are "embedded" in app, we can just implement what's missing in backends there

By Alexey Min on 2021-01-29T11:34:53

Yeah I didn't say it wasn't fixable, just that the input stuff in its current form won't work without a lot of work to fix it. And if we're having to implement our own input management, then is the resulting app really that less complicated? At least by farming out that stuff to SDL, there's a larger project with more people/users, who are working on it.

By clayton craft on 2021-01-29T19:19:41

is the resulting app really that less complicated?

It probaly will be somewhat complicated (in backend drivers) yes

At least by farming out that stuff to SDL, there's a larger project with more people/users, who are working on it.

The problem with SDL that even in this larger project no one has worked on framebuffer backend. And we need to keep our stuff that runs from initramfs working on the most (250+) devices that we support. It should be working good with smallest requirements to the system. Given that DirectFB backend in SDL2 is a mess right now, we're back to option 1) implement fbdev backend in SDL, which will require:

• implement graphics output
• implement input subsystem and integrate it with SDL2
• implement whatever is needed for vibrator subsystem

Which seems at least the same amount of work, or even more, that would be needed to switch to lvgl:

• fix libinput support (that should handle all input devices)
• implement vibro feedback (?)

By Alexey Min on 2021-01-30T07:54:43

Just for reference and as discussed in #devel:postmarketos.org here is a small POC for an lvgl-based on-screen keyboard: https://gitlab.com/cherrypicker/unl0kr

By Johannes Marbach on 2021-09-03T11:16:27

Small update on the POC: I've made good progress upstreaming fixes for lvgl's input processing. External pointer and keypad devices work with their libinput driver now. My PR for adding support for full hardware keyboards, including different keymaps, and my other PR for adding automatic input device discovery are in review. The only remaining issue with input processing is supporting multiple devices at once. I have a hacky workaround for this in the unl0kr repo but am still discussing the best approach for getting it upstreamed with the lvgl maintainers.

Meanwhile, as I was waiting for my PRs to be reviewed, I had a little time to play with some of the lvgl widgets. I already have dynamic keyboard layout switching, toggling between light and dark mode and disclosing the cleartext password done. All of these were quite easy to add, just utilizing the widget UI.

Here's a small screencap. Note that for now I focused on functionality, not usability. That means the placement, sizing and styling of UI elements didn't receive much thought yet.

VID_20210906_131354_168_2_2

By Johannes Marbach on 2021-11-14T13:12:53

Since the once-POC is starting to look better and better I took some time today to note down open tasks for making it into a possible osk-sdl replacement. You can find all issues here and those under the v1 milestone are the ones I think are essential for a first version.

@craftyguy whenever you have some time, it would probably be good if you could skim over the v1 issues to see if I forgot anything important.

I'll be going back to work next week so my throughput on this project will, unfortunately, decline drastically. I'll move it along with whatever time I'll have. If anyone else wants to jump in as well, by all means, feel free to comment on the issues and send me merge requests.

By Johannes Marbach on 2021-09-21T20:08:02

mentioned in issue pmaports#1249

By Oliver Smith on 2021-09-29T19:52:15