Pinephone cedrus driver

changed the description

By Fabio Sgamma on 2020-09-24T14:06:21

changed the description

By Fabio Sgamma on 2020-09-24T14:15:59

changed milestone to %[device] PinePhone support

changed the description

By Fabio Sgamma on 2020-09-24T14:57:54

Hi, i'm making some tests and i have found that cedrus is enabled. The module is called "sunxi-cedrus" and on dmesg list the activation with "cedrus 1c0e000.video-codec: Device registered as /dev/video1". For some strange reason, sunxi-cedrus is not listed on lsmod, that's why i have not noticed it. On dmesg there is no error regarding cedrus, so it should have been enabled correctly.

By Fabio Sgamma on 2020-09-28T11:52:05

sunxi-cedrus is not listed on lsmod

That's because it's built-in in our kernel (=y), not as module (=m):

https://gitlab.postmarketos.org/postmarketos/pmaports/-/blob/905a5ce34bb9bed1e5e8f74318aba11cf79466b6/main/linux-postmarketos-allwinner/config-postmarketos-allwinner.aarch64#L4215

By Oliver Smith on 2020-09-28T11:52:05

Further testing: Using "cat /sys/class/video4linux/video1/name" report "cedrus", so it is the right device, but when i try to force mpv to use the "video1" device, it gives the same error in the original post. A very strange thing is that when running mpv on "video1" device, i clearly ear the sound of camera shatter that that open. The exact mpv command i have used is "mpv --hwdec=yes av://v4l2:/dev/video1 video_h264.mkv"

By Fabio Sgamma on 2020-09-24T17:55:02

Further testing 2: i have compiled and tried libva-v4l2-request and i got this error that seems to be know and still in progress: https://github.com/bootlin/libva-v4l2-request/issues/24. Seems to be an error related to differences between old and new kernel headers. A fork of libva-v4l2-request bump the kernel support to 5.7.15, maybe it is possible to update again against 5.8.0 : https://github.com/Sash0k/libva-v4l2-request . I have not tested this, i probably will give it a try tomorrow. Unfortunately i'm not good at low level coding, especially regarding arm boards. Maybe someone interested with the right skill can fix it. There is also a parallel effort to support libva-v4l2-request directly on ffmpeg. https://github.com/Kwiboo/FFmpeg/commits/v4l2-request-hwaccel-4.2.2

By Fabio Sgamma on 2020-09-24T19:44:18

After some additional research i have discovered that LibreElec use a patched FFMPEG that have libva_v4l2_request support directly integrated. The patch should be this: https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/multimedia/ffmpeg/patches/v4l2-request/ffmpeg-001-v4l2-request.patch Maybe who maintain the FFMPEG package could try to apply the patch and see if and how works. I know that PostmarketOS support a lot of devices and this patch (if work as expected), should enable basically a VAAPI driver on all Allwinner devices, not only Pinephone.

By Fabio Sgamma on 2020-09-28T11:52:03

This mega-patch has multiple patches squashed together, from the commit message:

Patch created using revisions f6ae50a..3e95632
from branch v4l2-request-hwaccel-4.3 of https://github.com/Kwiboo/FFmpeg

By Oliver Smith on 2020-09-28T11:52:03

Hi @uniformbuffer!

All of this looks a bit WIP... if it's actually working, we could consider forking a package and shipping this in postmarketOS. But the maintenance effort has to be reasonable.

Are there active efforts to upstream any of the forks you have linked to?
- If not, consider contacting the people who made them and asking them if they intend to continue/upstream this.
Can you try some more to make any of these forks work?

By Oliver Smith on 2020-09-29T18:42:32

I made some research and seems that the bootlin libva-v4l2-request is not a WIP, but a "working" implementation. The problem is that it rely on some kernel headers that are not stable, so they has changed in time, making it not working on new kernels (if i'm not wrong it was supposed to work with kernel 5.4.something). As far as i know all projects that try to make cedrus to work are not updated from June, so i don't know if someone is still developing. If you are interested i could try to update the bootlin libva-v4l2-request with the headers of kernel 5.8 (currently stable on Pinephone). This is not "my field", i develop at application level, not kernel/driver level, but i can make a try. Since i have a Pinephone i can test directly the developed patch.

By Fabio Sgamma on 2020-09-29T18:42:32

I have forked the libva_v4l2_request repository and something work using the "patched" branch: https://github.com/uniformbuffer/libva-v4l2-request/tree/patched.

I have updated the kernel headers to 5.8.0 and applied a patch among the pending ones that fix a problem with a function when compiling libva_v4l2_request for aarch64. After that i have followed the instruction here https://xnux.eu/devices/feature/cedrus-pp.html and vainfo recognize the library and all it's features. Mpv report to use vaapi-copy.

Unfortunately when running mpv with options for vaapi i got a green screen. I have this problem also with Firefox when acceleration layer is enabled. Since on firefox i have not setted to use vaapi (it is triggered by environment variable that i have used on a terminal, not globally), maybe the problem is not into the v4l2_request driver, but in something else. On the xnux size is reported "If your libva-v4l2-request, this should play the video in Linux console (no X or wayland running).", so maybe there is something wrong with surface management. To be noticed is that when running mpv with vaapi on Pinephone, the audio work. The tested video is a h254 encoded video at 1280x720p.

Also, when using mpv i do not have added the "-vo=drm" since i run it on Phosh (honestly i don't know how to switch to virtual terminal).

I will send updates as i found something new, Have a good day

By Fabio Sgamma on 2020-09-29T22:09:59

I made some additional testing and the problem seems to be related to surface allocation. Every time mpv is launched with hardware acceleration it report some surface destroy error. After some try, mpv is no more able to even start the green screen because v4l2 fail the buffer allocation (i imagine for out of space), so start using software decoding.

I will look for the root of the problem, probably there is some problems in surface management.

By Fabio Sgamma on 2020-09-29T22:53:25

Well, i have looked a bit at the code and seems to be more a "demostration" that a work in progress. There are a lot of unfreeded allocations and some of them are even not needed. To have a better prospective i need to learn a bit of VA API and V4L2.

By Fabio Sgamma on 2020-09-30T13:55:49

Maybe i make it. I have recreated the repository by pulling some pending requests and by removing the surface deinizialization on VaContext destuction since seems that is responsability of the client to do this, making mpv complaining about double free resource. Now seems working, mpv report "using vaapi-copy" without throwing any error, even if it is slower than software decoding. Maybe it's only a matter of optimizing the library. You can find the repo at https://github.com/uniformbuffer/libva-v4l2-request. Eventually use only "master" branch, other branches are tests that i make from time to time. I will look deeper into the code for some optimizations. Have a good day

By Fabio Sgamma on 2020-09-30T17:51:45

Hi, as i have read on another issue, you are going to ship Megi's kernel on PostmarketOS. This could also bring some good news for this issue since Megi's kernel have some specific fix for Cedrus and he have already made some videos about working hardware decoding. As he has stated here, hardware decoding is working with gstreamer 1.18 or patched ffmpeg. Maybe after Megi's kernel has been stabilized, more tests about this topic could be made. The only bad luck is that Alpine edge ships gstreamer up to version 1.16, so i need to install separately for testing.

By Fabio Sgamma on 2020-10-05T18:19:49

Hi, i have make some tests to make cedrus working after Megi's kernel update. As i have understood, a patch to enable v4l2 request api for ffmpeg exist, but i don't know if it has been merged. Anyway to enable it ffmpeg need to be compiled with "--enable-v4l2-request", and looking at the used compile flags obtained running "ffmpeg", it is not among them, so i suppose that it is not enabled anyway.

The second option is to use gstreamer 1.18 that should work out of the box. I have found a post on pine64 forum from Megi reporting that gstreamer 1.18 works (https://forum.pine64.org/showthread.php?tid=11506&pid=78411). Alpine Edge ships gstreamer 1.16, so i have tried to compile the package myself from git to make some testing, but after the build i do not get bad, ugly and good plugins, so i suppose i need to compile them in other ways. Also i have not found the way to temporary replace system gstreamer with the one i have compiled (sorry, i'm not enough skilled). I will keep trying to make it working, but i wanted to share this anyway, so maybe someone more expert can give me some suggestion (or a hand).

Have a good day

Edit: i have tried to get and compile the master branch of ffmpeg and it does not recognize the "--enable-v4l2-request", so i suppose that the patch has not been merged yet; so to get it working, a special fork need to be compiled.

By Fabio Sgamma on 2020-11-06T18:50:45

Thanks for sharing your progress, @uniformbuffer!

Consider joining the chat for live collaboration while packaging stuff etc.: https://wiki.postmarketos.org/wiki/Matrix_and_IRC

By Oliver Smith on 2020-11-14T17:28:42

I will surely give a look, but i'm not so skilled to join a conversation :D It will be surely a learning experience.

By Fabio Sgamma on 2020-11-14T17:29:18

Hi, i have noticed that Alpine Edge have updated the gstreamer version to 1.18, making possible to make some tests. I have tried to copy the gst-launch parameters on the Megi's videos (https://forum.pine64.org/showthread.php?tid=11506&pid=78411). He use a pipeline component called v4l2h264dec (for h264 decoding obviously), but when i use it, i got that it does not exist. As i have understood, it is a "dynamic pipeline", so it is available only if the running system have some "requirement", but looking on the internet i got no explanation about which they are. The only info about it that i have found is the source code (https://github.com/GStreamer/gst-plugins-good/blob/master/sys/v4l2/gstv4l2videodec.c). There is someone that know how to enable it?

Thanks for listening, have a nice day!

EDIT: this pipeline is part of libgstvideo4linux2, that is contained into gst-plugins-good. Looking at the build log (https://build.alpinelinux.org/buildlogs/build-edge-x86_64/community/gst-plugins-good/gst-plugins-good-1.18.1-r0.log) it report that v4l2h264codec (that should contain v4l2h264dec) is compiled and added to the package, so i have no idea why this pipeline is not available.

By Fabio Sgamma on 2020-11-16T19:00:41

Not sure if I read the build log right, but it seems like it ends up in gst-plugins-good-gtk (in alpine, we like to slice packages up as much as possible ):

[406/833] Compiling C object sys/v4l2/libgstvideo4linux2.so.p/gstv4l2mpeg4enc.c.o
[407/833] Compiling C object sys/v4l2/libgstvideo4linux2.so.p/gstv4l2mpeg4codec.c.o
[408/833] Compiling C object sys/v4l2/libgstvideo4linux2.so.p/gstv4l2mpeg2codec.c.o
[409/833] Compiling C object sys/v4l2/libgstvideo4linux2.so.p/gstv4l2jpegenc.c.o
[410/833] Compiling C object sys/v4l2/libgstvideo4linux2.so.p/gstv4l2h265enc.c.o
[411/833] Compiling C object sys/v4l2/libgstvideo4linux2.so.p/gstv4l2h265codec.c.o
[412/833] Compiling C object sys/v4l2/libgstvideo4linux2.so.p/gstv4l2h264enc.c.o
[413/833] Linking target ext/gtk/libgstgtk.so

See gst-plugins-good*:

https://pkgs.alpinelinux.org/packages?name=gst-plugins-good*&branch=edge&arch=aarch64

By Oliver Smith on 2020-11-16T19:00:41

They are exactly the pipelines i look for. Unfortunately after installing gst-plugins-good-gtk i still not get them available. Also for some strange reason, i cannot see the build log of that package, the server return 404.

Edit: mmm, seems that generating gst-plugins-good will also generate gst-plugins-good-gtk, that's why there is no build log on gst-plugins-good-gtk. Nevermind

By Fabio Sgamma on 2020-11-17T12:45:17

The pipeline component used in that example is v4l2slh264dec (not v4l2h264dec, and I think that sl stands for stateless).

I see it is included in gst-plugins-bad (see this merge request https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1141/diffs), but it is not compiled in the Alpine binary. I tried to compile it on my own, adding the -Dv4l2codecs=enabled option to the APKBUILD, but it fails on

[57/1099] Generating view_convert.frag.spv with a custom command ninja: job failed: /usr/bin/glslc -fshader-stage=fragment --target-env=vulkan1.0 -MD -MF ext/vulkan/shaders/view_convert.frag.d ../ext/vulkan/shaders/view_convert.frag -o ext/vulkan/shaders/view_convert.frag.spv.

By Daniele on 2020-12-04T15:48:00

->The pipeline component used in that example is v4l2slh264dec (not v4l2h264dec, and I think that sl stands for stateless).

Oh, i'm sorry, it was difficult to read since on that videos he do not show exactly all the command line, but only parts.

Looking at the error you get, seems like the decoder is based on vulkan. I don't know if you need a vulkan enabled machine to build (maybe you only need glslc to compile shaders), but for sure it is needed for running. Unfortunately there is no vulkan driver for mali 400-450 gpu, so we have to rely on cpu-based vulkan drivers. Alpine Edge provide mesa 20.3, an update that bring also Lavapipe, a cpu-based vulkan driver. Unfortunately to make that working, LLVM-11 is needed and only LLVM-10 is currently shipped, so it is not builded among the others mesa drivers. So..... we have to wait that some more bleeding edge packages are shipped from Alpine Edge.

Thanks a lot for your help, i spent a lot of time on searching for that decoders without success.

By Fabio Sgamma on 2020-12-04T15:48:00

Just to clarify... I'm not sure that the operation that fails is necessary to compile the v4l2slh264dec component. I have zero experience with building gstreamer and Alpine linux, so I just took the APKBUILD for gst-plugins-bad 1.18.1, added the -Dv4l2codecs=enabled meson option, and tried to build the package on the pinephone itself, with abuild -r. I have checked, and glslc was available, but I did not have much time to investigate further.

By Daniele on 2020-12-05T13:38:33

A quick update: I added also the -Dvulkan=disabled option to the APKBUILD, and I have successfully compiled gst-plugins-bad with v4l2slh264dec.

Unfortunately I couldn't really use it, because I couldn't make it work with any videosink. Probably it's more difficult for me because I'm testing on X11, and not on wayland.

I could not have kmssink to display the video anywhere from ssh after ending the X session. I could use the autovideosink to actually show an h264 video using the v4l2slh264dec, but to make it work I had to add the videoconvert component in the pipeline, which I think is what is killing performance (100% cpu and completely stuttering with a 15Mbps h264 video).

But if you're using wayland you should try and see if it works for you. If you prefer, here is the gst-plugins-bad apk that I compiled. gst-plugins-bad-1.18.1-r0.apk

By Daniele on 2020-12-06T10:59:30

The glimagesink shows the first frame of the video, but then it sits there. After some time the xserver freezes, and dmesg shows

[ 175.544046] [drm:lima_sched_timedout_job] *ERROR* lima job timeout

[ 175.544102] lima 1c40000.gpu: fail to save task state from gstglcontext pid 3717: error task list is full

[ 175.544111] lima 1c40000.gpu: pp task error 0 int_state=0 status=1

[ 175.544117] lima 1c40000.gpu: pp task error 1 int_state=0 status=1

By Daniele on 2020-12-07T13:21:51

Sounds like https://gitlab.freedesktop.org/lima/linux/-/issues/33

By Alexey Min on 2020-12-07T13:21:51

Update:

GST_GL_API=gles2 GST_GL_PLATFORM=egl gst-launch-1.0 filesrc location=jellyfish-15-mbps-hd-h264.mkv ! matroskademux ! queue ! h264parse ! v4l2slh264dec ! glimagesink works with good framerate and low cpu utilization, but wrong colors. The blue becomes red.

GST_GL_API=gles2 GST_GL_PLATFORM=egl gst-launch-1.0 filesrc location=jellyfish-15-mbps-hd-h264.mkv ! matroskademux ! queue ! h264parse ! v4l2slh264dec ! videoconvert ! video/x-raw, format=BGR ! glimagesink gives the right colors, but is extremely slow and uses 100% of the cpu

GST_GL_API=gles2 GST_GL_PLATFORM=egl gst-launch-1.0 filesrc location=jellyfish-15-mbps-hd-h264.mkv ! matroskademux ! queue ! h264parse ! v4l2slh264dec ! glupload ! glcolorconvert ! 'video/x-raw(memory:GLMemory), format=BGR' ! glimagesink is ok with performance, but the colors are wrong again.

Anyone with more experience with gstreamer has suggestions?

By Daniele on 2020-12-06T14:13:09

The wrong colors is a known problem that i have discovered recently (https://github.com/celluloid-player/celluloid/issues/580). Seems have something to do with some errors in GDK and in gtk 4.0 have been already fixed. Also as far as i have understood, lima driver do not work correctly if used with opengl, but do work correctly with opengl-es. In fact, after getting hardware decoding, my dream # 2 is to enable webrender on firefox, but do not work because gpu driver have some problems. Anyway..... wow, you have made so much progress in few days, thanks for sharing. I will try to compile the bad-plugins to make some tests.

By Fabio Sgamma on 2020-12-07T16:05:20

I have builded gst-plugins-bad with the option you have used and tried the 3 configuration of gst-launch: The first perform good, but have the color problem. The second have the color right, but the video do not start, it stop at first frame. The third perform bad (seems like the video start and stop continuously) and have color problem.

So we have to wait that gtk updates a bit. Meanwhile i try to look for some gstreamer applications that scale well on mobile. The only video player that i'm aware of using gstreamer is totem, but do not scale well on phone.

Thanks again for your help!

Edit: i have tested it on PinePhone Phosh(wayland) PMOS Edge

By Fabio Sgamma on 2020-12-07T16:40:37

Thank you for testing. Since you're on wayland, can you the waylandsink?

Should be something like gst-launch-1.0 filesrc location=jellyfish-15-mbps-hd-h264.mkv ! matroskademux ! queue ! h264parse ! v4l2slh264dec ! waylandsink. I think you don't need the opengl env vars.

By Daniele on 2020-12-07T17:06:18

Hi, unfortunately do not work. I have tried with and without gl env variables, but the result is the same:

Setting pipeline to PAUSED ...

** (gst-launch-1.0:3447): WARNING **: 13:02:53.389: Wayland compositor is missing the ability to scale,>
Pipeline is PREROLLING ...
WARNING: from element /GstPipeline:pipeline0/GstMatroskaDemux:matroskademux0: Delayed linking failed.
Additional debug info:
gst/parse/grammar.y(540): gst_parse_no_more_pads (): /GstPipeline:pipeline0/GstMatroskaDemux:matroskade>
failed delayed linking some pad of GstMatroskaDemux named matroskademux0 to some pad of GstQueue named >
ERROR: from element /GstPipeline:pipeline0/GstMatroskaDemux:matroskademux0: Internal data stream error.
Additional debug info:
../gst/matroska/matroska-demux.c(5860): gst_matroska_demux_loop (): /GstPipeline:pipeline0/GstMatroskaD>
streaming stopped, reason not-linked (-1)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

I'm not very familiar with gstreamer, so i don't know how to deal with this error. Anyway i have noticed that waylandsink do not work even on software rendering. I think there is an error somewhere.

By Fabio Sgamma on 2020-12-08T12:09:31

Possibly fixed with this lima fix https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8549 merged to mesa master.

By Erico Nunes on 2021-02-02T19:27:40

added device-pine64-pinephone label

No updates for a year so closing. Please re-open if it's still an issue

By Bart Ribbers on 2022-01-28T15:39:55

closed

@Zapeth asked me to re-open this since it's still something they want. So, re-opening!

By Bart Ribbers on 2022-02-07T00:36:59

Hi, you didn't reopen the issue.

By madr on 2022-02-07T00:36:59

Pinephone cedrus driver

Describe your issue

What's the expected behaviour?

What's the current behaviour?

How to reproduce your issue?

What device are you using?

On what postmarketOS version did you encounter the issue?

On what environment did you encounter the issue?

Phone environments

Other

How did you get postmarketOS image?

What's the build date of the image? (in dd-mm-yyyy format)

Additional information

Child items 0

Activity