[0.12.1 armhf, rip zero 2w, Bullseye] Issues with video playback, error in ofGstUtils.

[sebsongs]

Hi,

Just FYI, I'm fairly new to openFrameworks.

I'm trying to get 720p and 1080p h.264 video files playing on a Raspberry Pi Zero 2W using openFrameworks and the ofVideoPlayer.

I'm running 2023-05-03-raspios-bullseye-armhf.img.xz and have followed all instructions for installing OF. I installed version 0.12.1 from git.

As of now I just have the basic empty app with the ofVideoPlayer playing and looping a single video file. My first test was with the Fingers.mov video that's available in the example project, and that played back fine. But playing back any other mp4 or mov file has not worked yet. Sound IS working though, it's just the video that is not streaming as it should.

I'm running the OF app in command line using the "startx ./app -- -s off" command. And I know the app is running properly as there is sound from the video.

Video file is an mp4 downloaded from YouTube in 720p resolution.
14,6 MB size
H.264
29.97 FPS
299,09 kbps
ITU-R BT.709
1-1-1

The error message I get is:
modeset(0): Initialising kms colour map for depth 24, 8 abc.
[ error ] ofGstUtils: gstHandleMessage(): embedded video player halted for plugin, module v4l2h264dec0 reported: No valid frames decoded before end of stream
[warning] ofGstUtils: didn't received EOS in 5s, closing pipeline anyway

I think that the problem probably has to do with Gstreamer or the codecs I'm using? I've found some threads suggesting to install FFmpeg, and make sure that all the codec scripts for OF have been run - and I have done all that. Video playback is still a no go.

If it helps, playing back the files with ffplay works perfectly fine.

I'd much appreciate some help here. :)

[ofTheo]

Hmm I see that error message connected to this firmware issue:
raspberrypi/firmware#1673

Did you do install_codecs.sh for the OF install scripts?

Also did you try:
movie.setPixelFormat(OF_PIXELS_NATIVE);

There is a very long thread here that discusses speeding up HD video for a Rpi 4 - some of which might be relevant:
https://forum.openframeworks.cc/t/raspberry-pi-ofvideoplayer-fullhd-movie/26926/

cc @dimitre who has done some video stuff with RPI

[dimitre]

in fact I never did video with RPI only other interactive stuff.
@sebsongs are you using it headless? is the video displayed or something else?
Kimi K2 suggested this:
ofSetVideoDecoder(OF_VIDEO_DECODER_GST); // Use GStreamer software pipeline

[sebsongs]

Thanks for pitching in @ofTheo and @dimitre !

I did indeed run install_codecs.sh. I also just tried to compile a version setting the pixel format. No change.

I've read the threads you linked and I found some things in there that I haven't tried. Will try them out tomorrow and report back.

I'll try some other video formats and sizes as well to rule things out.

[sebsongs]

Alright, I've done some more testing and come a little bit further.

I tried recoding my mp4 file to 30fps instead of 29,97, that made no change.
Then I tried making a h.265 .mkv rendering instead, and now I do get sound and video, but the framerate of the video is really slow. I think it's probably the same behaviour as here: .

movie.setPixelFormat(OF_PIXELS_NATIVE); makes no difference whatsoever.

Also, I can't really see a difference between running the Legacy non-GL driver or the GL full KMS driver. Would that mean perhaps that OpenGL is not being used and that could be why I'm getting so poor performance?

@dimitre I can't see reference anywhere to ofSetVideoDecoder(OF_VIDEO_DECODER_GST);. What is it and where in the code should I put it?

are you using it headless? is the video displayed or something else?

I am planning to use it headless, right now I'm connected via HDMI to a screen and have keyboard and mouse directly connected to the rPi. I'm not working in X though, but working from command line and writing code in Nano. Running the compiled app with "startx ./app -- -s off".

My app consists of a purple background in 1920x1080, and I draw the video on a 1280x720 rectangle that's offset 100x100, just so I know that the app starts and is actively drawing on the screen.

So, several problems here to try to fix:

• h.264 mp4 720p and 1080p files will not display any video, unless I use the Fingers.mp4 file which is really low resolution (is resolution the issue?)
• The h.265 mkv 720p file I made does display video albeit with a really low framerate.
• In both cases audio works just fine.

Is there any way to confirm that my app is actually using the GPU and OpenGL to display the video? I'm suspicious that I might be stuck in software rendering, and that's why it works so poorly.

Next I will render some different lower resolution mp4 and mkv files and see if there's a threshold where it starts to work.

Thanks again, and let me know if you have more suggestions!

Edit: I just tried mp4 480p h.264, no image here either. Also tried encoding an mkv file with 720p MPEG-4. There was image but again really low framerate.

[dimitre]

I would test mjpg as a codec as it will be much lighter to decode than h264/5 in a tiny computer.
file will be larger and frame access more immediate

[sebsongs]

I would test mjpg as a codec as it will be much lighter to decode than h264/5 in a tiny computer. file will be larger and frame access more immediate

I'm encoding that now with ffmpeg, directly on the RPi, so it will take a little while.
Though, as I wrote before, playing the files with ffplay works perfectly fine, so there is evidence that the RPi can handle it.

[sebsongs]

@dimitre I just tested my newly encoded .avi file with mjpeg codec. Video is showing but only at 1-2 fps. Way worse than h.265. Hmm.

[sebsongs]

Another update!

I tried playing the different videos with Gstreamer directly, and it's working way better in some instances.
I'm using the command: gst-launch-1.0 -v playbin uri=file:///pathtofile

Anything coded with h.264 just won't display an image, but sound works - so the h.264 thing seems to be a gstreamer issue, not anything to do with OpenFrameworks.

However, previously I got mkv files with h.265 encoding to work albeit with very slow frame rates in OF. Playing them with gst-launch is flawless, no lag, no issues at all.
Also, the mkv file with mpeg4 encoding is also working perfectly. Both these files are 720p. I've yet to encode 1080p mkv files using h.265 and mpeg4 but I will try that next.

So, still something is going on with the ofVideoPlayer, as I can play back some files perfectly fine with Gstreamer, but the same files are having issues in OF.

Again, I'm a newbie when it comes to OpenFrameworks, so I don't know how gstreamer is implemented. I'm sure there are a lot more to it than what I've discovered here.

[ofTheo]

The main issue I believe with OF's implementation is that we have to grab the pixels from the video so we end up a CPU read whereas GStreamer can play the video along a hardware accelerated pipeline.

It should be possible to find a faster playback combo, but it sometimes involves playing around with the codecs and OF pixel format being used.

Adding @jvcleave as he might have some insight.

[sebsongs]

The main issue I believe with OF's implementation is that we have to grab the pixels from the video so we end up a CPU read whereas GStreamer can play the video along a hardware accelerated pipeline.

Ah, that makes sense, I was thinking that perhaps OF can't utilize the hardware acceleration the same way.

It should be possible to find a faster playback combo, but it sometimes involves playing around with the codecs and OF pixel format being used.

The problem for me is that I'd like my final "thing" that I'm making to accept many different video formats and not having to ask people to encode to specific video formats. Maybe I'm better off using gst-play or ffplay directly in this case - or getting more capable hardware. However I'm trying to keep costs down so it's tricky to find the sweetspot.

[sebsongs]

Just FYI, I'm moving away from OF for this project. Experimenting with the gst-play application, I've had really snappy performance for what I want to do. So I'm investing some time into learning to compile a custom version of that application to cater to my exact needs.

Still interested in solutions to this problem though, if someone has more insight.

[dimitre]

Great! one middle way of doing this is using OF, but avoiding ofVideoPlayer, making your own class from scratch.
if you do this and are willing to share your findings maybe OF can benefit from it, as Gstreamer code is more than decade old.

[jvcleave]

The main issue I believe with OF's implementation is that we have to grab the pixels from the video so we end up a CPU read whereas GStreamer can play the video along a hardware accelerated pipeline.

yes out of the box I think this is true. ofVideoPlayer does just use the system GStreamer install, which does support hardware-accelerated H.264 decoding on RPI. On older Pis (pre-Pi 4), this often went through OpenMAX, and GStreamer had OMX plugins for that path. On Raspberry Pi 4 and newer OS releases, OpenMAX has been deprecated and the stack has moved to V4L2 for hardware video decode. I would assume GStreamer uses V4L2 to handle fully hardware-accelerated pipelines where decode → upload → render stays on the GPU.

If you don’t need pixels and only want to render via shaders or write directly to a texture, that path can remain fully GPU-accelerated and be quite fast. The RPI3 did pretty well with 720p.

If you don't even need textures the RPI can basically render directly to the screen which can do 1080, etc

If you need pixels (e.g. opencv), you will inevitably pay a cost because the decoded frames must be read back from GPU memory into system RAM. If you really dive deep you may be able to find an RPI specific hack to make this faster.

if you want to get as close to a drop in for ofVideoPlayer I would look at ofVideoPlayer::setPipelineWithSink().

• Build and validate a GStreamer pipeline outside of openFrameworks that uses V4L2 decode and a GPU-friendly sink.

• Once that works, see if i could use the same pipeline for use inside ofVideoPlayer via setPipelineWithSink .

also ofVideoPlayer isn't too hard to re-implement which is what I did with ofRPIVideoPlayer

as for "accept many different video formats" you are always going to need to get to h264 somehow to make it fast. the RPI has hardware acceleration for encoding as well though so I would assume gst/V4L2 have a path for this as well