Hardware H264/H265(HEVC) decode acceleration for Raspberry Pi 4 64bit

[MELSunny]

Raspberry pi officially announced the 64bit OS on Feb 2022.
On 64bit Raspbian, I found the following hardware acceleration is available on ffmpeg of rpi's distribution RPi-Distro/ffmpeg.
H264 decode/encode
H265 decode

It just need a single commend to enable the V4L2 decoder. (For more informations: jc-kynesim/hello_drmprime)
sudo dtoverlay rpivid-v4l2

I downloaded two sample h265 videos

wget http://www.jell.yfish.us/media/jellyfish-3-mbps-hd-hevc.mkv
wget http://www.jell.yfish.us/media/jellyfish-3-mbps-hd-hevc-10bit.mkv

and tried to convert videos from h265/h265-10bit to h264 with RPi-Distro/ffmpeg, here is the output.

ffmpeg -hwaccel drm -i file:"jellyfish-3-mbps-hd-hevc.mkv" -codec:v h264_v4l2m2m -b:v 16M -codec:a aac -f mp4 jellyfish-3-mbps-hd-hevc_converted.mkv
ffmpeg version 4.3.3-0+rpt3+deb11u1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+rpt3+deb11u1 --toolchain=hardened --incdir=/usr/include/aarch64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom 
--enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq 
--enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx 
--enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-mmal --enable-neon --enable-v4l2-request --enable-libudev --enable-epoxy --enable-sand --libdir=/usr/lib/aarch64-linux-gnu --arch=arm64 --enable-pocketsphinx --enable-libdc1394 --enable-libdrm --enable-vout-drm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, matroska,webm, from 'file:jellyfish-3-mbps-hd-hevc.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso4hvc1iso6
    MAJOR_BRAND     : iso4
    MINOR_VERSION   : 1
    ENCODER         : Lavf56.3.100
  Duration: 00:00:30.10, start: 0.067000, bitrate: 2827 kb/s
    Stream #0:0(und): Video: hevc (Main), yuv420p(tv), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 29.97 tbc (default)
    Metadata:
      CREATION_TIME   : 2016-02-04 02:54:49
      LANGUAGE        : und
      HANDLER_NAME    : hevc@GPAC0.5.2-DEV-rev565-g71748d7-ab-suite
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[hevc @ 0x5587ac9df0] Hwaccel V4L2 HEVC stateless V2; devices: /dev/media0,/dev/video19
[h264_v4l2m2m @ 0x5587ade130] Using device /dev/video11
[h264_v4l2m2m @ 0x5587ade130] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x5587ade130] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x5587ade130] Failed to set gop size: Invalid argument
Output #0, mp4, to 'jellyfish-3-mbps-hd-hevc_converted.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso4hvc1iso6
    MAJOR_BRAND     : iso4
    MINOR_VERSION   : 1
    encoder         : Lavf58.45.100
    Stream #0:0(und): Video: h264 (h264_v4l2m2m) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 16000 kb/s, 29.97 fps, 11988 tbn, 29.97 tbc (default)
    Metadata:
      CREATION_TIME   : 2016-02-04 02:54:49
      LANGUAGE        : und
      HANDLER_NAME    : hevc@GPAC0.5.2-DEV-rev565-g71748d7-ab-suite
      encoder         : Lavc58.91.100 h264_v4l2m2m
[mp4 @ 0x5587adcab0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0x5587adcab0] Encoder did not produce proper pts, making some up.
frame=  900 fps= 50 q=-0.0 Lsize=   52408kB time=00:00:29.99 bitrate=14312.3kbits/s speed=1.65x
video:52403kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.009842%

ffmpeg -hwaccel drm -i file:"jellyfish-3-mbps-hd-hevc-10bit.mkv" -codec:v h264_v4l2m2m -b:v 16M -pix_fmt yuv420p -codec:a aac -f mp4 jellyfish-3-mbps-hd-hevc-10bit_converted.mkv
ffmpeg version 4.3.3-0+rpt3+deb11u1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+rpt3+deb11u1 --toolchain=hardened --incdir=/usr/include/aarch64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d 
--enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex 
--enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-mmal --enable-neon --enable-v4l2-request 
--enable-libudev --enable-epoxy --enable-sand --libdir=/usr/lib/aarch64-linux-gnu --arch=arm64 --enable-pocketsphinx --enable-libdc1394 --enable-libdrm --enable-vout-drm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, matroska,webm, from 'file:jellyfish-3-mbps-hd-hevc-10bit.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso4hvc1iso6
    MAJOR_BRAND     : iso4
    MINOR_VERSION   : 1
    ENCODER         : Lavf56.3.100
  Duration: 00:00:30.10, start: 0.067000, bitrate: 2986 kb/s
    Stream #0:0(und): Video: hevc (Main 10), yuv420p10le(tv), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 29.97 tbc (default)
    Metadata:
      CREATION_TIME   : 2016-02-04 02:39:21
      LANGUAGE        : und
      HANDLER_NAME    : hevc@GPAC0.5.2-DEV-rev565-g71748d7-ab-suite
File 'jellyfish-3-mbps-hd-hevc-10bit_converted.mkv' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[hevc @ 0x5586f19240] Hwaccel V4L2 HEVC stateless V2; devices: /dev/media0,/dev/video19
[h264_v4l2m2m @ 0x5586f2d140] Using device /dev/video11
[h264_v4l2m2m @ 0x5586f2d140] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x5586f2d140] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x5586f2d140] Failed to set gop size: Invalid argument
Output #0, mp4, to 'jellyfish-3-mbps-hd-hevc-10bit_converted.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso4hvc1iso6
    MAJOR_BRAND     : iso4
    MINOR_VERSION   : 1
    encoder         : Lavf58.45.100
    Stream #0:0(und): Video: h264 (h264_v4l2m2m) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 16000 kb/s, 29.97 fps, 11988 tbn, 29.97 tbc (default)
    Metadata:
      CREATION_TIME   : 2016-02-04 02:39:21
      LANGUAGE        : und
      HANDLER_NAME    : hevc@GPAC0.5.2-DEV-rev565-g71748d7-ab-suite
      encoder         : Lavc58.91.100 h264_v4l2m2m
[mp4 @ 0x5586f2bac0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0x5586f2bac0] Encoder did not produce proper pts, making some up.
frame=  900 fps= 32 q=-0.0 Lsize=   52526kB time=00:00:29.99 bitrate=14344.7kbits/s speed=1.08x
video:52521kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.009842%

RPI4/4GB(RAM) CPU load usage during the conversion is around 0.9 on average, and the converted videos can be correctly played.
However, jellyfin-ffmpeg is not working with the following error.
I am grateful if jellyfin team could provide hardware encode acceleration for RPI, and make the RPI a cost-effective home media centre with jellyfin.

/usr/lib/jellyfin-ffmpeg/ffmpeg -hwaccel drm -i file:"jellyfish-3-mbps-hd-hevc.mkv" -codec:v h264_v4l2m2m -b:v 16M -codec:a aac -f mp4 jellyfish-3-mbps-hd-hevc_converted.mkv
ffmpeg version 4.4.1-Jellyfin Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-shared --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-static --enable-gmp --enable-gnutls --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray 
--enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --toolchain=hardened --enable-cross-compile --arch=arm64 --cross-prefix=/usr/bin/aarch64-linux-gnu-
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, matroska,webm, from 'file:jellyfish-3-mbps-hd-hevc.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso4hvc1iso6
    MAJOR_BRAND     : iso4
    MINOR_VERSION   : 1
    ENCODER         : Lavf56.3.100
  Duration: 00:00:30.10, start: 0.067000, bitrate: 2827 kb/s
  Stream #0:0(und): Video: hevc (Main), yuv420p(tv), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 29.97 tbc (default)
    Metadata:
      CREATION_TIME   : 2016-02-04 02:54:49
      LANGUAGE        : und
      HANDLER_NAME    : hevc@GPAC0.5.2-DEV-rev565-g71748d7-ab-suite
File 'jellyfish-3-mbps-hd-hevc_converted.mkv' already exists. Overwrite? [y/N] y
Device creation failed: -14.
[hevc @ 0x559a63efc0] No device available for decoder: device type drm needed for codec hevc.
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_v4l2m2m))
Device setup failed for decoder on input stream #0:0 : Bad address

[nyanmisaka]

Blocked by jc-kynesim/rpi-ffmpeg#42

[da-anda]

is the linked blocker the reason why there is no "enable h265 HW decoding" setting in JF atm? h264 is already available.

[PimTournaye]

Hi, has there been any progress towards this on this side of things? Sadly don't know enough about low level stuff to help out with this, but if I can help any other way, please let me know.

[Arthedian]

Hi,
@nyanmisaka could you please tell us if there is any progress? It would help a lot.

[nyanmisaka]

@Arthedian
This has been blocked by the fact that scale_v4l2m2m video scaler it not available in upstream. I'm not versed in Linux V4L2 and the closed source Broadcom GPU so you'd better turn to https://github.com/jc-kynesim/rpi-ffmpeg for help. It seems the author jc-kynesim is an employee of the RPi Foundation.