video: introduction of STM32 VENC driver for H264 hardware video compression #92884
+1,678
−15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Video buffers being often quite big, it might not be possible to allocate them from the default heap. For that reason, rely on the mem sw attr heap allocator in order to design from where to allocate buffers using which attributes. Signed-off-by: Alain Volmat <alain.volmat@foss.st.com>
Add MEM_SW_ATTR on the PSRAM memory in order to have it become accessible via the mem_attr_heap framework. On this device since we need to allocate a rather large framebuffer, we can do that on the PSRAM memory. Signed-off-by: Alain Volmat <alain.volmat@foss.st.com>
Video encoder requires non-cacheable memory. Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Add YUV420 semi-planar pixel format (NV12). Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Add YUV420 semi-planar pixel format (NV12). Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Addition of description for the STM32 Video encoder (VENC). Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
The STM32 video encoder (VENC) peripheral is a hardware accelerator allowing to compress RGB/YUV frames into H264 video bitstream chunks. Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Add node describing the venc in stm32n6.dtsi Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Enable video encoder on stm32n6570_dk discovery board. Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Sync with video capture sample. Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Add video compression support to lowerize network bandwidth. To visualise camera content on host PC, use GStreamer command line: $> gst-launch-1.0 tcpclientsrc host=<board ip address> port=5000 ! decodebin ! autovideosink sync=false Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Add YUV420 semi-planar support (NV12). This is the video encoder prefered pixel format. Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Allow to configure the number of allocated capture frames. This allows to make tradeof between framerate versus memory usage. 2 buffers allows to capture while sending data (optimal framerate). 1 buffer allows to reduce memory usage but capture framerate is lower. Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Add configuration files for the stm32n6570_dk board. This enables streaming over ethernet of the images captured by MB1854 camera module compressed in 1920x1088 H264 video bitstream. Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Ethernet over USB support. Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
Disable ethernet phy support to use ethernet over USB. Signed-off-by: Hugues Fruchet <hugues.fruchet@foss.st.com>
|
The following west manifest projects have changed revision in this Pull Request:
⛔ DNM label due to: 1 added project and 1 unreachable repo Note: This message is automatically posted and updated by the Manifest GitHub Action. |
|
| @@ -58,6 +58,20 @@ config VIDEO_I2C_RETRY_NUM | |||
| The default is to not retry. Board configuration files or user project can then | |||
| use the number of retries that matches their situation. | |||
|
|
|||
| config VIDEO_BUFFER_USE_MEM_ATTR_HEAP | |||
| bool "Use mem attr api for video buffer" | |||
There was a problem hiding this comment.
Suggested change
| bool "Use mem attr api for video buffer" | |
| bool "Allocate video buffer from mem_attr heap" |
+ maybe this should depends on MEM_ATTR_HEAP
| default n | ||
|
|
||
| config VIDEO_BUFFER_MEM_SW_ATTRIBUTE | ||
| int "Mem SW attribute for video buffer" |
There was a problem hiding this comment.
Suggested change
| int "Mem SW attribute for video buffer" | |
| int "Heap allocations attribute" |
Comment on lines
+70
to
+73
| Mem SW attribute for video buffer: | ||
| 1: ATTR_SW_ALLOC_CACHE | ||
| 2: ATTR_SW_ALLOC_NON_CACHE | ||
| 4: ATTR_SW_ALLOC_DMA |
There was a problem hiding this comment.
It would be best to avoid duplicating information from the header.
Proposal:
Suggested change
| Mem SW attribute for video buffer: | |
| 1: ATTR_SW_ALLOC_CACHE | |
| 2: ATTR_SW_ALLOC_NON_CACHE | |
| 4: ATTR_SW_ALLOC_DMA | |
| Attribute to request when performing allocations from mem_attr heap. | |
| Refer to include/zephyr/dt-bindings/memory-attr/memory-attr-sw.h for | |
| the list of allowed values (one of ATTR_SW_*). | |
| For example, if you want to use ATTR_SW_ALLOC_CACHE defined as: | |
| #define ATTR_SW_ALLOC_CACHE BIT(0) | |
| This symbol should be set equal to 1 (equal to (1 << 0)). |
Comment on lines
+33
to
+35
| #define VIDEO_COMMON_HEAP_ALLOC(align, size, timeout) \ | ||
| mem_attr_heap_aligned_alloc((CONFIG_VIDEO_BUFFER_MEM_SW_ATTRIBUTE << \ | ||
| DT_MEM_SW_ATTR_SHIFT), align, size) |
There was a problem hiding this comment.
Suggested change
| #define VIDEO_COMMON_HEAP_ALLOC(align, size, timeout) \ | |
| mem_attr_heap_aligned_alloc((CONFIG_VIDEO_BUFFER_MEM_SW_ATTRIBUTE << \ | |
| DT_MEM_SW_ATTR_SHIFT), align, size) | |
| #define VIDEO_COMMON_HEAP_ALLOC(align, size, timeout) \ | |
| mem_attr_heap_aligned_alloc(DT_MEM_SW(CONFIG_VIDEO_BUFFER_MEM_SW_ATTRIBUTE), \ | |
| align, size) |
Comment on lines
+58
to
+60
| #if defined(CONFIG_VIDEO_BUFFER_USE_MEM_ATTR_HEAP) | ||
| mem_attr_heap_pool_init(); | ||
| #endif |
There was a problem hiding this comment.
Nit: should check that return value is 0 or -EALREADY
| { | ||
| /* VENC HW REset */ | ||
| __HAL_RCC_VENC_FORCE_RESET(); | ||
| k_sleep(K_MSEC(1)); |
There was a problem hiding this comment.
Maybe k_busy_wait with a lower delay (if upper bound for tRESET is known and <<< 1ms).
Although polling a "is IP under reset" bit would be better. (Ideally, we would even call reset_line_toggle with info from DT, but driver doesn't perform polling...)
Comment on lines
+710
to
+711
| if (ret) | ||
| return -1; |
There was a problem hiding this comment.
Suggested change
| if (ret) | |
| return -1; | |
| if (ret) { | |
| return -1; | |
| } |
or even ret < 0 and propagate ret
| } | ||
| LOG_DBG("output=%p\n", output); | ||
|
|
||
| if (!(data->frame_nb % 30) || data->resync) { |
There was a problem hiding this comment.
If this 30 is framerate dependent, it should come from a macro.
| static int video_stm32_venc_set_stream(const struct device *dev, bool enable, | ||
| enum video_buf_type type) | ||
| { | ||
| struct video_stm32_venc_data *data = dev->data; |
There was a problem hiding this comment.
Suggested change
| struct video_stm32_venc_data *data = dev->data; | |
| ARG_UNUSED(type); | |
| struct video_stm32_venc_data *data = dev->data; |
| ISR_DIRECT_DECLARE(stm32_venc_isr) | ||
| { | ||
| VENC_EWL_TypeDef *enc = &ewl_instance; | ||
| u32 hw_handshake_status = READ_BIT(VENC_REG(BASE_HEncInstantInput >> 2U), (1U << 29U)); |
There was a problem hiding this comment.
No HW define for (1 << 29)? (Nit: should be 1U << 29)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces H264 hardware video compression thanks to VENC peripheral of STM32N6.
A new stm32 video encoder driver has been introduced which relies on vc8000nanoe software stack [1].
This is a first basic porting to enable H264 encoding use-case.
RAM memory resources required by VENC encoding process are allocated in external PSRAM, camera frames
captured by DCMIPP and encoded by VENC are also allocated in external PSRAM.
The video encoder driver currently lack of controls to configure the encoding process, default configuration
targets H264 1080p 2Mb/s constant bitrate.
In order to test, the tcpserversink video sample has been enhanced to support video compression in order to stream small H264 compressed video chunks instead of big raw uncompressed images.
To build sample: (flash using ST-Link and boot with FSBL [2])
The default configuration allows to capture and stream 1080p camera content from STM32N6 which can be received, decoded and displayed by an host PC using a GStreamer command line:
Tests have been made using ethernet over USB between STM32N6 and host PC thanks to the temporary last 2 commits.
When using ethernet over USB, set the ethernet interface through USB before starting the GStreamer command line:
To find the right interface, search for "enx" in kernel log
[1] zephyrproject-rtos/hal_stm32#295
[2] https://docs.zephyrproject.org/latest/boards/st/stm32n6570_dk/doc/index.html