-
Notifications
You must be signed in to change notification settings - Fork 7.6k
Add parallel interface (DVP) support for ov5640 driver #76124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add parallel interface (DVP) support for ov5640 driver #76124
Conversation
635e3a1
to
73e0ea7
Compare
73e0ea7
to
77c2eeb
Compare
Dear, Please, @ngphibang, @josuah, @erwango, and @loicpoulain, could you take a look at the draft PR to check if I am on the right path? Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this addition!
Progressively, more features of these complex sensors can be integrated from Linux and other sources into Zephyr.
This looks like almost ready to go. See some few feedback inline. Feel free to dismiss it or address it as relevant.
drivers/video/ov5640.c
Outdated
fmt.width = 1280; | ||
fmt.height = 720; | ||
if (ov5640_is_dvp(dev)) { | ||
/* Set default format to 480x272 RGB565 */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment: This looks like one particular display rather than a common resolution format, though it is also good that the defaults did get tested on actual hardware.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this version of the DVP feature does not support the current resolution configuration (1280x720), it would be better to set it up for the minimal resolution supported by DVP.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah right I understand better now thank you.
Did you try various resolutions and noticed that 480x272 RGB565 was the highest one supported by DVP?
I do not see this resolution anywhere on Linux or the datasheet.
Just curious. :)
DVP should support resolutions higher than 480x272, but due to my board's memory (I'm using only the internal SRAM) and display limitations, I was able to test 160x120, 320x240, and 480x272.
For future version, I'll try to use the external SDRAM and others resolution.
This might need rebasing once #76144 (not in draft state) is merged. |
dts/bindings/video/ovti,ov5640.yaml
Outdated
@@ -5,7 +5,7 @@ description: OV5640 CMOS video sensor | |||
|
|||
compatible: "ovti,ov5640" | |||
|
|||
include: i2c-device.yaml | |||
include: [i2c-device.yaml, video-interfaces.yaml] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this will not have any effects as the properties defined in video-interfaces.yaml
are for an endpoint node, not for the ov5640 node itself. The difficulty here (and the reason why I let #74415 always as draft) is that we cannot know the endpoint is actually a child or a grand child node of the video device node (e.g. ov5640) as ports node is optional, so that we don't know if we should use 1 or 2 levels of child-binding in the video-interfaces.yaml
:
- child-binding:
child-binding:
So, I think the solution for now is that either we:
- Don't include
video-interfaces.yaml
, and just declare the properties in the devicetree. It will compile without any problem. Thevideo-interfaces.yaml
will still serve as a reference. - Or add
compatible: "nxp,zephyr-endpoint"
in the endpoint node and invideo-interfaces.yaml
@josuah Do you have a solution for this ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dear, @ngphibang and @josuah,
If I change from include: [i2c-device.yaml, video-interfaces.yaml]
to include: i2c-device.yaml
, I got the compile error below:
devicetree error: 'bus-type' appears in /soc/i2c@58001c00/ov5640@3c in /home/charlesdias/zephyrproject/zephyr/build/zephyr/zephyr.dts.pre, but is not declared in 'properties:' in /home/charlesdias/zephyrproject/zephyr/dts/bindings/video/ovti,ov5640.yaml
I think that the video-interfaces.yaml
is making an effect on the ov5640 node. Does this make sense?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's because you declared these properties on the ov5640 node. All the properties in video-interfaces.yaml
should be declared in the endpoint subnode (together with remote-endpoint
)
Thanks, @josuah and @ngphibang for your suggestions. I'll fix them during the weekend. |
@CharlesDias Thanks for adding this dvp mode to the ov5640 driver. As being said last time in your raised issue, we were also working on supporting changing frame rates and some controls to the ov5640 driver at the same time. Here is the PR. Sorry that we couldn't push our PR earlier to facilitate your work due to some difficulties in the impelmentation. If possible, could you take into account our changes ? (IMO, some commits in our PR could make the dvp support easier and your PR is a bigger change so it seems easier to rebase your change on ours than vice versa ?) I am available for any helps if needed. |
For sure, @ngphibang, I'll do the rebase by the end of the week. Thanks for being available! |
77c2eeb
to
c0038bd
Compare
Rebase with #76144 |
c0038bd
to
7e2dfdf
Compare
Apply review suggestions. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @josuah, I considered this initial version ready to go (apart from the reviewer's suggestions and the PR dependencies). I added the [DNM] to avoid merging it before the dependencies. I'm not sure if this is the right approach. |
The DNM label gives me a hint that this might be made for this purpose but I would not bet an arm on it... |
Hi @josuah and @ngphibang. Since PR #76144 has been merged, I'm returning to work on this PR. Could you please explain how to retrieve the driver's bus-type? Considering that I have the DT file below: Thanks!
|
|
||
port { | ||
ov5640_ep_out: endpoint { | ||
remote-endpoint-label = "mipi_csi2rx_ep_in"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe having a video RX peripheral remote-endpoint was the reason why you needed this file.
There is an emulated/fake video RX peripheral which you might be able to use for this purpose if you wish to remain hardware independent.
I do not see how using NXP hardware would be an issue for the build-all test though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, @josuah ,
could you please share more information about this "emulated/fake video RX peripheral"? Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Why do you add NXP imxrt1064 overlay to test ov5640 dvp mode ? Does this HW support ov5640 camera sensor ? AFAIU, this should be an ST board (with DCMI), no ?
- This is a built test so I think we need to make a real board overlay to involve the actual related drivers and actual code into the test (for imxrt1170 : ov5640 in csi2 mode --> mipicsi2rx --> csi, for ST board : ov5640 in dvp --> DCMI) as the purpose of the test is to find (compilation) bugs in real drivers.
- Another reason not to use the emulated video drivers is it will test either csi2 or dvp mode, while we need to test both. We already test csi2 on rt1170 so just need to test dvp on the ST board.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you please share more information about this "emulated/fake video RX peripheral"? Thanks!
I missed that message sorry. Here is the PR that introduced it for your own curiosity
#79482
Here is an example of integration for testing the API itself:
https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/drivers/video/api
This PR got merged a bit fast: I am not against replacing it if something else (i.e. sw_generator) makes more sense!
it will test either csi2 or dvp mode
this should be an ST board (with DCMI)
Not sure why I did not think of it, DCMI as a target sounds ideal.
I will be a bit busy for the next 3 weeks, so cannot offer the resources for it until then.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think these are the two unresolved points that could need action. The rest seems ready to land AFAICT.
https://github.com/zephyrproject-rtos/zephyr/pull/76124/files#r1806182604
https://github.com/zephyrproject-rtos/zephyr/pull/76124/files#r1806168527
Thanks once again.
Hi, @josuah, The first one is done. I leave some comments in the second. Thanks for your review! |
@@ -144,7 +147,7 @@ struct ov5640_data { | |||
const struct ov5640_mode_config *cur_mode; | |||
}; | |||
|
|||
static const struct ov5640_reg init_params[] = { | |||
static const struct ov5640_reg init_params_common[] = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe that there are some registers that are needed to be initialized only for CSI2. And if so, these need to be refactored into init_params_csi2()
because leaving them in init_params_common()
will be redundant for dvp mode. But well, seeing that it's difficult to check, we can leave this point for the future. The essential thing is it works for both modes so it's ok for me.
@@ -650,6 +875,11 @@ static int ov5640_stream_start(const struct device *dev) | |||
static int ov5640_stream_stop(const struct device *dev) | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks
@@ -897,15 +1127,15 @@ static int ov5640_enum_frmival(const struct device *dev, enum video_endpoint_id | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW, should the frmival callback in the ov5640 driver be removed as well if it is not tested
Oops, sorry, please forget my above nonsense comment, it seems coming from an AI-chatbot not me (maybe I was in hurry while writting this :( ).
My point was just that you can leave the framerate implementation for the future, it's not a blocking point. But since ov5640 driver now supports both dvp and csi2 modes, leaving it as-is will make people think that set/get/enum_frival()
is actually working / valid for both modes while it is not the case here.
So I mean we should add
if (ov5640_is_dvp()) {
return -ENOTSUP;
}
into set/enum_frival()
(get_frival() does not need it because it works for both modes I think) to clearly say that it is not implemented for DVP in ov5640 driver. I think it will be not an issue at all if DCMI driver does not implement these callback and the VIDEO API returns -ENOSYS.
Hope it is clear now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks
@@ -53,6 +55,13 @@ | |||
reg = <0x3>; | |||
reset-gpios = <&test_gpio 0 0>; | |||
powerdown-gpios = <&test_gpio 1 0>; | |||
|
|||
port { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to go into a board-specific overlay, here the ST board (with DCMI) that you tested the ov5640 in dvp mode with all the connection with the DCMI, otherwise it will not be built I think. Just take the imxrt1170 overlay in this same folder as a reference.
BTW to use the port and endpoint, you need to modify the DCMI driver to use the new video-interface bindings (it's already done in another PR if I remember well ?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ngphibang, thanks! This test framework is a mysterious for me! :) Do you have any reference material when I can learn more about it?
BTW to use the port and endpoint, you need to modify the DCMI driver to use the new video-interface bindings (it's already done in another PR if I remember well ?)
Correctly. This was made by another PR. I haven't had any need to modify the DCMI driver so far.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ngphibang and @josuah, I removed the imxrt1064 overlay and created on for stm32h7b3i_dk. However, two twister test failed. The twister-build (2) complained against imxrt1064 and twister-build (3) against native_sim.
Could you please give me some help? Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@CharlesDias : It's because in the ov5640 driver, you get the bus type with:
.bus_type = DT_PROP(DT_CHILD(DT_INST_CHILD(n, port), endpoint), bus_type),
but when the ov5640 driver is tested on the native_sim (drivers.video.build test) and the mimxrt1064_evk (drivers.video.mcux_csi.build test) platforms, the property bus-type
is not defined.
So, to fix it, I think the ov5640 node should be removed in the app.overlay
as ov5640 driver tests are already covered in drivers.video.mcux_csi.build
, drivers.video.mcux_csi2rx.build
and drivers.video.stm32_dcmi.build
(the one you will add) tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The bus_type is initialized with the default value VIDEO_BUS_TYPE_CSI2_DPHY when bus-type property is not declared in the device tree. This way, I don't need to make changes on the video test folder for now.
I'll add the test case for DCMI driver in another PR. I have to study the test framework fir
|
||
port { | ||
ov5640_ep_out: endpoint { | ||
remote-endpoint-label = "mipi_csi2rx_ep_in"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Why do you add NXP imxrt1064 overlay to test ov5640 dvp mode ? Does this HW support ov5640 camera sensor ? AFAIU, this should be an ST board (with DCMI), no ?
- This is a built test so I think we need to make a real board overlay to involve the actual related drivers and actual code into the test (for imxrt1170 : ov5640 in csi2 mode --> mipicsi2rx --> csi, for ST board : ov5640 in dvp --> DCMI) as the purpose of the test is to find (compilation) bugs in real drivers.
- Another reason not to use the emulated video drivers is it will test either csi2 or dvp mode, while we need to test both. We already test csi2 on rt1170 so just need to test dvp on the ST board.
52ce484
to
37229cc
Compare
@ngphibang, I added the
inside of |
|
||
port { | ||
ov5640_ep_out: endpoint { | ||
remote-endpoint-label = "dvp_ep_in"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where is the dvp_ep_in
? I think you need to add the receiver (DCMI) node. Moreover, a test case should be added for the dcmi driver in testcase.yaml
as well, otherwise it will not be tested (your overlay is not used at all).
Also, people have forgot to add test cases when they added their new drivers and reviewers forgot to mention that too :-). They are:
- stm32_dcmi
- esp32_dvp
- smartdma
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, @ngphibang
I can add this test case in another PR. I have to learn the test framework first. :-)
@@ -53,6 +55,13 @@ | |||
reg = <0x3>; | |||
reset-gpios = <&test_gpio 0 0>; | |||
powerdown-gpios = <&test_gpio 1 0>; | |||
|
|||
port { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@CharlesDias : It's because in the ov5640 driver, you get the bus type with:
.bus_type = DT_PROP(DT_CHILD(DT_INST_CHILD(n, port), endpoint), bus_type),
but when the ov5640 driver is tested on the native_sim (drivers.video.build test) and the mimxrt1064_evk (drivers.video.mcux_csi.build test) platforms, the property bus-type
is not defined.
So, to fix it, I think the ov5640 node should be removed in the app.overlay
as ov5640 driver tests are already covered in drivers.video.mcux_csi.build
, drivers.video.mcux_csi2rx.build
and drivers.video.stm32_dcmi.build
(the one you will add) tests
@@ -0,0 +1,49 @@ | |||
/* | |||
* Copyright (c) 2024, Charles Dias <charlesdias.cd@outlook.com> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2025
37229cc
to
321cda9
Compare
Rebase |
321cda9
to
39f5c8e
Compare
Fix format code |
e937904
to
fd558f8
Compare
@ngphibang and @josuah The I'll add the test case for DCMI driver in another PR. I have to study the test framework first! :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having a default bus type set to MIPI CSI allows to preserve backward compatibility, so no migration guide required I think.
Thank you for the few review/modification cycles! LGTM.
@CharlesDias It's ok to add test case in another PR. I created this issue to remind us. Just a nit, should the commit title be "drivers: video: ov5640: add DVP support" to indicate its scope is only on ov5640 ? otherwise LGTM |
Improve the ov5640 video driver to provide parallel interface (DVP) support Signed-off-by: Charles Dias <charlesdias.cd@outlook.com>
fd558f8
to
4a29357
Compare
Hi @ngphibang and @josuah Commit message updated. Could you please remove the DNM flag? Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fantastic work, thanks @CharlesDias, let's get this in before the feature freeze.
Related to the issue #74353.
These PRs are requires:
This PR is related to:
Testing on STM32H7B3I-DK
west build -p -b stm32h7b3i_dk --shield st_b_cams_omv_mb1683 samples/subsys/video/capture_to_lvgl/
west build -p -b stm32h7b3i_dk --shield st_b_cams_omv_mb1683 samples/drivers/video/capture_to_lvgl/
document_5147857391924020551.mp4