What is the format of camera frames received using the kvswebrtcViewer?

[setareh-soltanieh]

I'm currently working on integrating this codebase to stream camera frames from my mobile device into ROS. So far, I've successfully created a ROS package from the project and converted the Viewer's code from C to C++, allowing me to create a ROS node from this codebase.

Now, I'm aiming to publish my camera frames to a ROS topic (sensor_msgs/Image). However, to achieve this, I need to read each image and convert it to the cv::Mat format before publishing it to ROS.

My challenge lies in understanding how the image data is being received. I've observed that the PBYTE->frameData pointer points to the first element of the image buffer. However, without knowledge of the image's width and height, I'm unable to correctly convert it to a cv::Mat object.

Could someone provide guidance on how I can retrieve the width and height of the image data in order to properly convert it to a cv::Mat format?

Thank you for your assistance!

[setareh-soltanieh]

I'm still working on the same issue, but haven't figured out how to solve this.
I am using libav to convert H264 camera frames to cv::Mat but the problem is that I can only use this conversion for key frames. Also, the only key frames are the first and third frame while I was expecting to receive key frames periodically. Can anyone help me in converting each frame to cv::Mat?

[stefankiesz]

Hi Setareh, what are you using to encode your frames? The key frame interval should be dependent on the encoder settings. As for converting each frame into an image using this method, either setting the encoder to only use key frames or decoding the frames prior to image cv::Mat conversion could be possible. Is using something like a GStreamer pipeline possible with your project?