Ultralytics
integrations/sony-imx500/ #17496
giscus[bot]
bot
Replies: 17 comments 44 replies
-
|
👋 Hello, thank you for your interest in Ultralytics and exploring integrations with the Sony IMX500! 🚀 For those looking to dive deeper into this topic, the Ultralytics Docs are a great starting point. They contain sections dedicated to various integrations, including Sony IMX500, with detailed instructions on how to export your YOLO models efficiently. If you're here to report a 🐛 bug, please provide a minimum reproducible example. This will help us better understand and assist with the issue you're facing. For general usage, upgrading to the latest version of Ultralytics might resolve some issues you encounter. Make sure your environment is updated with the latest package: pip install -U ultralyticsFor interactive discussions, consider joining the Ultralytics community on Discord 🎧, our Discourse forum, or participate in threads on our Subreddit. Our team will assist you soon, but for now, these resources should help you get started! 😊 EnvironmentsFeel free to run YOLO in any of these verified environments with preinstalled dependencies:
StatusOur continuous integration (CI) ensures all models and integrations, including Sony IMX500, are tested across platforms. Check our CI status here for the latest updates:
|
Beta Was this translation helpful? Give feedback.
-
|
I ran into some python dependency issues when trying to run the 1st example. i started with python 3.12. ultralytics asks for tensorflow 2.12 which seems to clash with the requirements of the converter tools. Downgrading python to 3.11 is not enough. Tensorflow version must be higher than 2.12. I tested on ubuntu 22 machine. Found solution after some testing. Here is the project file. [project]
name = "ultralytics-imx500"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = "~=3.11.0"
dependencies = [
"imx500-converter[pt]==3.14.3",
"model-compression-toolkit==2.1.1",
"sony-custom-layers==0.2.0",
"tensorflow~=2.14.0",
"ultralytics>=8.3.31",
]It could also be good to mention that converter installation at some point might ask for sudo pwd since it requires java. |
Beta Was this translation helpful? Give feedback.
-
|
@alexander-sony thank you for sharing your solution and insights on resolving the dependency issues. It's great to see the community collaborating to overcome challenges. If you have further questions or need additional assistance, feel free to ask here. |
Beta Was this translation helpful? Give feedback.
-
|
Hello Alexander-Sony, root@5c41d2cb95d6:/datasets/dataset# yolo export model=best.pt format=imx PyTorch: starting from 'best.pt' with input shape (1, 3, 416, 416) BCHW and output shape(s) ((1, 3549, 4), (1, 3549, 8)) (5.9 MB) During handling of the above exception, another exception occurred: Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I found it myself. Installing openjdk-17-jdk before starting the export prevents the problem. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for sharing your solution, Jochen! Pre-installing |
Beta Was this translation helpful? Give feedback.
-
|
Hi there it doesnt support for yolov8n-obb model because it shows me |
Beta Was this translation helpful? Give feedback.
-
|
@ARCHANGELOVE it seems you're encountering an issue with the |
Beta Was this translation helpful? Give feedback.
-
|
Yep noted but as i know the box decoding for OBB is differ from AABB because the model convertion using yolov8n.pt works fine and the conversion was successful yolo export model=yolov8n-obb.pt format=imx i face the error when i convert the obb model my ultralytics vertion is in latest aswell |
Beta Was this translation helpful? Give feedback.
-
|
It seems like the |
Beta Was this translation helpful? Give feedback.
-
|
Hello there, Are there any tips on IMX exporting for later YOLO models such as V11? |
Beta Was this translation helpful? Give feedback.
-
|
@marcivanmanalac to export later YOLO models like YOLOv11 to IMX format, you can follow a similar process as mentioned for YOLOv8, using the |
Beta Was this translation helpful? Give feedback.
-
|
Hi Glenn, Yes, that was what I had attempted in the following format but I get a |
Beta Was this translation helpful? Give feedback.
-
|
@marcivanmanalac the error occurs because IMX export currently supports only YOLOv8 detection models and is not implemented for YOLOv11. Please refer to the Sony IMX500 Export guide for supported workflows. YOLOv11-specific export capabilities may require future updates. |
Beta Was this translation helpful? Give feedback.
-
|
There is a problem with the instructions to install picamera2, it seems to create issues for the imx500 examples in https://github.com/raspberrypi/picamera2/tree/main/examples/imx500. Has it been tested with existing imx500 examples in the picamera2 repo? Proper picamera2 installation is with apt, see https://github.com/raspberrypi/picamera2 |
Beta Was this translation helpful? Give feedback.
-
|
@alexander-sony thank you for highlighting this. The installation via |
Beta Was this translation helpful? Give feedback.
-
|
problem with ultralytics version >8.3.34, the instruction: |
Beta Was this translation helpful? Give feedback.
-
|
@alexander-sony the error indicates a mismatch between the input dimensions expected by the model and the data provided. Ensure the input image matches the expected size (default: 640x640) by resizing or specifying |
Beta Was this translation helpful? Give feedback.
-
|
Can we do face recognition(not just detection) with this approach? |
Beta Was this translation helpful? Give feedback.
-
|
@HelloCigar face recognition is out of scope for YOLO models, which are designed for object detection, classification, segmentation, and pose estimation. For facial recognition tasks, consider exploring specialized libraries like FaceNet or Mediapipe. Let me know if you have any YOLO-specific questions! |
Beta Was this translation helpful? Give feedback.
-
|
https://huggingface.co/arnabdhar/YOLOv8-Face-Detection |
Beta Was this translation helpful? Give feedback.
-
|
@HelloCigar using YOLOv8 for face classification (labeling faces with names) can work if your dataset is well-annotated. However, this approach is not true "face recognition" as it lacks the specialized feature extraction and matching capabilities of dedicated face recognition models like FaceNet. The main disadvantage is that YOLOv8 may struggle with nuanced identity recognition compared to models tailored for such tasks. Let me know if you need further clarification! |
Beta Was this translation helpful? Give feedback.
-
|
Great guide and I managed to get my first model trained and converted to run on the imx500 on a raspberry pi4 for some custom object detection. Initially I trained on the yolov8s but when I tried to export it it did not work. I then trained the yolov8n with the same data and it worked fine (doubled checked and the yolov8s defualt does not export either) Is there a way to get the yolov8s models to export so I can run it on the imx500 with the tools or does it only work for the yolov8n? I am new to the Yolov8 models and the imx500 so was just wondering (the s version performed much better than the n version on my dataset) as once I get the yolov8n packaged up on the raspberry PI it seems to be only about 2MB is size so with the input tensors would think it could handle a bigger model as I think the imx500 has 8MB available for the model and input tensors. |
Beta Was this translation helpful? Give feedback.
-
|
@dave-ct thank you for your detailed feedback! Currently, the IMX500 export supports YOLOv8 detection models like YOLOv8n but does not extend to YOLOv8s. This limitation is related to the model's architecture and compatibility with the IMX500 conversion tools. While the IMX500 hardware has sufficient memory, its constraints on model complexity and processing may require further optimizations or updates to support larger models like YOLOv8s in the future. You can stay updated via our documentation or explore adjustments to your dataset to improve YOLOv8n performance. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @dave-ct , I was able to convert the yolov8n model into a suitable format for the IMX500 and run it successfully as well. However, I couldn’t convert any models other than yolov8n. Have you tried converting any models other than yolov8n, and if so, were you successful? |
Beta Was this translation helpful? Give feedback.
-
|
Are there any plans to support yolov8n-pose or any other version of a yolo pose model in the foreseeable future. I have tried using the above tools with yolov8n-pose and I obtained this result: |
Beta Was this translation helpful? Give feedback.
-
|
@awajs currently, the IMX500 export supports only YOLOv8n detection models and not pose models like YOLOv8n-pose. For updates on future support, please stay tuned to the Ultralytics GitHub repository and documentation. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@awajs thank you for your question. The IMX500 export currently only supports YOLOv8n detection models as documented in our Sony IMX500 integration guide. While we don't have an official timeline for pose model support, we welcome community contributions to expand compatibility. For updates, please monitor our GitHub repository and documentation. |
Beta Was this translation helpful? Give feedback.
-
|
hello i ran succesfully the standard yolov8 model converted in imx in my pi ai cam, now i'm wondering where i can modify the parameter of detection, such as confidence, resolution ecc... i don't see anything like the config json of the models tath comes preinstalled with the pi cam dependency |
Beta Was this translation helpful? Give feedback.
-
|
The IMX500 export doesn’t generate a separate configuration file, so detection parameters must be adjusted at inference time in your deployment script or application. |
Beta Was this translation helpful? Give feedback.
-
|
ok, i see, is there some documentation about this topic, and which would be the right way to extrapolate recognition data from the real time detection? |
Beta Was this translation helpful? Give feedback.
-
|
See the Raspberry PI AI Documentation which further down the page explains how to us it for libcamera and picamera2. There are also exmamples of python code on github for python. Hope that helps you. |
Beta Was this translation helpful? Give feedback.
-
|
For adjusting detection parameters like confidence and resolution with your IMX-converted YOLOv8 model on Raspberry Pi, modify these settings directly in the inference script arguments when running See the picamera2 examples for implementation details and our IMX500 export guide for deployment parameters. |
Beta Was this translation helpful? Give feedback.
-
|
in the object detection sample, how do you crop and save the detected obj? assuming only 1 is detected |
Beta Was this translation helpful? Give feedback.
-
|
@HelloCigar to crop and save a detected object from YOLO inference results when only one detection exists, access the bounding box coordinates from from PIL import Image
results = model.predict("image.jpg")
im = Image.open("image.jpg")
box = results[0].boxes.xyxy[0].cpu().numpy() # [x1,y1,x2,y2]
cropped = im.crop(box)
cropped.save("object.jpg")For production use, add conditional checks for detections and handle multiple objects. The detection coordinates remain consistent across export formats like IMX500. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, is there a journal / paper that discusses IMX 500 for yolo ? |
Beta Was this translation helpful? Give feedback.
-
|
@AmndanDwii for specific academic publications on IMX500 with YOLO, we recommend referring to Sony's official technical resources at their AITRIOS developer portal. While Ultralytics doesn't publish academic papers, our implementation details and benchmarks are documented in the IMX500 integration guide and demonstrated in our Edge AI blog post. |
Beta Was this translation helpful? Give feedback.
-
|
Can sony imx run yolo12, does it run the same way as in yolov8? |
Beta Was this translation helpful? Give feedback.
-
|
@AmndanDwii yOLO12 IMX500 support isn't currently available, but the YOLOv8 IMX500 export guide shows our general approach for sensor-optimized deployments. We recommend following the Sony IMX500 Export Guide for YOLOv8 as reference, and stay tuned to our documentation for future YOLO12 updates. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I got a bunch of warnings during the execution (export_log.txt), but the imx_model seemed to work properly in the end. I then packaged it via IMX500 Packager and test it using the imx500_object_detection_demo.py. |
Beta Was this translation helpful? Give feedback.
-
|
The IMX500 output denormalization details are covered in the Sony IMX500 documentation under the What's Under the Hood? section. For bounding boxes, apply the raw_boxes = results[0].numpy() # raw output tensor for boxes (ordinal=0)
normalized_boxes = raw_boxes * 0.03125 # apply scale factorLet us know if this resolves the coordinate scaling issue! |
Beta Was this translation helpful? Give feedback.
-
|
I think there’s been a misunderstanding. I thought there were other changes to be made besides rescaling. Also, I cannot find the "What's Under the Hood?" section anywhere here https://developer.sony.com/imx500. Could you be more precise about where I should look? |
Beta Was this translation helpful? Give feedback.
-
|
The "What's Under the Hood?" section refers specifically to our Ultralytics IMX500 documentation, not Sony's site. For box decoding, you'll need to handle both scale factors and output tensor ordering. Here's the full post-processing snippet: # After inference
boxes = results[0].numpy() * 0.03125 # scale boxes (ordinal=0)
scores = results[1].numpy() # confidence scores (ordinal=1)
classes = results[2].numpy() # class indices (ordinal=2)
valid = results[3].numpy()[0] # num valid detections (ordinal=3)
# Slice actual detections using valid count
boxes = boxes[:valid]
scores = scores[:valid]
classes = classes[:valid]This accounts for the 4-output-tensor structure documented in our IMX500 integration guide. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @glenn-jocher , thank you very much for all the support. It finally works! There was no need to rescale the boxes by 0.03125. I was missing the --bbox-normalization flag when launching the script. Once I added the flag, everything worked fine. |
Beta Was this translation helpful? Give feedback.
-
|
Great to hear you resolved it! The |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I followed the steps in this page to convert the yolov8n model with imx500 converter and tested it with raspberry ai camera. But I get 29.5 ms for the inference time with 640x640 input tensor size. So I wonder why my inference time is better than your result (58.82ms). Which version of imx500 converter did you use? |
Beta Was this translation helpful? Give feedback.
-
|
Also, how did you measure the inference time? Did you use imx500.get_kpi_info(metadata) or calculate the elapsed time of picam2.capture_metadata()? |
Beta Was this translation helpful? Give feedback.
-
|
@skim-sony congratulations on achieving better inference times than our documented benchmarks! The performance difference could be due to several factors including using a newer IMX500 converter version, firmware optimizations, or hardware differences (Pi 5 vs Pi 4). For measurement methodology, our benchmarks typically use a combination of methods including the IMX500's native profiling via It's great to see improved performance in newer implementations - would you mind sharing which converter version and Raspberry Pi model you're using? This could help other community members optimize their deployments. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I have used Pi5 and tried both 3.14.3 and 3.16.1 for imx500-converter. And both produces 29.5ms for inference time. It seems like you also use Pi5, so I wonder why our results are different. I'd like to reproduce your results 58.82 ms to learn what improves the inference time. So it would be helpful if you could share some info how to sync with your environment. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@skim-sony that's interesting you're seeing better performance! The discrepancy could be due to several factors:
For replicating our methodology, we used:
Ultimately, faster performance is a positive outcome! Your 29.5ms results demonstrate how the platform continues to improve. If you'd like to share more details about your implementation, the community would certainly benefit. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, per picamera2 docs, creating apps with "a camera window embedded into the GUI is through Qt", is there no other way to embed the video feed in other GUI frameworks? |
Beta Was this translation helpful? Give feedback.
-
|
In the official IMX500 documentation provided by Sony (https://developer.sony.com/tinyml-23/developer-tools/documentation/imx500-evaluation-kit-user-guide?version=2023-11-28&progLang=), they describe the process of generating the network.fpk and network_info.txt files that will be used to deploy the model to IMX500, using the Packager.sh script. However, when I attempt to apply the same procedure to the packerOut.zip file I created by following the tutorial on https://docs.ultralytics.com/integrations/sony-imx500, I am unable to run the model on the IMX500. Also I couldn't run your implementation because it says that some packages are not found which I believe is related to the version of the OS. Currently I use the Pi OS that is provided by SONY for compatibility issues. I could run some other deep learning models following the steps in the (https://developer.sony.com/tinyml-23/developer-tools/documentation/imx500-evaluation-kit-user-guide?version=2023-11-28&progLang=), but I believe, there must be something different related to the creation of packerOut.zip file. Could you assist in resolving this issue with the conversion process? @glenn-jocher |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
integrations/sony-imx500/
Learn to export Ultralytics YOLOv8 models to Sony's IMX500 format to optimize your models for efficient deployment.
https://docs.ultralytics.com/integrations/sony-imx500/
Beta Was this translation helpful? Give feedback.
All reactions