Improve the speed on a mobile device of a YOLOv8 OD model

[zfrankowski]

Good afternoon everyone!

I have a YOLOv8 model that I have trained to detect street signs and am using it on a Samsung Galaxy S24 Ultra. It only detects if I'm driving really slow and even then it takes about 1.5s to detect a sign. Is there anyone way of speeding this up? I don't really want to drop the image size as it's an object detection model and that might lose detection accuracy. I exported it to tflite and am graciously using (instead of reinventing the wheel) https://github.com/surendramaran/YOLO Android Studio project.

I'm newish to YOLO and would appreciate any comments or feedback!!

Thanks - Zack

[UltralyticsAssistant]

👋 Hello @zfrankowski, thanks for starting the discussion and for using Ultralytics 🚀 This is an automated response to help you get quick guidance while an Ultralytics engineer reviews your post and assists you soon.

We recommend a visit to the Docs for general usage tips and examples:

• Python: https://docs.ultralytics.com/usage/python/
• CLI: https://docs.ultralytics.com/usage/cli/
• Main Docs: https://docs.ultralytics.com/

If this is a 🐛 Bug Report, please provide a minimum reproducible example (MRE) to help us debug it: https://docs.ultralytics.com/help/minimum-reproducible-example/

If this is a custom training ❓ Question, please share as much detail as possible, including:

• Dataset samples (a few images)
• Training command and logs
• Export details (format, args)
• Inference environment and logs
  And verify you’re following our Tips for Best Training Results: https://docs.ultralytics.com/guides/model-training-tips/

Join the Ultralytics community where it suits you best:

• Discord for real-time chat 🎧: https://discord.com/invite/ultralytics
• Discourse for in-depth threads: https://community.ultralytics.com/
• Subreddit for community tips: https://www.reddit.com/r/Ultralytics/

Upgrade

Please upgrade to the latest ultralytics package with all requirements in a Python>=3.8 environment with PyTorch>=1.8 to ensure you’re on the newest version:

pip install -U ultralytics

Requirements: https://github.com/ultralytics/ultralytics/blob/main/pyproject.toml

Environments

YOLO may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide: https://docs.ultralytics.com/yolov5/environments/google_cloud_quickstart_tutorial/
• Amazon Deep Learning AMI. See AWS Quickstart Guide: https://docs.ultralytics.com/yolov5/environments/aws_quickstart_tutorial/
• Docker Image. See Docker Quickstart Guide: https://docs.ultralytics.com/yolov5/environments/docker_image_quickstart_tutorial/

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLO Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

No worries—on Android you select a TFLite delegate via Interpreter.Options (GpuDelegate or NnApiDelegate) and set threads; the TensorFlow Lite GPU delegate quickstart and the Android NNAPI overview show how, and you can quickly test performance without coding using the Ultralytics Android App guide.

[zfrankowski]

Great, thanks Glenn! I will definitely check it out!

[glenn-jocher]

Perfect—when you test, A/B GPU vs NNAPI (Hexagon) and FP16 vs INT8 using the Ultralytics Android App guide and export via the TFLite export guide; share results and we’ll help tune.

[zfrankowski]

Sounds great, I really appreciate the help! Hopefully I can get it going - my Android Studio skills are rusty but I'll try my best!!

[zfrankowski]

I got the nnapi delegate added however it says the GPU delegate on my cell phone isn't available. I even asked Gemini for code verification it didn't seem to find anything wrong. I'm going to go try it in the real world now. I'm not sure how to actually benchmark test it though - I'm sorry I sound stupid, still new to this. :(

** UPDATE: I noticed that the inference time went down from 1.5s running on CPU to 0.5s running on the NNANPI delegate. it's a start :) I find that if I drive at 30km/h it will detect signage but any faster it doesn't.

[Y-T-G]

What's the size of the model?

Ultralytics Flutter App uses GPUDelegate, and you can easily get under 30ms inference with that for nano models.

https://github.com/ultralytics/yolo-flutter-app

You can test the app from Play Store by downloading Ultralytics HUB.

[zfrankowski]

Hi Y-T-G - The current model is 22Mb. I'm training a slightly larger model, but will definitely try it in Flutter and in the Ultralytics Hub. I will post my results soon. I'm going to train tomorrow on the new model.

Thanks - Zack

[glenn-jocher]

22 MB is mid-size; for real-time on mobile go smaller, not larger—train a YOLO11n/s, export to TFLite with FP16 or INT8 and NMS, then A/B NNAPI vs GPU using the Ultralytics Android App. To gain speed without losing much accuracy, limit classes/max_det to your signs and crop the camera ROI instead of increasing imgsz. See the TFLite export guide and the Ultralytics Android App docs for the exact steps.