zh/models/yoloe/

[giscus[bot]]

zh/models/yoloe/

YOLOE 是一种实时开放词汇检测和分割模型，它通过文本、图像或内部词汇提示对YOLO 进行了扩展，从而能够以最先进的零镜头性能检测任何对象类别。

https://docs.ultralytics.com/zh/models/yoloe/

[daiyangyang945]

I converted YOLOE to ONNX, but it only supports single-image input. How can I implement visual prompts?

[UltralyticsAssistant]

👋 Hello, thank you for your interest in Ultralytics 🚀! We recommend checking out the Docs for comprehensive guides, including Python and CLI examples, where many common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us investigate the issue efficiently.

If you have a custom training ❓ Question, please include detailed information such as dataset samples, training logs, and confirm you are following our Tips for Best Training Results.

Join the Ultralytics community in a way that suits you:

• Real-time chat: Discord 🎧
• In-depth discussions: Discourse
• Share experiences: Subreddit

Upgrade

To ensure you have the latest features and bug fixes, upgrade the ultralytics package and all requirements in a Python>=3.8 environment with PyTorch>=1.8:

pip install -U ultralytics

Environments

YOLO models can be run in any of these up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLO Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

This is an automated response 💡. An Ultralytics engineer will also review your discussion and assist you soon!

[glenn-jocher]

Hello @daiyangyang945,

When working with YOLOE exported to ONNX with visual prompts, you'll need to handle the preprocessing differently since ONNX exports typically flatten the model architecture and remove dynamic operations.

For visual prompts with ONNX models, the recommended approach is to:

1. Pre-process your visual prompts in Python before feeding them to the ONNX model
2. Extract the visual embeddings from your reference examples using the original PyTorch model
3. Feed these pre-computed embeddings along with your input image to the ONNX model

You'll need to modify your export settings to ensure the model preserves the inputs needed for visual prompts. Try using a custom export configuration that specifies the visual prompt input:

from ultralytics import YOLOE

# Load your YOLOE model
model = YOLOE("yoloe-11l-seg.pt")

# Set up dynamic inputs for the ONNX model
model.export(format="onnx", dynamic=True, imgsz=640, simplify=True)

Then when performing inference, you'll need to handle the visual prompt preprocessing separately and format the inputs correctly for your ONNX runtime.

Note that full visual prompt support in ONNX exports is still being improved. If you encounter specific issues, could you share more details about your exact implementation and the error you're seeing?

[Chappie123]

Hello, I have some datasets for bbox, like {class, minx, miny, maxx, maxy}
could you provide a detailed instruction of how to train my own datasets?

[HLHSM]

This is the code I used to test yoloe inference:

from ultralytics import YOLOE

Initialize a YOLOE model

model = YOLOE("yoloe-v8m-seg.pt") # or select yoloe-11s/m-seg.pt for different sizes

Set text prompt to detect person and bus. You only need to do this once after you load the model.

names = ["person", "bus"]
model.set_classes(names, model.get_text_pe(names))

Run detection on the given image

results = model.predict("girl.png")

Show results

results[0].show()

PS C:\Users\SYC\Desktop\projects\YOLOE_test> & D:/anaconda3/envs/RL/python.exe c:/Users/SYC/Desktop/projects/YOLOE_test/test.py
Traceback (most recent call last):
File "c:/Users/SYC/Desktop/projects/YOLOE_test/test.py", line 8, in
model.set_classes(names, model.get_text_pe(names))
File "D:\anaconda3\envs\RL\lib\site-packages\ultralytics\models\yolo\model.py", line 271, in get_text_pe
return self.model.get_text_pe(texts)
File "D:\anaconda3\envs\RL\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "D:\anaconda3\envs\RL\lib\site-packages\ultralytics\nn\tasks.py", line 1030, in get_text_pe
model = self.clip_model if cache_clip_model else build_text_model("mobileclip:blt", device=device)
File "D:\anaconda3\envs\RL\lib\site-packages\ultralytics\nn\text_model.py", line 347, in build_text_model
return MobileCLIPTS(device)
File "D:\anaconda3\envs\RL\lib\site-packages\ultralytics\nn\text_model.py", line 286, in init
self.tokenizer = clip.clip.tokenize
AttributeError: module 'clip' has no attribute 'clip'

how to solve it?

[Maya7991]

YOLOE ONNX inference doesn't accept text prompts.

Goal: Run YOLOE ONNX inference with text prompts.
Expected: ONNX inference model should accept 2 arguments: input image and text prompt.
Observed: ONNX inference model accepts only a single image input

The YOLOE model is imported to onnx using the given code:

from ultralytics import YOLOE
model = YOLOE("../yoloe-11s-seg.pt")
model.export(format="onnx", imgsz=640, dynamic=True)

I want to run ONNX inference with text prompt. I have already preprocessed my text into embeddings using CLIP as shown below.

clip_model_name = "openai/clip-vit-base-patch32"
tokenizer = CLIPTokenizer.from_pretrained(clip_model_name)
text_encoder = CLIPTextModel.from_pretrained(clip_model_name)
# Prepare text prompt
prompts = ["person", "bus"]
inputs = tokenizer(prompts, return_tensors="pt", padding=True, truncation=True)
with torch.no_grad():
    text_embeddings = text_encoder(**inputs).last_hidden_state[:, 0, :]  # shape: [num_prompts, 512]
text_embeddings_np = text_embeddings.cpu().numpy().astype("float32")

But I cannot pass it as input the onnx model during inference, because I checked the input parameters and it accepts only 1 input image

session = ort.InferenceSession("yoloe-11s-seg.onnx") 
print("ONNX model inputs:", [i.name for i in session.get_inputs()])
print("ONNX model outputs:", [o.name for o in session.get_outputs()])

This prints:

ONNX model inputs: ['images']
ONNX model outputs: ['output0', 'output1']

It shows only 1 input which is used for the input image. Where can I plugin the text embedding/prompt?