integrations/ray-tune/

[giscus[bot]]

integrations/ray-tune/

Optimize YOLOv8 model performance with Ray Tune. Learn efficient hyperparameter tuning using advanced search strategies, parallelism, and early stopping.

https://docs.ultralytics.com/integrations/ray-tune/

[sachmatkris]

Hi, I set up the parameter tuning proccess as explained but I am getting the following exception:

2024-09-26 17:55:38,910 ERROR tune_controller.py:1331 -- Trial task failed for trial _tune_617b3_00000
Traceback (most recent call last):
File "C:\Users\Kristijonas\anaconda3\lib\site-packages\ray\air\execution_internal\event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "C:\Users\Kristijonas\anaconda3\lib\site-packages\ray_private\auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "C:\Users\Kristijonas\anaconda3\lib\site-packages\ray_private\client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "C:\Users\Kristijonas\anaconda3\lib\site-packages\ray_private\worker.py", line 2691, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "C:\Users\Kristijonas\anaconda3\lib\site-packages\ray_private\worker.py", line 871, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(SyntaxError):

How do I fix this? Please dont tell me that it might be CUDA/Anaconda related...

[glenn-jocher]

It seems like there might be a syntax error in your code. Please double-check your script for any typos or incorrect syntax. If the issue persists, consider reviewing the Ray Tune documentation for additional troubleshooting tips.

[UltralyticsAssistant]

👋 Hello, thank you for your interest in Ultralytics and exploring the integration with Ray Tune 🚀! We recommend checking out the Docs to explore more about this integration and other features. The Ray Tune Integration Guide provides comprehensive steps for optimizing your YOLOv8 model performance with advanced hyperparameter tuning techniques.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us identify and address any issues.

An Ultralytics engineer will also assist you soon. In the meantime, feel free to participate in our community for real-time support. Join our Discord 🎧 for live discussions, or take part in the community on Discourse and Subreddit.

Upgrade

Ensure you have the latest ultralytics package along with all requirements. Use a Python>=3.8 environment with PyTorch>=1.8 to confirm you have resolved the issue with the latest package:

pip install -U ultralytics

Environments

You can work in any of the following environments where all necessary dependencies are preloaded:

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If the badge is green, all Ultralytics CI tests are passing, ensuring the smooth operation of YOLOv8 across various systems.

[sachmatkris]

I found out the issue was setting device = 0 in the model.tune() and specifying absolute path for the .data.yaml, which is not the same what YOLO does when started from within an opened folder on VScode.

[glenn-jocher]

@sachmatkris it seems like you've identified the issue with the device setting and path specification. If you need further assistance, feel free to ask!

[sachmatkris]

@sachmatkris it seems like you've identified the issue with the device setting and path specification. If you need further assistance, feel free to ask!

Another issue I have is using the following call all of my experiments with ray_tune return the same metrics:

search_space = { 'lr0': tune.loguniform(1e-5, 1e-1), 'momentum': tune.uniform(0.6, 0.95), }

result_grid = model.tune(data = 'C:/Users/Kristijonas/Desktop/Mokslo šaknys karčios/AI/Projects/masinu_numeriai/formated_dataset_new/data.yaml', imgsz = 1280, space = search_space, batch = 16, epochs = 50, patience = 5, use_ray = True, iterations = 20, device='cuda' )

e.g.

[glenn-jocher]

@sachmatkris it seems like the issue might be with the search space configuration. Ensure that the hyperparameters have enough variability and that the dataset path is correctly set. If the problem persists, consider adjusting the search space or reviewing the dataset configuration. For more details, you can refer to the Ray Tune documentation.

[sachmatkris]

unfortunately that is not the issue because the results are exactly the same for different learning rates - impossible.

…

On Fri, 27 Sept 2024 at 19:06, Glenn Jocher ***@***.***> wrote: @sachmatkris <https://github.com/sachmatkris> it seems like the issue might be with the search space configuration. Ensure that the hyperparameters have enough variability and that the dataset path is correctly set. If the problem persists, consider adjusting the search space or reviewing the dataset configuration. For more details, you can refer to the Ray Tune documentation <https://docs.ultralytics.com/integrations/ray-tune/>. — Reply to this email directly, view it on GitHub <#16505 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3WH6HFTKYE2WLTJVGR6HWTZYV7A5AVCNFSM6AAAAABO5HD4QCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANZXG43TCNQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[glenn-jocher]

@sachmatkris if you're seeing identical results for different learning rates, it might be worth checking the hyperparameter search space and ensuring the dataset configuration is correct. Adjusting these settings could help resolve the issue. For further guidance, please refer to the Ray Tune documentation.

[sirtris]

Hi,
you mention that the .tune() function integrates well with wandb.
Are there any plans to make it work with mlflow as well?
When I try to use it now no model metric, parameters or artifacts, etc. are logged.
It would also be nice to have one hyperparameter sweep as a parent run and the individual runs as child runs of that parent run.
Best,

[glenn-jocher]

@sirtris currently, there are no plans to integrate the .tune() function with MLflow, but we appreciate your feedback and will consider it for future updates.

[sirtris]

Could you point me to some resources so I could maybe look into it doing it myself?

[glenn-jocher]

For resources on integrating MLflow with Ray Tune, I recommend checking the Ray Tune documentation for any updates or community contributions. This might provide insights into logging metrics and managing runs with MLflow.

[alifazeli00]

Why is there no mention of beta 2(Adamw) in ultralithic documents?

[glenn-jocher]

@alifazeli00 the documentation currently focuses on the default hyperparameters and their tuning ranges. If you're looking for specific details on beta2 for AdamW, it might not be explicitly covered. For more information on hyperparameter tuning, you can explore the Ray Tune documentation linked in the guide.

[alifazeli00]

thank you, It has been mentioned that momentum is essentially beta1. Is this the same for beta2?

[glenn-jocher]

Beta2 is not explicitly mentioned in the documentation as it typically defaults to 0.999 in AdamW, which is a common standard setting. For specific adjustments, you may need to modify the optimizer settings directly in your training script.

[NegroAmigo]

Which versions of YOLO are supported by Ray Tune? I have project with YOLOv8, will it be suitable to use with Ray Tune?

[glenn-jocher]

@NegroAmigo yes, YOLOv8 fully supports Ray Tune integration for hyperparameter tuning. You can use the same model.tune() method shown in our Ray Tune integration guide by simply loading your YOLOv8 model (YOLO('yolov8n.pt')). The tuning process works identically across supported YOLO versions. For implementation details, see the Ultralytics Ray Tune documentation.

[wisrovi]

como lo combino con MLFlow?

[wisrovi]

tambien necesito:

model.add_callback("on_train_start", self.on_train_start)
model.add_callback("on_train_end", self.on_train_end)

pero dichas funciones no se disparan

[wisrovi]

agradezco mucho sus valiosas aportaciones, de verdad que es una funcionalidad increible y magnifica

[glenn-jocher]

@wisrovi to integrate MLFlow with Ultralytics YOLO hyperparameter tuning, you'll need to create a custom Ray Tune reporter that logs metrics to MLFlow. While we don't have native MLFlow integration like Weights & Biases, you can implement it by adding MLFlow logging callbacks to your tuning script. The Ray Tune documentation provides guidance on MLFlow integration that you can adapt for YOLO workflows.

[Fatima-khanjar]

Hello!
I hope this message finds you well.

I'm trying to use Ray Tune for hyperparameter optimization with the following code:

results = model.tune(
    data="data.yaml",  
    epochs=30,                 
    iterations=150,           
    batch=16,            
    optimizer="AdamW",         
    plots=False,               
    save=False,                
    val=False,                 
    device=0,
    workers=8,
    use_ray=True,
    project="ray_results"
)

However, I'm getting a path-related error that I can't figure out:

FileNotFoundError: [WinError 3] The system cannot find the path specified...

It seems to be related to the length of the path Ray is trying to create. Do you know how I can fix this?

Also, I have a question. I came across Ray Tune while working with Genetic Algorithms, and I was wondering: would you recommend using Ray Tune or Genetic Algorithms for hyperparameter optimization? Are they similar in how they work, or are there major differences between them? What advantages does Ray Tune offer over Genetic Algorithms?

Thank you for your help!

[Fatima-khanjar]

Hello Community!!

Just following up on my own post — I managed to solve the issue on Windows 11.

Here’s what worked for me:

1. Opened gpedit.msc as Administrator
2. Navigated to: Computer Configuration > Administrative Templates > System > Filesystem
3. Enabled the setting “Enable Win32 long paths"

After doing this, the problem was resolved.

Posting this in case it helps someone else!

[glenn-jocher]

@Fatima-khanjar Thank you for sharing your solution to the Windows path length issue! This is helpful information that could benefit other users experiencing similar problems.

Regarding your question about Ray Tune versus Genetic Algorithms: They're different approaches to hyperparameter optimization. Ray Tune offers several advantages including:

1. Multiple search strategies beyond genetic algorithms (Bayesian optimization, grid search, random search)
2. Built-in parallel execution that efficiently utilizes available resources
3. Early stopping via ASHA scheduler to terminate underperforming trials
4. Distributed computing capabilities for scaling across multiple machines

Ultralytics actually supports both approaches - the internal tuner uses genetic algorithms focusing on mutation, while the Ray Tune integration provides more advanced search strategies and better parallelization. Ray Tune is generally recommended for more complex search spaces and when you have multiple GPUs available.

[Fatima-khanjar]

Subject: Questions Regarding Using Ray Tune for YOLOv11 Hyperparameter Tuning

Hello!

I hope this message finds you well.

#hyperparameter tuning using tune()
from ray import tune
from ray.tune import ResultGrid
from ray.tune.examples.mnist_pytorch import train_mnist
from ultralytics import YOLO
import os
import torch
import matplotlib.pyplot as plt

# Define a YOLO model
model = YOLO("yolo11s.pt").to('cuda')

tune_dir = 'ray_results'
os.makedirs(tune_dir, exist_ok=True)


# Tune hyperparameters using ray tune
result_grid = model.tune(
    data= "data.yaml",  
    epochs=10,                 
    iterations=10,             
    batch = 16,            
    device = 0,
    workers=0,
    use_ray=True,
    gpu_per_trial =1,
    name = "tuning",
    project = tune_dir
)

Now, load the results after the tuning experiment is finished
exp_name = "tuning"
experiment_path = f"{tune_dir}/{exp_name}"  # Get the name of the experiment
print(f"Loading results from {experiment_path}...")

restored_tuner = tune.Tuner.restore(experiment_path, trainable=train_mnist)
result_grid = restored_tuner.get_results()

# Basic Experiment-Level Analysis: Check for errors
if result_grid.errors:
    print("One or more trials failed!")
else:
    print("No errors!")

# Basic Trial-Level Analysis: Print configurations and last reported metrics for each trial
for i, result in enumerate(result_grid):
    print(f"Trial #{i}: Configuration: {result.config}, Last Reported Metrics: {result.metrics}")

# Plotting the Entire History of Reported Metrics for a Trial
# Assuming "mean_accuracy" is part of the metrics
for i, result in enumerate(result_grid):
    plt.plot(
        result.metrics_dataframe["training_iteration"],
        result.metrics_dataframe["mean_accuracy"],  # Modify this depending on your metrics
        label=f"Trial {i}",
    )

plt.xlabel("Training Iterations")
plt.ylabel("Mean Accuracy")
plt.legend()
plt.show()

---

#hyperparameter tuning using tuner.fit()
import os 
from ray import tune
from ray.tune import ResultGrid
from ray.tune.examples.mnist_pytorch import train_mnist
from ray.tune import Result
from pathlib import Path

tune_dir = Path("ray_results").resolve().as_uri()
exp_name = "tune_analyzing_results"

tuner = tune.Tuner(
    trainable=train_mnist,
    param_space={
        "lr": tune.uniform(0.001, 0.1),
        "momentum": tune.uniform(0.1, 0.9),
    },
    run_config=tune.RunConfig(
        name=exp_name,
        stop={"training_iteration": 100},
        checkpoint_config=tune.CheckpointConfig(
            checkpoint_score_attribute="mean_accuracy",
            num_to_keep = 5,
        ),
        storage_path=tune_dir,
    ),
    tune_config=tune.TuneConfig(mode="max", metric="mean_accuracy", num_samples=3),
)
result_grid: ResultGrid = tuner.fit()


experiment_path = os.path.join(tune_dir, exp_name)
print(f"Loading results from {experiment_path}...")

restored_tuner = tune.Tuner.restore(experiment_path, trainable=train_mnist)
result_grid = restored_tuner.get_results()

# Check if there have been errors
if result_grid.errors:
    print("One of the trials failed!")
else:
    print("No errors!")

num_results = len(result_grid)
print("Number of results:", num_results)

# Iterate over results
for i, result in enumerate(result_grid):
    if result.error:
        print(f"Trial #{i} had an error:", result.error)
        continue

    print(
        f"Trial #{i} finished successfully with a mean accuracy metric of:",
        result.metrics["mean_accuracy"]
    )

results_df = result_grid.get_dataframe()
results_df[["training_iteration", "mean_accuracy"]]

print("Shortest training time:", results_df["time_total_s"].min())
print("Longest training time:", results_df["time_total_s"].max())

best_result_df = result_grid.get_dataframe(
    filter_metric="mean_accuracy", filter_mode="max"
)
best_result_df[["training_iteration", "mean_accuracy"]]


# Get the result with the maximum test set `mean_accuracy`
best_result: Result = result_grid.get_best_result()

# Get the result with the minimum `mean_accuracy`
worst_performing_result: Result = result_grid.get_best_result(
    metric="mean_accuracy", mode="min"
)

best_result.config
best_result.path
# Get the last Checkpoint associated with the best-performing trial
best_result.checkpoint
# Get the last reported set of metrics
best_result.metrics
result_df = best_result.metrics_dataframe
result_df[["training_iteration", "mean_accuracy", "time_total_s"]]
best_result.metrics_dataframe.plot("training_iteration", "mean_accuracy")
ax = None
for result in result_grid:
    label = f"lr={result.config['lr']:.3f}, momentum={result.config['momentum']}"
    if ax is None:
        ax = result.metrics_dataframe.plot("training_iteration", "mean_accuracy", label=label)
    else:
        result.metrics_dataframe.plot("training_iteration", "mean_accuracy", ax=ax, label=label)
ax.set_title("Mean Accuracy vs. Training Iteration for All Trials")
ax.set_ylabel("Mean Test Accuracy")

---

I have been following your guide on using Ray Tune for hyperparameter tuning with YOLOv11. I was able to run the experiment using the use_ray=True argument, and it worked successfully—thank you for the helpful explanation!

However, I still have a few questions and points of confusion:

1. Naming the Trials:
   How can I change or customize the names of the individual trials when using model.tune(..., use_ray=True)? Right now, the trials are auto-named, and I’d like more meaningful names to help keep track of different configurations.
   

2. Retrieving the Best Hyperparameters:
   After the tuning process is complete, I would like to extract the best hyperparameter configuration to reuse it for future training. I noticed in other examples (e.g., train_mnist) that you use tuner.fit() to retrieve and analyze results, including the best configuration.
   So my question is: when using model.tune() directly (as in the case of YOLOv11), is there a method to extract the best hyperparameters without using tuner.fit()? Or is it necessary to re-structure the experiment using tune.Tuner to get access to that kind of detailed analysis?

3. Using tune.Tuner() with YOLOv11:
   In your other examples, the tuning process is managed using tune.Tuner(...), but I’m not sure how to apply this approach to YOLOv11, since there’s no explicit trainable function like train_mnist. Should we always use model.tune() for YOLO models, or is there a way to integrate YOLOv11 into a Tuner workflow for more advanced customization and analysis?

Thank you so much for your time and support! I would really appreciate any clarification or guidance on these points.

Best regards,

[glenn-jocher]

@Fatima-khanjar Thank you for your detailed questions about Ray Tune integration with YOLO11!

1. Naming Trials: Currently, the high-level model.tune() API doesn't provide direct control over individual trial naming. Ray Tune automatically generates these names. You can use the name parameter to name the overall experiment (as you've done with "tuning"), but individual trial names remain auto-generated.

2. Retrieving Best Hyperparameters: When using model.tune() with Ray Tune, the best hyperparameters are automatically saved to runs/[task]/tune/best_hyperparameters.yaml. This file contains the optimal configuration and can be loaded for future use. You can also examine tune_results.csv for all trial results.

3. Using tune.Tuner(): While possible, creating a custom trainable function for YOLO with the low-level Ray Tune API requires more work. You'd need to wrap the YOLO training process in a function that configures and trains the model while reporting metrics to Ray Tune. For most users, the high-level model.tune() API is recommended as it handles this integration automatically.

For more advanced customization, you might consider extending the Ray Tune integration in Ultralytics by referring to the source code in the ultralytics/utils/tuner.py file.

[Fatima-khanjar]

Hello!
Thank you for your clarification. I will continue using model.tune() and use the following line to retrieve the best hyperparameters, since the best_hyperparameters.yaml file wasn’t automatically saved in the output directory of the Ray tuning:
best_config = analysis.get_best_config(metric="metrics/mAP50(B)", mode="max")`

I have a question: I read that Optuna uses the TPE (Tree-structured Parzen Estimator) search algorithm, which is more powerful than the default search algorithms in Ray Tune. Since Ray Tune provides scalability and distributed tuning, is it possible to combine both—using Optuna’s TPE algorithm with Ray Tune while still calling model.tune()? If so, how can I integrate them?

I also have another question that’s unrelated. I noticed that YOLOv11 comes in different versions, from n to xl. I understand that the choice depends on the dataset size and the task complexity. But what are the actual criteria used to determine the complexity of a task?

best regrads,

[glenn-jocher]

@Fatima-khanjar

Regarding using TPE with Ray Tune - yes, it's possible! Ray Tune already supports TPE through the HyperOpt integration. You can use it by installing the HyperOpt package and specifying the search algorithm:

pip install hyperopt

from ray.tune.search.hyperopt import HyperOptSearch
from ray.tune.search import ConcurrencyLimiter

hyperopt_search = HyperOptSearch(metric="metrics/mAP50(B)", mode="max")
limited_search = ConcurrencyLimiter(hyperopt_search, max_concurrent=4)

# Then in your model.tune() call
model.tune(
    data="data.yaml",
    search_alg=limited_search,
    use_ray=True,
    # other parameters
)

For your question about YOLO11 model size selection, the criteria typically include:

• Computational resources: Larger models (l, x, xl) require more GPU memory and processing power
• Dataset complexity: More complex datasets with small/numerous objects benefit from larger models
• Inference speed requirements: Smaller models (n, s) offer faster inference for real-time applications
• Accuracy needs: Larger models generally provide higher accuracy at the cost of speed

You can check the performance trade-offs in the Ultralytics documentation to make an informed decision based on your specific constraints.

[Fatima-khanjar]

Hello,
I hope this message finds you well.

I followed your instructions to integrate Ray Tune with the TPE algorithm using HyperOpt:

hyperopt_search = HyperOptSearch(metric="metrics/mAP50(B)", mode="max")
limited_search = ConcurrencyLimiter(hyperopt_search, max_concurrent=1)

Then, I used the following code to tune hyperparameters with Ray Tune:

tuning_results = model.tune(
    data="data.yaml",
    epochs=50,                # Number of epochs for each tuning iteration
    iterations=100,           # Total number of tuning iterations
    batch=16,
    device=0,
    workers=0,
    grace_period=30,
    search_alg=limited_search,
    use_ray=True,
    gpu_per_trial=1,
    project=tune_dir
)

However, I encountered a syntax error:
SyntaxError: 'search_alg' is not a valid YOLO argument.
Could you please advise me on how to fix this?

I have another question regarding the output directory of Ray Tune. It shows that inside the runs folder, there are subfolders named train (equal to the number of iterations) and a tune folder that contains subdirectories also equal to the number of iterations.

My question is: should I focus on the train directories? Why are they generated, and what's their purpose?

Also, if I want to monitor the overall progress of the fine-tuning process — for example, in genetic tuning, we usually look at a graph of the fitness score vs. iteration — where should I look for that kind of visualization or information?

best regards,

[glenn-jocher]

@Fatima-khanjar

Regarding your issues with HyperOptSearch and Ray Tune integration:

1. The search_alg parameter is not directly exposed in the high-level model.tune() API when use_ray=True. Currently, the integration uses Ray Tune's default search algorithm. To use TPE via HyperOpt, you would need to modify the source code in ultralytics/utils/tuner.py or create a custom implementation.

2. About the directory structure:

   • The train directories (train1, train2, etc.) contain individual training runs for each trial with different hyperparameter sets
   • The tune directory contains the aggregated results of all trials and should be your primary focus

3. For monitoring progress:

   • Check tune/tune_results.csv for all trial results
   • Look for tune/best_fitness.png which shows fitness score progression over iterations
   • tune/best_hyperparameters.yaml contains the optimal configuration found

The plots in the tune directory (especially tune_scatter_plots.png if available) provide valuable visualizations of the hyperparameter search space and how different parameters affected model performance.

If you need more advanced search algorithms, you might want to explore creating a custom Ray Tune implementation that better exposes the underlying search algorithm options.

[Emeyer87]

Hello,

I've noticed that every Iteration with use_ray = True is downloading yolo11n.pt. I'm attempting to use this on an existing instance segmentation model I built but wanted to ensure the tuning is running against the model I've assigned, and not a fresh Detection model.

from ultralytics import YOLO
from ray import tune

model_path = r"C:/Users/Me/Downloads/NashvilleSegmentation.pt"
data_path = r"C:/Users/Me/Downloads/dataset/data.yaml"
model=YOLO(model_path)

search_space = {
"degrees": tune.uniform(0.0, 12.0)
}

results = model.tune(
data=data_path,
epochs=20, # keep it tiny
iterations=20, # exactly one trial
optimizer="AdamW",
space=search_space,
device=0,
seed=0,
workers=2,
amp=True,
batch=16,
patience=12,
plots=False,
save=False,
project="test_tune",
name="degrees_tune_01",
lr0=0.00114,
use_ray=True,
gpu_per_trial=1

)