YOLOv8 gray images train

[zzw725]

Everyone！
When training grayscale images with YOLOv8, the training set and validation set performed well, with P, R, and mAP reaching 99.5%, 100%, and 99.9%, respectively. However, the test set performed poorly, with two categories identified, and category 1 accuracy at 0%, with the model completely identifying it as category 2. What could be the reason for this, and how can this issue be resolved?

[UltralyticsAssistant]

👋 Hello @zzw725, thank you for reaching out and sharing your experience with Ultralytics 🚀! For new users and troubleshooting, we recommend checking out the Docs, where you’ll find Python and CLI usage examples, as well as answers to many frequently asked questions.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us identify and debug the issue.

If you’re looking for training advice or have custom dataset questions, please include as much detail as possible, such as dataset image examples, configuration files, and training logs, and make sure you’re following our Tips for Best Training Results.

You’re also invited to join the Ultralytics community in whatever way suits you best! For real-time chat, join us on Discord 🎧. For in-depth discussions, try Discourse. Or join our Subreddit to exchange knowledge with the community.

Upgrade

Please ensure you are using the latest ultralytics package and all requirements in a Python>=3.8 environment with PyTorch>=1.8 to see if your issue is resolved in the latest version:

pip install -U ultralytics

Environments

YOLO can be run in any of the following up-to-date verified environments (with all dependencies, including CUDA/CUDNN, Python, and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLO Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

This is an automated response to help guide your troubleshooting. An Ultralytics engineer will also review your discussion and assist you soon!

[glenn-jocher]

This looks like a classic overfitting issue where your model memorized the training data but fails to generalize to new test data. The perfect training/validation performance (99.9% mAP) combined with 0% accuracy on one test class is a strong indicator.

Here are the most likely causes and solutions:

Data Distribution Shift: Your test set likely has different characteristics (lighting, contrast, noise levels) compared to your train/val sets. Ensure all datasets undergo identical preprocessing and augmentation.

Insufficient Regularization: Add stronger data augmentation during training - increase mixup, mosaic, copy_paste parameters, and reduce lr0 to prevent overfitting.

Train/Val/Test Split Issue: Verify your test set truly represents unseen data and wasn't inadvertently similar to your validation set during the split.

For grayscale training specifically, ensure you're using channels: 1 in your dataset YAML configuration as mentioned in the COCO8-Grayscale documentation.

Consider reducing model complexity (try YOLOv8n instead of larger variants) and implement early stopping based on validation metrics to prevent overfitting.

[zzw725]

Thank you for your reply! When I trained using the same dataset with three RGB channels, the validation set and test set performed well. What are the reasons for overfitting in grayscale images? It is strange that this problem does not occur with colour images.

[glenn-jocher]

The difference between RGB and grayscale performance stems from information loss during the channel reduction. When converting to grayscale, you lose color information that may be critical for distinguishing between your two categories - the model now has 3x less input data but the same network capacity.

This creates an information bottleneck where the model tries to memorize training patterns more aggressively since it has fewer discriminative features available. RGB images provide color cues that help generalization, while grayscale forces the model to rely solely on intensity and texture patterns.

Try reducing your model size (use YOLOv8n instead of larger variants) to better match the reduced information content, or increase augmentation diversity with stronger hsv_h, hsv_s, hsv_v parameters to create more varied intensity patterns during training. You might also consider if color information is actually essential for your specific classification task.

[zzw725]

OK! Thank you for your guidance.
The current model is YOLOv8n. Next, I will try to increase the amount of data and improve the model to make it more suitable for grayscale images.

[zzw725]

OK! Thank you for your guidance.
The current model is YOLOv8n. Next, I will try to increase the amount of data and improve the model to make it more suitable for grayscale images.

[zzw725]

I recall that I changed the presentation format of the dataset images.

1. Each image contained 3-4 detection targets;
2. Each image contained only one detection target.
   When I input the model using method 1, the grayscale training model performed well, with good results for the training set, test set, and validation set. However, when I used method 2 for grayscale training, the above overfitting phenomenon occurred.

[glenn-jocher]

That's a crucial insight! The data structure change from multiple targets per image to single targets per image is likely the root cause of your overfitting issue, especially in grayscale.

When each image contains only one detection target, your model can more easily memorize class-specific spatial patterns and backgrounds, particularly with reduced grayscale information. Multiple targets per image force the model to learn more generalizable features since it must handle varied contexts within the same image.

The reason this didn't affect RGB training as severely is because color provides additional discriminative information that helps prevent this memorization. With grayscale's reduced information content, the model becomes more susceptible to overfitting when the data structure is simplified.

To address this with single-target images, increase your mosaic augmentation probability and mixup parameters to artificially recreate the multi-target diversity that worked well in your first approach. This will help the model see varied combinations during training even with single-target source images.

[zzw725]

Thanks for your reply!
The above are the training results.

1. The box loss of train and val first drops to the lowest point, then slowly rises. How can this be avoided?
2. After turning off mosaic enhancement, the loss value and accuracy will both drop significantly.

[glenn-jocher]

Excellent observations! Both patterns confirm the overfitting diagnosis:

1. Rising validation loss after initial drop: This is classic overfitting behavior. Implement early stopping by setting patience=10 in your training arguments to automatically stop when validation metrics stop improving. You can also reduce lr0 from the default 0.01 to something like 0.005.

2. Performance drop without mosaic: This confirms mosaic augmentation is crucial for your single-target grayscale setup. The significant drop when disabling mosaic shows it's artificially creating the multi-target diversity your model needs. Keep mosaic enabled (default mosaic=1.0) and consider increasing mixup=0.15 and copy_paste=0.3 for additional regularization.

The rising box loss pattern in your results plot is the model starting to memorize rather than generalize - early stopping will prevent this and likely give you better test performance.