Poor performance on sythbuster Dataset

[ydvaayan]

Hey thanks for the amazing work!

I tested the model on Sythbuster Dataset (1000 images each from dalle2, dalle3,firefly, midjourney_v5 each compared separately with RAISE_1k images) with 16 patch size. The performance for all models is around or below random chance (best avg accuracy performance being 0.639 for midjourney trained model and worst being 0.417 for adm trained model). A lot of times not loading the weight gives better performance(0.5).

I was wondering if you have shared the correct weight files because the results you have shared in paper are pretty good and the results I am getting are pretty bad. Its also surprising that midjourney trained model is not able to distinguish midjourney images(acc of just 0.504)

For simplicity i didnt use any augmentation yet during testing

[csdahunzi]

We retested the trained model on GenImage and did not find any problems. You can check the details carefully.

[junjichan]

I found that the model performed well on ai images with extremely clean colors, but some ai images could easily be mistaken for natural images