Abnormal cost running SWE bench lite #73
Description
Hi authors, thanks for the great work!
I'm currently testing GPT-4o-mini on SWE bench lite with text-embedding-3-small as the embedding model. A whole run cost me more than $500, with about $450 during retrieval phase. I'm wondering if this is the case or something went wrong.
In the paper, the average cost per question is $0.70 when using GPT-4o, which means ~$210 running the whole SWE bench lite.
It would be really helpful if anyone would like to share the chosen LLM and embedding model with the corresponding cost of one complete run!