SWE-bench Score?

[ofirpress]

Hi!
I'm wondering what your SWE-bench score is? We'd love to have a submission from your team.

Thanks :)

[TomLucidor]

Seconding this but also because Roo-Code is not a full-on auto-agent it is hard to compare

[mrubens]

Hi @ofirpress, @cte and I will work on pulling together a submission. Thanks for reaching out!

[ofirpress]

Roo-Code is not a full-on auto-agent it is hard to compare

We've previously had submissions from non-fully-auto agents such as devin. I think it's doable.

[SmartManoj]

For instance, sympy__sympy-20590:

Steps

1. Run the instance image docker run -it --name a1 swebench/sweb.eval.x86_64.sympy_1776_sympy-20590 bash

2. Connect to IDE via the Dev Container extension.

3. Get the Problem statement and enter it into the task field.

from datasets import load_dataset
swebench = load_dataset('princeton-nlp/SWE-bench', split='test')
instance_id = 'sympy__sympy-20590'
filtered = swebench.filter(lambda x: x['instance_id'] == instance_id)
print(filtered[0]['problem_statement'])

Allow the agent to execute all commands.
Model: anthropic/claude-3.7-sonnet:beta
The agent didn't ask any questions. Need to add some prompts if it asks any.
Should eval after #3217 for reproducible outputs.

The agent completed the task.
task_history.md
git_diff.md

[BradKML]

@SmartManoj if the agent did not ask any questions, then it likely means either (a) the agent llm did not read the prompt on what it is supposed to do (b) the swe-bench library likely failed to feed the information to the agent, so please debug them by using "deep research" tools to assist accordingly (e.g. Perplexica)

[SmartManoj]

The task is simple. So, the agent completed it directly.

[BradKML]

Noted with thanks, do you think this can be automated into a tool that can allow for easy LiveSWEBench / SWE-Bench to add/check for their leaderboards?

[SmartManoj]

Yes.