[Feature]: Aider-inspired RepoMap

[neubig]

What problem or use case are you trying to solve?

Aider has a functionality to create a RepoMap, which is a concise description of the repo in text format, with the most relevant/important parts highlighted.

Describe the UX of the solution you'd like

It would be nice to have a RepoMap class within OpenDevin that can be used by any agent to pull in a description of the repo.

Do you have thoughts on the technical implementation?

• copy the aider repomap.py into OpenDevin. This could probably be put in either a new indexing folder here, or in the memory folder.
• make it possible to import the repomap into one of our competitive agents, such as CodeAct. This functionality should be optional, like the github message (details).
• run experiments on SWE-Bench to see if this functionality improves accuracy

Describe alternatives you've considered

We could also implement this from scratch, or create improved code search functionality.

Additional context

• Parent issue Create Aider Agent #120

[neubig]

@ryanhoangt will probably take a look at this

[tobitege]

Reference: #2248 Add Aider-inspired RepoMap

[0xdevalias]

Describe alternatives you've considered

We could also implement this from scratch, or create improved code search functionality.

See also:

• Explore using stack graphs for better code search / navigation / context / repo map / etc #742

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[0xdevalias]

IMO this shouldn't be closed as stale

[FellowTraveler]

repomap and graphrag are critical features IMO for coding agents.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[TomLucidor]

Is this the reason why when the GitHub repo is too large, OpenHands will hang? That there are no systematic way of understanding/ignoring files and folders (including extensions)?

[neubig]

Hey @TomLucidor, sorry about the trouble!

If you have a large repo OpenHands should not hang, although it might become confused. The hanging behavior is likely due to something else. We'd be happy to help diagnose this, but in order to do so we'll need to be able to reproduce the problem. The easiest way for us to do this is if you can press the "feedback" button when you're encountering this behavior, or share logs. If you can do that it'd be great if you can open a new issue. I'd also be happy to discuss on slack.

[xingyaoww]

Hey @ryanhoangt can you self-assign this one?

[ryanhoangt]

Sure!

[BradKML]

As a little heads up, has anyone checked these repos out? (something along the lines of "codebase prompt")

• https://github.com/cyclotruc/gitingest
• https://github.com/mufeedvh/code2prompt
• https://github.com/raphaelmansuy/code2prompt
• https://github.com/kamilstanuch/codebase-digest

[enyst]

I found this kind of tool helpful, but not for feeding it as such to openhands, more for local work in or alongside vscode/cursor. It works for large codebases, to dump into Gemini, and go.

It's not great (Gemini is not great, at least pre-2.0), but I get something out of it. For now, I doubt it has the precision to be useful for the agent, considering the tradeoffs here. Have you tried to give it like that, in a message?

[BradKML]

@enyst what do you have in mind compared to the established tools made specifically for chatbots? Cus I am tempted but not even sure whether to put/generate it in a text file for the repo, OR to throw it directly in the dialogue box. Asking second opinions

[ryanpeach]

Just wanted to point out, you can import aider.repomap and get this feature for free.

[ryanpeach]

@rbren where is the PR that completed this?

[raymyers]

Just noting here that @xingyaoww discussed the reason for closing this issue in today's community meetup (Jan 31, 2025), so until more info is added here, it should be in the recording about 15 min in.

[xingyaoww]

I believe @ryanhoangt can explain more about this, but in summary, we've been trying something like this: All-Hands-AI/openhands-aci#41 for integrating Aider-inspired Repo map -- and it seems we didn't get it working on claude after a few attempts (e.g., performance on SWE-Bench didn't improve when adding those file skeleton) -- hoang feel free to correct me if I'm wrong, and it will be nice if you can share some numbers too!

I found this kind of tool helpful, but not for feeding it as such to openhands, more for local work in or alongside vscode/cursor. It works for large codebases, to dump into Gemini, and go.

As Engel said, there's a significant trade-off here:

• For single-turn systems where the LLM typically doesn't have the autonomy to look around and execute actions (e.g., I need to paste context to ChatGPT/Claude for it to generate another block of code), such type of RepoMap is essential -- an example of this would be Aider that primarily focuses on "file-editing" action, as opposed to executing general bash command like "grep -n".
• However, for a flexible multi-turn agentic system like OpenHands (CodeAct) -- sometimes adding more of this info actually did more harm than good. Basically, if you don't tell the agent anything, it would think, "I need to poke around and figure out right context,"; but if you tell it something like RepoMap (but it could be incomplete!) -- the agent would trust your RepoMap and think that's everything they need to know to complete the task. Here's another bitter lesson that happened recently (fix: revert #5506 for SWE-Bench performance regression #6491): we think providing less context of the file tree structure to the agent would help, but it actually hurt things a lot - so definitely, "what humans think is useful might not hold true for autonomous multi-turn agents" - and that's why we depends on evals for different design decisions!

I would think things like RepoMap were initially created to build context for a "single-turn LLM-based system."
However, they may not be useful for multi-turn systems like OpenHands, because the agent has the autonomy to look for stuff they need -- and a lot of times, it did better than "our human-curated repo map based on our intuition of what the model needs" -- another vivid reminder of the bitter lesson 😢 .

Here are some baseline code localization performance numbers on a paper I've been working on with some folks (haven't released yet! i can share the preprint here when we get it online):

You can see even if we don't have any system like "RepoMap" to manually construct context within a large repo - whereas other systems like "agentless" and "moatless" all have RepoMap-style systems - they didn't really outperform OpenHands on everything (if not worse!).

Based on these existing results, I've personally feel it is probably less productive to keep pushing towards this direction compared to other things after spending a lot of effort here -- but I could be totally wrong, and would be open to re-open this PR if people from the community is interested in taking over and actually making it work ❤

[xingyaoww]

cc @jimwhite -- thanks for the great discussion in today's community meeting! 👆 hopefully this response above would better answer your question :)

[aymannadeem]

Hey @xingyaoww! You mentioned that agents often do better when left to explore versus being given a repo map. Could you share some specific examples where the agent's autonomous exploration led to better solutions than when provided with structured context?

[BradKML]

I am thinking something like eza can make dynamic folder maps https://github.com/eza-community/eza
Right now when I am testing OH often times the agent would forget the pwd or context of where the terminal is, after cding in or out, which is annoying and warrant some kind of tool that shows both ls and hints at pwd.

[enyst]

often times the agent would forget the pwd or context of where the terminal is, after cding in or out, which is annoying and warrant some kind of tool that shows both ls and hints at pwd.

The LLM is already supposed to get in the prompt, after each command, the current directory:

OpenHands/openhands/events/observation/commands.py

Line 159 in fe8b927

ret += f'\n[Current working directory: {self.metadata.working_dir}]'

If that is not happening, maybe you could post a new issue, with some log with it missing or wrong? But I think it's also possible that we send it and sometimes the LLM ignores it anyway (depending on LLM).

[BradKML]

It might be an LLM issue then @enyst since I am heavily leaning on DeepSeek v3 atm, but it can't invoke the observation whenever it moves around working directories it seems?

[enyst]

It's automatic. The LLM asks for a command to execute ('cd ...'), the environment performs it and responds with an observation which includes cwd as above.

[ryanpeach]

Very interesting writeup @xingyaoww

[xingyaoww]

As promised! I'm sharing official result above and our preprint here: https://arxiv.org/abs/2503.09089

@czlll, @tangxiangru, @Hydrapse and I were working to figure out localization problems in coding agents using OpenHands. Finally be able to share the pre-print:

• We build a graph-based repository indexing (that only requires static analyzer)
  • Allow LLMs to perform complex multi-hop navigation & reasoning
  • only takes a few seconds per codebase (faster than RAG!)
• LLM as agents equipped with this indexing perform better than the existing tools we have in openhands (e.g., grep via bash)
• And most interesting to me, we finally show that frontier model + agent can outperform plain embedding-based or RAG-based procedure (Agentless/Moatless)
  • This probably tells us "a strong agent + graph-based indexing is all you need". There's no need to construct the embedding for the ENTIRE repo for retrieval -- it adds cost of embedding & a lot of engineering challenges: you need to update the embedding when the codebase is updated; you need to wait for the codebase to be indexed when you first open a new repo
  • The only downside here is that running these strong agents are expensive
• 👆 BUT! We were able to train a Qwen 2.5 32B model that almost perform on-par as Claude-3.5 -- and the fine-tuned 7B model is already competitive with OpenHands + sonnet 3.5 in function-level Acc@10. This should greatly help with the cost

@Zhaoling Chen will help integrate this LocAgent into OpenHands in the coming weeks so stay tuned 🙂

[paul-gauthier]

Looks like an interesting paper. Thanks for the somewhat oblique and generic citation to aider.

Is there a reason you didn't directly cite aider's repo-map, discuss it in the related work section or include it in your experimental evaluations?

At a minimum, your work seems quite strongly "inspired by" aider's repo-map. You've been participating in this GitHub issue titled "Aider-inspired RepoMap" for months, discussing your attempts to implement and evaluate a version of it in OpenHands.

Your top-listed contribution sounds very familiar.

Our contributions address critical gaps in existing approaches: We introduce a heterogeneous graph representation that captures both explicit and implicit code relationships, enabling efficient multi-hop reasoning. Our lightweight graph-based indexing process takes only seconds per repository and requires minimal storage.

Aider's original repo-map code and blog article were published in May 2023. A follow up blog article was published in Oct 2023, describing the tree-sitter implementation of repo-map.

[jmvldz]

Very exciting @xingyaoww! Thank you for sharing and for the follow up on this thread.

[xingyaoww]

@paul-gauthier Sorry for the oversight & appreciate the feedback! I didn't realize there's a specific blog directly discussing the "repo-map" - we will be sure to cite and discuss them in the next revision of the preprint :) (cc @czlll)

I have to admit, even though we are discussing this under the thread of Aider-inspired RepoMap, the question we were trying to answer in this paper is more about which is better, "agentic-based information gathering" or "passively provided information" (e.g., via embedding, where the LLM has few options to control the information being retrieved). IRRC, the original AiderMap is designed primarily for the latter category, and it didn't immediately jump into my mind when reviewing the paper 😅 .

But now, reflecting back, the indexing approach we took does look similar to what AiderMap was doing under the hood and we will certainly discuss it and give credits.

I'd say the differences between LocAgent vs. Aider are more about "how LLMs are using these graph-based indexing." My (maybe outdated) knowledge of Aider is that it automatically constructs a RepoMap for LLM (based on previously mentioned keywords, etc) to provide the context for LLM to perform edit (more similar to the passively provided information); whereas LocAgent is more about allowing LLM to use tools to explicitly traverse the graph (more leaning towards agentic-based information gathering - which is something we are hoping to continue to do in OpenHands).

[tangxiangru]

@paul-gauthier We will cite aider's repo-map in our next version. Thanks.