[Feature Request] Enable indexing of remote and local docs to bridge local-model knowledge gaps, and auto-include documentation

[sealad886]

Background

Roo Code’s Codebase Indexing graduated from experimental to a core feature recently, providing semantic search across the repository’s code. However, local models frequently lack comprehensive, up-to-date, or domain-specific knowledge (e.g., Python packaging best practices, framework internals, nuanced API usage), leading to context-switching, manual documentation searches, or costly/inconsistent RAG prompts.

Bridging this gap by integrating both external and internal documentation—and doing so intelligently—would make Roo Code vastly more effective and user-friendly.

Proposal

What is being proposed?
Two major upgrades for Roo Code:

1. Intelligent Inclusion of Documentation in Search:
   Roo Code should automatically and intelligently surface both code and documentation results in the IDE, closing knowledge gaps for local models—-without requiring the user to specify when to “look up docs.” Users simply ask their questions in natural language (code, planning, troubleshooting, etc.), and Roo Code handles the rest:

   • Unified Search: Results include both relevant code snippets and documentation excerpts (from remote and local sources), ranked together for relevance.
   • Automatic Doc Lookup: Roo Code decides when documentation lookup is needed based on its confidence in code-based answers. This could be triggered by:
     • Low similarity scores for code matches
     • LLM self-reporting low confidence (via a prompt such as “On a scale of 1–5, how confident are you?”)
     • High output variance between Roo Code modes (“ensemble disagreement”)
     • Low token-level log-probabilities (where supported)
   • Zero Extra Steps: No manual toggling or re-prompting—the system “just works.” The user does not need to specify code vs. docs or remember which documentation site to search.

2. Indexing of External Documentation Sources:

   • Remote Docs: Official web documentation (e.g., Python Packaging User Guide), framework docs, API specs, internal wikis, or any other web-based source can be added as sources to the index.
   • Local Docs: Local Markdown, HTML, reStructuredText, and other documentation formats in the project repo or on local network shares are also indexed.
   • Backend: Uses a lightweight crawler/proxy (e.g., Nginx plus a custom fetcher), which periodically retrieves, parses, chunks, and embeds external docs. All embeddings (code and docs) are stored in the same Qdrant instance, supporting unified similarity search.
   • Results Labeling: Results are clearly labeled (“from code” vs. “from docs”) in the IDE, with clickable breadcrumbs and the option to open documentation in a browser.

Why does this matter?

• Roo Code will proactively close knowledge gaps inherent in local models—-proactively preventing hallucinations in low- and high-powered models.
• Developers get a single, streamlined search workflow. No more context-switching or manual doc lookups—-best-practice guidance and relevant code context appear side-by-side.
• Teams can point Roo at their most trusted and authoritative sources (internal, private, public, or third-party), ensuring up-to-date answers and accelerating onboarding and code quality.

How it works (Smart Lookup Pipeline)

flowchart TD
    U[User submits natural language query] --> V[Initial Roo Code search #40;codebase_search, source=cod#41;]
    V --> W{Is top code match confident?}
    W -- Yes --> X[Surface code results only]
    W -- No --> Y[Auto-trigger docs search #40;codebase_search, source=docs#41;]
    Y --> Z[Merge code & doc results]
    Z --> M[Display final results in IDE, labeled by source]
    X --> M

Loading

Example Use Cases

• User asks: “How do I publish a wheel?”
  • If code search yields low confidence, Roo surfaces the relevant section from packaging.python.org alongside any local packaging scripts or configs.
• User requests: “What is the order of middleware execution?”
  • Roo merges Django docs excerpts with relevant code snippets for side-by-side clarity.

Technical Architecture

flowchart TD
  subgraph Backend Crawler/Proxy
    A[Config: doc URLs & globs] --> B[Crawler Scheduler]
    B --> C[Cache HTML/MD]
    C --> D[Chunk & Embed Worker]
    D --> E[Qdrant “docs” Collection]
    F[Nginx Proxy] --- B
    F --- D
  end

  subgraph Roo Code IDE
    U[User Query] --> V[Initial codebase_search#40;source=code#41;]
    V --> W{Top Score ≥ 0.5?}
    W -- Yes --> X[Show Code Results]
    W -- No  --> Y[Auto-trigger codebase_search#40;source=docs#41;]
    Y --> Z[Merge Code + Doc Results]
    Z --> M[Rerank & Display Unified Results]
  end

Loading

Why Not Just RAG?

• Persistent, Incremental Index: One-time embedding with incremental updates vs. on-the-fly HTTP+LLM calls every time.
• Offline & Firewall-Safe: Cached docs index works behind proxies, VPNs, or in air-gapped setups.
• Single Relevance Ranking: Code and docs live in the same vector space—no prompt engineering required.
• Lower Latency & Cost: Avoids repeated external fetches and prompt tokens; supports batch updates.

References

• Codebase Indexing Docs
• codebase_search Tool
• Packaging User Guide – Overview
• RAG vs. Indexing
• Learning-to-Rank (Wikipedia)
• Nginx Reverse Proxy