[Feature Request] Code Index needs robust throttling configurability

[sealad886]

Configurable Throttling for Index Process

Introduction

Following the rollout of the Code Index module, it’s essential to introduce a mechanism that modulates indexing throughput in response to both static policies and dynamic system conditions. Configurable throttling delivers that capability, preventing resource contention during spikes and ensuring predictable performance under variable load.

Objectives and Success Criteria

• Throughput Control: Enforce document-per-second caps with minimal variance.
• Resource Stability: Keep CPU and memory utilization within safe thresholds, even at peak ingestion.
• Adaptivity: Automatically adjust limits based on queue depth and system health signals.
• Visibility: Surface real-time metrics on ingest rate, queue length, and throttle status.

Throttling Concepts

Configurable throttling comprises two complementary strategies, enhanced by adaptive logic:

1. Rate Limiting
   • Smooths bursts by capping the number of index operations per second.
   • Ensures downstream services and storage layers are not overwhelmed.
2. Concurrency Control
   • Restricts the number of parallel indexing workers.
   • Balances workload distribution and preserves headroom for critical tasks.
3. Adaptive Adjustment
   • Periodically evaluates runtime metrics (e.g., queue size, CPU load).
   • Dynamically scales rate and concurrency up or down, reacting to backlogs or underutilization.

Together, these controls create a feedback loop that keeps indexing throughput within operator-defined safe bounds while reacting in real time to changing system conditions.

Technical Rationale and Benefits

Benefit	Explanation
Operational Stability	Prevents “indexing storms” that could saturate compute resources, leading to degraded performance or failures.
Predictable Performance	Guarantees the indexing subsystem consumes only its allotted share of resources, safeguarding user-facing services from contention.
Cost Containment	Caps cloud or cluster usage, reducing the risk of unexpected overages during sudden workload surges.
SLA Compliance	Maintains throughput and latency guarantees by adhering to configurable ingest policies, a critical requirement for enterprise deployments.
Scalability Across Environments	Enables low-volume development settings with tight limits and high-volume production environments with more generous caps—all via configuration, without code changes.
Extensibility	New health signals (e.g., I/O wait, GC pauses) or alternative throttling strategies can be incorporated seamlessly, thanks to the modular concept.

Conclusion

Configurable throttling is the logical next evolution in hardening the code-indexing pipeline. By combining rate limiting, concurrency control, and adaptive adjustments, it delivers a stable, cost-efficient, and predictable indexing service—fully tunable via configuration and observable through real-time metrics. This approach ensures that as workload patterns shift, our system remains resilient, performant, and cost-effective.

[nikhil-swamix]

Yes, ollama struggles with high concurrent requests, and outright fails in my case!
please bring this to attention.

[sealad886]

You could try increasing your queue length by setting OLLAMA_MAX_QUEUE to something higher. The default it ridiculously low, iirc.