Provide an inference-aware routing layer capable of making dynamic decisions based on real-time metrics and model states and factors like KV cache, prefix-cache, workload distribution, or latency requirements.

Status: In progress
Owner: Ernest Wong