Adds EP-10411: Gateway API Inference Extension Support (#10420)

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>
Co-authored-by: Tim Flannagan <timflannagan@gmail.com>

Author: danehans

design/10411.md

@@ -0,0 +1,86 @@
+# EP-10411: Gateway API Inference Extension Support
+
+* Issue: [#10411](https://github.com/kgateway-dev/kgateway/issues/10411)
+
+## Background
+
+This EP proposes adding [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main) (GIE) support. GIE is an open source project that originated from [wg-serving](https://github.com/kubernetes/community/tree/master/wg-serving) and is sponsored by [SIG Network](https://github.com/kubernetes/community/blob/master/sig-network/README.md#gateway-api-inference-extension). It provides APIs, a scheduling algorithm, a reference extension implementation, and controllers to support advanced routing of GenAI/ML network traffic.
+
+## Motivation
+
+Provide a standards-based approach to load balancing inference workloads.
+
+## Goals
+
+The following list defines goals for this EP.
+
+* Provide initial GIE support allowing for easy experimentation of advanced LLM traffic routing via the [Endpoint Selection Extension](https://gateway-api-inference-extension.sigs.k8s.io/#endpoint-selection-extension) (ESE), one of GIE's reference extension implementations.
+* Allow users to enable/disable this feature.
+* Implement GIE as a kgateway plugin.
+* Add [InferencePool](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/api/v1alpha1/inferencepool_types.go) as a supported HTTPRoute backend reference.
+* Provide the ability to manage the GIE deployment.
+* Provide e2e testing of this feature.
+* Provide initial user documentation, e.g. quick start guide.
+
+## Non-Goals
+
+The following list defines non-goals for this EP.
+
+* Run production traffic using this feature.
+* Provide kgateway-specific GIE extensions.
+* Support non-GIE traffic routing functionality that may be achieved through integration with kgateway-specific APIs.
+* Provide stats for the initial GIE implementation since it lacks a metrics endpoint.
+* Secure the gRPC connection between Gateway and GIE implementations.
+* Support kgateway upgrades when this feature is enabled.
+
+## Implementation Details
+
+The following sections describe implementation details for this EP.
+
+### Configuration
+
+Update the [configuration](https://github.com/kgateway-dev/kgateway/blob/main/install/helm/kgateway/values.yaml) API to enable/disable this feature.
+The feature will be disabled by default:
+
+```yaml
+inferenceExtension:
+  enabled: false
+```
+
+For the initial implementation, no other configuration parameters will be exposed to users. Since Inference Extension is decoupled from Gateway API,
+GatewayParameters will not be used for runtime configuration.
+
+__Note:__ A user must install the Inference Extension CRDs before installing kgateway with this feature enabled.
+
+### Plugin
+
+Add GIE ESE as a supported [plugin](https://github.com/kgateway-dev/kgateway/tree/main/internal/kgateway/extensions2/plugins). The plugin will:
+
+* Manage [Endpoints](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/endpoint.html) based on the [InferencePool](https://gateway-api-inference-extension.sigs.k8s.io/api-types/inferencepool/) resource specification. The Gateway implementation, e.g. Envoy proxy, will forward matching requests using the [External Processing Filter](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ext_proc/v3/ext_proc.proto#external-processing-filter-proto) (ext proc) to the ESE deployment. The ESE is responsible for processing the request, selecting an Endpoint, and returning the selected Endpoint to Envoy for routing.
+* Manage route filters to redirect requests to the ESE ext proc server.
+* Manage the ext proc server cluster.
+
+### Controllers
+
+* Add a controller to reconcile InferencePool custom resources, manage status, trigger the ESE deployment, etc.
+* Controllers should run only if the feature is enabled and GIE CRDs exist.
+* Update RBAC rules to allow controllers to access GIE custom resources.
+
+### Deployer
+
+Since GIE extensions are decoupled from the Gateway implementation, a separate [deployer](https://github.com/kgateway-dev/kgateway/tree/main/internal/kgateway/deployer)
+will be created to manage the required ESE resources, e.g. Deployment, Service, etc.
+
+### Translator and Proxy Syncer
+
+* Add InferencePool as a supported HTTPRoute backend reference.
+* Update the [translator](https://github.com/kgateway-dev/kgateway/tree/main/internal/kgateway/translator) package to handle InferencePool references from the HTTPRoute type.
+* Enhance the [proxy_syncer](https://github.com/kgateway-dev/kgateway/tree/main/projects/gateway2/proxy_syncer) to translate the InferencePool custom resource into the IR and sync with the proxy client. When an HTTPRoute references an InferencePool, ensure the Envoy ext_proc filter is attached or the cluster references the ESE cluster.
+
+### Reporting
+
+* Update the [reporter](https://github.com/kgateway-dev/kgateway/tree/main/projects/gateway2/reports) package to support status reporting, e.g. `ResolvedRefs=true` when HTTPRoute references an InferencePool.
+
+## Open Questions
+
+1. ~~Is a new plugin type required or can an existing type be utilized, e.g. UpstreamPlugin?~~ A new plugin type will be created specific to GIE.