Make span available in http.Request Context

[abh]

I don't know fi this is an otelconnect issue or a general connectrpc issue (I'm new to both).

I have a mix of interceptors (including this one) and http.Handlers. It'd be very helpful if this interceptor added the request span to the http.Request context.

Right now to have tracing in the http.Handler middleware I have to use otelhttp which adds another span to the trace, or do similar work to "recapture" the traceparent from the incoming headers.

[emcfarlane]

Hey @abh , it currently isn't possible to update the requests context from an interceptor as the interceptor will always be called within the connect handler. Updating the context using WithContext creates a shallow copy so any changes to the request won't be visible outside. We are looking at improving support for HTTP middleware by adding Handler and Client options for middleware in connect-go.

[emcfarlane]

As a workaround for this issue you may create a new middleware layer above the authn middleware to set a parent trace and attach it to the request context. This will provide all subsequent middleware and interceptors with access to the span. As an example:

func RequestStartMiddleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ctx, span := trace.StartSpan(r.Context(), "observability.RequestStart")
		defer span.End()
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

[mattdowdell]

On a related note, I started looking at using https://github.com/connectrpc/authn-go and noticed fairly quickly that it's effectively a HTTP middleware rather than a connect interceptor. The apparently reason for this is that validating authorization credentials is a good thing to do before unmarshaling the request body. That totally makes sense, particularly for public APIs. However, it means that when observability is provided by an interceptor such as otelconnect, there is no record of failures captured by HTTP middleware.

For example, if credentials are rejected for any reason by the authn middleware, otelconnect will not report that. Similarly, if another error occurs in the HTTP->RPC flow, such as message couldn't be parsed, otelconnect won't notice. To be fair, I don't think this is markedly different to what gRPC does, but it does represent a loss of observability you wouldn't see in plain HTTP servers.

Would you consider supporting a middleware in otelconnect too? Most tracing propagators I've seen work on request headers, and I note there's a private function in connect-go that maps codes to HTTP statuses so it's possible to capture most of the same information at the HTTP level. However, I'm less sure about the span events and metrics that operate on messages, particularly for streaming. It may be that such events simply cannot be observed in middleware. That'd be a shame, but perhaps it'd mean that both middleware and interceptor have a place.