Client observations creating too many uri tag values #4290
Description
After observability support was added to RestTemplateEurekaHttpClient in #4255 by using the RestTemplateBuilder metrics are being generated for Eureka client requests.
The ClientRequestObservationConvention adds the URI template of the requests as a low cardinality observability/metric key-value/tag.
The current implementations of EurekaHttpClient based on RestTemplate and WebClient assemble the full URL of the requsts and pass it to the client implementation. This causes a high number of values for the URI tag in the metrics, especially if the lastDirtyTimestamp changes often (due to e.g. flaky health of some component). By default only 100 values are allowed for the URI tag, additional values would be dropped and the metrics using these values are ignored.
Could be mitigated by passing at least the lastDirtyTimestamp as a uriVariable to the RestTemplate calls.
Version: spring-cloud-netflix-eureka-client 4.1.2
Sample of generated (prometheus) metrics
The following metrics where available after making the Eureka server unavailable twice and thus causing a status change to UNKNOWNand dirying of the InstanceInfo.
# HELP http_client_requests_seconds_max
# TYPE http_client_requests_seconds_max gauge
http_client_requests_seconds_max{client_name="localhost",error="ResourceAccessException",exception="ResourceAccessException",method="GET",outcome="UNKNOWN",status="CLIENT_ERROR",uri="/eureka/apps/delta"} 0.0410048
http_client_requests_seconds_max{client_name="localhost",error="ResourceAccessException",exception="ResourceAccessException",method="PUT",outcome="UNKNOWN",status="CLIENT_ERROR",uri="/eureka/apps/<appName>/<instanceId>?status=UP&lastDirtyTimestamp=1720440338928"} 0.0
http_client_requests_seconds_max{client_name="localhost",error="ResourceAccessException",exception="ResourceAccessException",method="PUT",outcome="UNKNOWN",status="CLIENT_ERROR",uri="/eureka/apps/<appName>/<instanceId>?status=UP&lastDirtyTimestamp=1720441059188"} 0.0409643
http_client_requests_seconds_max{client_name="localhost",error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/eureka/apps/"} 0.0054987
http_client_requests_seconds_max{client_name="localhost",error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/eureka/apps/delta"} 0.0476062
http_client_requests_seconds_max{client_name="localhost",error="none",exception="none",method="POST",outcome="SUCCESS",status="204",uri="/eureka/apps/<appName>"} 0.0078605
http_client_requests_seconds_max{client_name="localhost",error="none",exception="none",method="PUT",outcome="CLIENT_ERROR",status="404",uri="/eureka/apps/<appName>/<instanceId>?status=UP&lastDirtyTimestamp=1720440338928"} 0.0
http_client_requests_seconds_max{client_name="localhost",error="none",exception="none",method="PUT",outcome="CLIENT_ERROR",status="404",uri="/eureka/apps/<appName>/<instanceId>?status=UP&lastDirtyTimestamp=1720441059188"} 0.0194635
http_client_requests_seconds_max{client_name="localhost",error="none",exception="none",method="PUT",outcome="SUCCESS",status="200",uri="/eureka/apps/<appName>/<instanceId>?status=DOWN&lastDirtyTimestamp=1720440308847"} 0.0
http_client_requests_seconds_max{client_name="localhost",error="none",exception="none",method="PUT",outcome="SUCCESS",status="200",uri="/eureka/apps/<appName>/<instanceId>?status=UP&lastDirtyTimestamp=1720440338928"} 0.0
http_client_requests_seconds_max{client_name="localhost",error="none",exception="none",method="PUT",outcome="SUCCESS",status="200",uri="/eureka/apps/<appName>/<instanceId>?status=UP&lastDirtyTimestamp=1720441059188"} 0.0066927
http_client_requests_seconds_max{client_name="localhost",error="none",exception="none",method="PUT",outcome="SUCCESS",status="200",uri="/eureka/apps/<appName>/<instanceId>?status=UP&lastDirtyTimestamp=1720442649723"} 0.0058572