Disable Instrumentation for Some Endpoints

[reverendken]

I might be missing something in the documentation, but I've been trying to find a way to disable instrumentation for some endpoints when building with orchestrion and I can't find a way. Specifically, the /ready endpoint, which our load balancer uses, spams the trace logs because it's being called multiple times per second.

Using manual instrumentation, I think I could do something like this:

    r := chi.NewRouter()
    r.Use(chitrace.Middleware(	
	chitrace.WithIgnoreRequest(func(r *http.Request) bool {
		return r.URL.Path == "/health"
	}),
    ))

However, I can't seem to find a way to use any of the chitrace configuration functions (or, for that matter, those of any other tracer library) when I'm using orchestrion. Did I miss something in the docs or is what I'm trying to do not possible?

EDIT: My team has also pointed out that we've tried this in the endpoint handler, which doesn't seem to stop the traces to the /ready endpoint:

//orchestrion:ignore
func Ready() http.HandlerFunc {
	return func(w http.ResponseWriter, _ *http.Request) {
		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
		w.WriteHeader(http.StatusOK)
	}
}

[aminemat]

same issue here.
I also set a number of env vars without much success

  DD_TRACE_HEALTH_METRICS_ENABLED: "false"
  DD_TRACE_RESOURCE_FILTER_REGEX: "^(GET|HEAD|POST|PUT|DELETE|OPTIONS) /health.*"
  DD_TRACE_IGNORE_RESOURCE_NAMES: "GET /health,HEAD /health,OPTIONS /health"
  DD_HTTP_SERVER_BLACKLIST_TRACES: "/health"  

[mtoffl01]

Hi there,

Unfortunately, this isn't currently configurable in Orchestrion. However, the team plans to bring integration-level configuration—like what's available in dd-trace-go via manual instrumentation—to Orchestrion soon. In the meantime, you can use sampling rules to prevent traces from this endpoint from being ingested. See: https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_mechanisms/?tab=go#in-tracing-libraries-user-defined-rules

You can set the following environment variable on your service: DD_TRACE_SAMPLING_RULES='[{"resource": "GET /health", "sample_rate": 0}]'

Or via programmatic configuration:

rules := []tracer.SamplingRule{
    tracer.TagsResourceRule(nil, "GET /health", "", "", 0)
}
tracer.Start(tracer.WithSamplingRules(rules))
defer tracer.Stop()

More information here: https://pkg.go.dev/github.com/DataDog/dd-trace-go/v2@v2.0.1/ddtrace/tracer

If you are unfamiliar with ingestion, these docs are a good place to start:

• Trace pipeline: https://docs.datadoghq.com/tracing/trace_pipeline/
• Ingestion controls (mechanisms for configuring ingestion): https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls/?tab=remotely

[genesor]

Hi @aminemat

unfortunately DD_TRACE_SAMPLING_RULES does not work with Orchestrion.

Orchestrion is using dd-trace-go under the hood, everything related to the core of the tracer works the same as with manual instrumentation.

I created a test app today to ensure that DD_TRACE_SAMPLING_RULES was working as expected with Orchestrion. The code is available on this repo: https://github.com/genesor/chi-orchestrion-sampling, can you try to replicate use the same value for the sampling rule on your code ?

Note that the resource name needs to be an exact match to target the specific trace you want to ignore. If you have the default orchestrion.tool.go you also have the net/http span generated (but should be targeted the same way as they share the same name)

cc @reverendken feel free to check the repo as well. I've added your /health endpoint as well 😃

[aminemat]

@genesor thank you for taking the time to create a repo!
unfortunately, this is still not working for us, the tracer logs show the sampling rules are set but healthcheck calls still show in Datadog APM

Tracer Startup Logs

Datadog Tracer v2.0.0 INFO: DATADOG TRACER CONFIGURATION

{
  "date": "2025-06-27T18:31:19Z",
  "os_name": "Alpine Linux",
  "os_version": "unknown",
  "version": "v2.0.0",
  "lang": "Go",
  "lang_version": "go1.24.0",
  "env": "REDACTED",
  "service": "REDACTED",
  "agent_url": "http://REDACTED/v0.4/traces",
  "agent_error": "",
  "debug": false,
  "analytics_enabled": false,
  "sample_rate": "NaN",
  "sample_rate_limit": "100",
  "trace_sampling_rules": [
    {
      "resource": "get_/health",
      "sample_rate": 0
    },
    {
      "resource": "GET /health",
      "sample_rate": 0
    }
  ],
  "span_sampling_rules": null,
  "sampling_rules_error": "",
  "service_mappings": null,
  "tags": {
    "runtime-id": "c313fb32-6fdc-41b0-8cd6-2b61316504f0"
  },
  "runtime_metrics_enabled": false,
  "runtime_metrics_v2_enabled": false,
  "profiler_code_hotspots_enabled": true,
  "profiler_endpoints_enabled": true,
  "dd_version": "v1.0.2-pre",
  "architecture": "amd64",
  "global_service": "ffoxrelay",
  "lambda_mode": "false",
  "appsec": false,
  "agent_features": {
    "DropP0s": true,
    "Stats": true,
    "StatsdPort": 8125
  },
  "integrations": {
    "AWS SDK": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "AWS SDK v2": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Bun": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "BuntDB": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Cassandra": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Consul": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Elasticsearch v5": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Elasticsearch v6": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "FastHTTP": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Fiber": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Gin": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Google API": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Gorilla Mux": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Gorm v1": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Graph Gophers GraphQL": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "GraphQL-Go GraphQL": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "HTTP": {
      "instrumented": true,
      "available": false,
      "available_version": ""
    },
    "HTTP Router": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "HTTP Treemux": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "IBM sarama": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Kafka (confluent)": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Kafka (confluent) v2": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Kafka v0": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Kubernetes": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "LevelDB": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Logrus": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Memcache": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "MongoDB": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "MongoDB (mgo)": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Negroni": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "PGX": {
      "instrumented": true,
      "available": true,
      "available_version": "v5.7.5"
    },
    "Pub/Sub": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Redigo": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Redis": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Redis v7": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Redis v8": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Redis v9": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Rueidis": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "SQL": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "SQLx": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Shopify sarama": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Twirp": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Valkey": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "Vault": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "chi": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "chi v5": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "echo v4": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "gRPC": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "go-pg v10": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "go-restful v3": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "gqlgen": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "log/slog": {
      "instrumented": true,
      "available": false,
      "available_version": ""
    },
    "miekg/dns": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    }
  },
  "partial_flush_enabled": false,
  "partial_flush_min_spans": 1000,
  "orchestrion": {
    "enabled": false
  },
  "feature_flags": [],
  "propagation_style_inject": "datadog,tracecontext,baggage",
  "propagation_style_extract": "datadog,tracecontext,baggage",
  "tracing_as_transport": false,
  "dogstatsd_address": "REDACTED"
}

[genesor]

@aminemat thank you for providing the startup logs 👍

It seems that you are not using chi because it's not instrumented.

    "chi": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },
    "chi v5": {
      "instrumented": false,
      "available": false,
      "available_version": ""
    },

The sampling rules we've provided are meant to drop traces fitting a specific resource name GET /health and GET /ready that are generated by the contribs in dd-trace-go

In order to help you I would need the following info:

• Which HTTP framework/router are you using ? Could you show me the snippet of how you're defining the route that you want to skip ?
• Could you provide me with the tracer logs of the HTTP span created/finished that you want to ignore ?

[aminemat]

sure thing, we're using the native net/http router with Go 1.24.

mux := http.NewServeMux()
mux.HandleFunc("GET /health", healthHandler.Health)

[genesor]

Hi @aminemat

I created a demo branch for the native net/http router and enabled net/http instrumentation on it, in the master branch only chi.v5 was enabled this could be the reason why it was failing on your side. https://github.com/genesor/chi-orchestrion-sampling/tree/net/http

Here the trace debug logs produced when I hit /health and as you can see the correct sampling decision is applied _sampling_priority_v1:-1

2025/07/01 10:12:48 Datadog Tracer v2.0.1 DEBUG: Started Span: dd.trace_id="6863988000000000642af5df837f3600" dd.span_id="7217851693147305472" dd.parent_id="0", Operation: http.request, Resource: GET /health, Tags: map[component:net/http http.host:localhost:8181 http.method:GET http.route:/health http.url:http://localhost:8181/health http.useragent:curl/8.7.1 language:go runtime-id:cec61569-d633-4770-8eac-fe9a5a691f41 span.kind:server], map[_dd.profiling.enabled:0 _dd.rule_psr:0 _dd.top_level:1 _sampling_priority_v1:-1]
2025/07/01 10:12:48 Datadog Tracer v2.0.1 DEBUG: Finished Span: dd.trace_id="6863988000000000642af5df837f3600" dd.span_id="7217851693147305472" dd.parent_id="0", Operation: http.request, Resource: GET /health, Tags: map[component:net/http http.host:localhost:8181 http.method:GET http.route:/health http.status_code:200 http.url:http://localhost:8181/health http.useragent:curl/8.7.1 language:go runtime-id:cec61569-d633-4770-8eac-fe9a5a691f41 span.kind:server], map[_dd.profiling.enabled:0 _dd.rule_psr:0 _dd.top_level:1 _dd.trace_span_attribute_schema:0 _sampling_priority_v1:-1 process_id:88100]

If you need further help I will need the tracer debug logs that can be enabled with DD_TRACE_DEBUG=true.

What could cause the trace to be kept would be if this span created is not the root span of the trace.