Fixing EMF output for detailed summary metrics #313

dricross · 2025-05-09T18:02:27Z

Description

Enabling the “DetailedMetrics” configuration for the awsemfexporter causes this OTeL Prometheus receiver → awsemfexporter pipeline to output n + 2 log events:

One log event for each quantile
One log event for count metric
One log event for sum metric

The original implementation for the DetailedMetrics functionality outputs count/sum metrics as one log event instead of two:

One log event for each quantile
One log event for count/sum metric

A recent change to the awsemfexporter caused the groupKey for count and summary metrics to be unique so they are no longer grouped together and therefore emitted as separate log events.

This PR restore the original functionality.

Testing

Agent integ tests: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/14972947252
AppSignals E2E test: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/14972947579

Followed something similar to the OTLP -> AMP integration test (https://github.com/aws/amazon-cloudwatch-agent-test/blob/main/test/amp/resources/otlp_pusher.sh) but configured the agent to push to EMF instead of AMP.

Use following OTLP metrics file with summary metric only:

{
  "resourceMetrics": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "my.service"
            }
          }
        ]
      },
      "scopeMetrics": [
        {
          "scope": {
            "name": "my.library",
            "version": "1.0.0",
            "attributes": [
              {
                "key": "my.scope.attribute",
                "value": {
                  "stringValue": "some scope attribute"
                }
              }
            ]
          },
          "metrics": [
            {
              "name": "my.summary",
              "unit": "ms",
              "description": "I am a Summary metric",
              "summary": {
                "dataPoints": [
                  {
                    "startTimeUnixNano": START_TIME,
                    "timeUnixNano": START_TIME,
                    "count": 100,
                    "sum": 5000.0,
                    "quantileValues": [
                      {
                        "quantile": 0.50,
                        "value": 43.5
                      },
                      {
                        "quantile": 0.95,
                        "value": 95.2
                      },
                      {
                        "quantile": 0.99,
                        "value": 97.8
                      }
                    ],
                    "attributes": [
                      {
                        "key": "my.summary.attr",
                        "value": {
                          "stringValue": "some value"
                        }
                      }
                    ]
                  }
                ]
              }
            }
          ]
        }
      ]
    }
  ]
}

Start with this agent config:

{
  "agent": {
    "metrics_collection_interval": 15,
    "run_as_user": "root",
    "debug": true
  },
  "logs": {
    "metrics_collected": {
      "otlp": {
        "http_endpoint": "127.0.0.1:1234"
      }
    }
  }
}

Restart agent to run translator, then manually update agent otel config since json config doesn't support setting detailed metrics:

exporters:
    awsemf:
..
        detailed_metrics: true
...

Start the agent without running translator by executing the binary directly:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml -envconfig /opt/aws/amazon-cloudwatch-agent/etc/env-config.json -otelconfig /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.yaml -pidfile /opt/aws/amazon-cloudwatch-agent/var/amazon-cloudwatch-agent.pid &

Run the otlp_pusher.sh

while true; export START_TIME=$(date +%s%N); do
  cat ./resources/otlp_metrics.json | sed -e "s/START_TIME/$START_TIME/" > otlp_metrics.json;
  curl -H 'Content-Type: application/json' -d @otlp_metrics.json -i http://127.0.0.1:1234/v1/metrics --verbose;
sleep 5; don

Check CloudWatch logs

Before changes (5 log events)

{
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent",
            "Dimensions": [
                [
                    "my.summary.attr",
                    "service.name",
                    "quantile"
                ]
            ],
            "Metrics": [
                {
                    "Name": "my.summary",
                    "Unit": "Milliseconds",
                    "StorageResolution": 60
                }
            ]
        }
    ],
    "Timestamp": "1746812484637",
    "Version": "0",
    "my.summary.attr": "some value",
    "quantile": "0.5",
    "service.name": "my.service",
    "my.summary": 43.5
}
{
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent",
            "Dimensions": [
                [
                    "my.summary.attr",
                    "service.name",
                    "quantile"
                ]
            ],
            "Metrics": [
                {
                    "Name": "my.summary",
                    "Unit": "Milliseconds",
                    "StorageResolution": 60
                }
            ]
        }
    ],
    "Timestamp": "1746812484637",
    "Version": "0",
    "my.summary.attr": "some value",
    "quantile": "0.95",
    "service.name": "my.service",
    "my.summary": 95.2
}
{
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent",
            "Dimensions": [
                [
                    "my.summary.attr",
                    "service.name",
                    "quantile"
                ]
            ],
            "Metrics": [
                {
                    "Name": "my.summary",
                    "Unit": "Milliseconds",
                    "StorageResolution": 60
                }
            ]
        }
    ],
    "Timestamp": "1746812484637",
    "Version": "0",
    "my.summary.attr": "some value",
    "quantile": "0.99",
    "service.name": "my.service",
    "my.summary": 97.8
}
{
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent",
            "Dimensions": [
                [
                    "my.summary.attr",
                    "service.name"
                ]
            ],
            "Metrics": [
                {
                    "Name": "my.summary_count",
                    "Unit": "Milliseconds",
                    "StorageResolution": 60
                }
            ]
        }
    ],
    "Timestamp": "1746812484637",
    "Version": "0",
    "my.summary.attr": "some value",
    "service.name": "my.service",
    "my.summary_count": 100
}
{
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent",
            "Dimensions": [
                [
                    "service.name",
                    "my.summary.attr"
                ]
            ],
            "Metrics": [
                {
                    "Name": "my.summary_sum",
                    "Unit": "Milliseconds",
                    "StorageResolution": 60
                }
            ]
        }
    ],
    "Timestamp": "1746812484637",
    "Version": "0",
    "my.summary.attr": "some value",
    "service.name": "my.service",
    "my.summary_sum": 5000
}