KafkaFetch metrics

[codyaray]

Where do the KafkaFetch metrics come from? I don't see anything in the code here that provides it, looks like its only bolt acks and emits. But I see it in my output. Baffling.

Can you shed some light on this mystery?

[velvia]

Sorry, I don't remember anything about KafkaFetch, maybe you can include
some output and refresh memory?

On Mon, Jun 16, 2014 at 11:19 AM, Cody A. Ray notifications@github.com
wrote:

Where do the KafkaFetch metrics come from? I don't see anything in the
code here that provides it, looks like its only bolt acks and emits. But I
see it in my output. Baffling.

Can you shed some light on this mystery?

—
Reply to this email directly or view it on GitHub
#7.

The fruit of silence is prayer;
the fruit of prayer is faith;
the fruit of faith is love;
the fruit of love is service;
the fruit of service is peace. -- Mother Teresa

[codyaray]

Thanks. I really appreciate your help.

Example Output

curl localhost:7000/metricz | python -m json.tool

{
    <snip>
    "kafka.consumer.FetchRequestAndResponseMetrics": {
        "-AllBrokersFetchRequestRateAndTimeMs": {
            "duration": {
                "max": 66.053208, 
                "mean": 5.158013333333333, 
                "median": 1.580337, 
                "min": 1.20686, 
                "p75": 2.1713345, 
                "p95": 60.44109029999992, 
                "p98": 66.053208, 
                "p99": 66.053208, 
                "p999": 66.053208, 
                "std_dev": 14.073510931389972, 
                "unit": "milliseconds"
            }, 
            "rate": {
                "count": 21, 
                "m1": 0.38805766751044357, 
                "m15": 0.21327794881777054, 
                "m5": 0.23950459512485547, 
                "mean": 1.1782846420670174, 
                "unit": "seconds"
            }, 
            "type": "timer"
        }, 
        "-AllBrokersFetchResponseSize": {
            "count": 21, 
            "max": 481612.0, 
            "mean": 25897.809523809523, 
            "median": 47.0, 
            "min": 47.0, 
            "p75": 7259.0, 
            "p95": 436580.6999999994, 
            "p98": 481612.0, 
            "p99": 481612.0, 
            "p999": 481612.0, 
            "std_dev": 104660.12123613227, 
            "type": "histogram"
        }, 
        "-host_storm01st1-port_9092-FetchRequestRateAndTimeMs": {
            "duration": {
                "max": 62.902036, 
                "mean": 4.92107161904762, 
                "median": 1.518957, 
                "min": 1.131581, 
                "p75": 1.990606, 
                "p95": 57.586455299999926, 
                "p98": 62.902036, 
                "p99": 62.902036, 
                "p999": 62.902036, 
                "std_dev": 13.408310787575564, 
                "unit": "milliseconds"
            }, 
            "rate": {
                "count": 21, 
                "m1": 0.4027702054581854, 
                "m15": 0.21437984055996545, 
                "m5": 0.24275566579277777, 
                "mean": 1.1879119373029434, 
                "unit": "seconds"
            }, 
            "type": "timer"
        }, 
        "-host_storm01st1-port_9092-FetchResponseSize": {
            "count": 21, 
            "max": 481612.0, 
            "mean": 25897.809523809523, 
            "median": 47.0, 
            "min": 47.0, 
            "p75": 7259.0, 
            "p95": 436580.6999999994, 
            "p98": 481612.0, 
            "p99": 481612.0, 
            "p999": 481612.0, 
            "std_dev": 104660.12123613227, 
            "type": "histogram"
        }
    }, 
    "storm.taskInfo.b-0": {
        "acks": {
            "count": 2148, 
            "event_type": "acks", 
            "m1": 44.088832222778166, 
            "m15": 19.863986902034526, 
            "m5": 23.536420561149804, 
            "mean": 105.89591328349071, 
            "type": "meter", 
            "unit": "seconds"
        }, 
        "emits": {
            "count": 2148, 
            "event_type": "emits", 
            "m1": 44.93373712610852, 
            "m15": 21.037535123501385, 
            "m5": 24.658333904879285, 
            "mean": 105.8964372090853, 
            "type": "meter", 
            "unit": "seconds"
        }
    }, 
    <snip>
}

[velvia]

Oh, I think those metrics are exposed through Kafka's JMX / JavaBeans API.

On Tue, Jun 17, 2014 at 11:11 AM, Cody A. Ray notifications@github.com
wrote:

Thanks. I really appreciate your help.

Example Output

curl localhost:7000/metricz | python -m json.tool

{

"kafka.consumer.FetchRequestAndResponseMetrics": {
"-AllBrokersFetchRequestRateAndTimeMs": {
"duration": {
"max": 66.053208,
"mean": 5.158013333333333,
"median": 1.580337,
"min": 1.20686,
"p75": 2.1713345,
"p95": 60.44109029999992,
"p98": 66.053208,
"p99": 66.053208,
"p999": 66.053208,
"std_dev": 14.073510931389972,
"unit": "milliseconds"
},
"rate": {
"count": 21,
"m1": 0.38805766751044357,
"m15": 0.21327794881777054,
"m5": 0.23950459512485547,
"mean": 1.1782846420670174,
"unit": "seconds"
},
"type": "timer"
},
"-AllBrokersFetchResponseSize": {
"count": 21,
"max": 481612.0,
"mean": 25897.809523809523,
"median": 47.0,
"min": 47.0,
"p75": 7259.0,
"p95": 436580.6999999994,
"p98": 481612.0,
"p99": 481612.0,
"p999": 481612.0,
"std_dev": 104660.12123613227,
"type": "histogram"
},
"-host_storm01st1-port_9092-FetchRequestRateAndTimeMs": {
"duration": {
"max": 62.902036,
"mean": 4.92107161904762,
"median": 1.518957,
"min": 1.131581,
"p75": 1.990606,
"p95": 57.586455299999926,
"p98": 62.902036,
"p99": 62.902036,
"p999": 62.902036,
"std_dev": 13.408310787575564,
"unit": "milliseconds"
},
"rate": {
"count": 21,
"m1": 0.4027702054581854,
"m15": 0.21437984055996545,
"m5": 0.24275566579277777,
"mean": 1.1879119373029434,
"unit": "seconds"
},
"type": "timer"
},
"-host_storm01st1-port_9092-FetchResponseSize": {
"count": 21,
"max": 481612.0,
"mean": 25897.809523809523,
"median": 47.0,
"min": 47.0,
"p75": 7259.0,
"p95": 436580.6999999994,
"p98": 481612.0,
"p99": 481612.0,
"p999": 481612.0,
"std_dev": 104660.12123613227,
"type": "histogram"
}
},
"storm.taskInfo.b-0": {
"acks": {
"count": 2148,
"event_type": "acks",
"m1": 44.088832222778166,
"m15": 19.863986902034526,
"m5": 23.536420561149804,
"mean": 105.89591328349071,
"type": "meter",
"unit": "seconds"
},
"emits": {
"count": 2148,
"event_type": "emits",
"m1": 44.93373712610852,
"m15": 21.037535123501385,
"m5": 24.658333904879285,
"mean": 105.8964372090853,
"type": "meter",
"unit": "seconds"
}
},

}

—
Reply to this email directly or view it on GitHub
#7 (comment).

The fruit of silence is prayer;
the fruit of prayer is faith;
the fruit of faith is love;
the fruit of love is service;
the fruit of service is peace. -- Mother Teresa

[codyaray]

Sure - but where in the metrics_storm codebase are they pulled in? I only see the bolt ack/emit meters in MetricsStorm.scala. Yet there they are in the output. So I must be missing some magic in here.

[velvia]

No magic - the CodaHale metrics library automatically pulls in all JMX metrics and exposes them in the servlet route.

-Evan
"Never doubt that a small group of thoughtful, committed citizens can change the world" - M. Mead

On Jun 18, 2014, at 10:22 AM, "Cody A. Ray" notifications@github.com wrote:

Sure - but where in the metrics_storm codebase are they pulled in? I only see the bolt ack/emit meters in MetricsStorm.scala. Yet there they are in the output. So I must be missing some magic in here.

—
Reply to this email directly or view it on GitHub.