GraphiteSendException: Socket closed before able to send data to ('x.x.x.x', 2003), with error: [Errno 32] Broken pipe

[stevezau]

Hey Guys,

I'm getting a lot of exceptions at random..

The graphite listener is on the same server so network issues seem unlikely. Any idea where i should start to look?

  File "./travel/services/payments.py", line 59, in charge
    m_client.send('latency', time_taken)
  File "/usr/local/lib/python2.7/dist-packages/graphitesend/graphitesend.py", line 360, in send
    return self._dispatch_send(message)
  File "/usr/local/lib/python2.7/dist-packages/graphitesend/graphitesend.py", line 262, in _dispatch_send
    self._handle_send_error(e)
  File "/usr/local/lib/python2.7/dist-packages/graphitesend/graphitesend.py", line 276, in _handle_send_error
    (self.addr, error)

[daniellawrence]

Hello @stevezau ,

By default this lib is going to try and make a connection to "graphite:2003".
Is it possible that carbon is bound to localhost:2003 and 'carbon' resolves to a real ip address?

A quick test could be the follow commands

getent hosts graphite
telnet graphite 2003
ss -ltpn |grep :2003

[stevezau]

Hi, I've told it to use a specific graphite ip:port. About 90% of the messages get through, randomly ones don't.

Sometimes it can spike where about 60-70 metrics could be sent within the space of 20-30 seconds? Could it be a max network connection issue?

[daniellawrence]

Depends on how your sending the metrics, size of you application, machine is running on, etc...

Grouping the metrics together and pushing at an interval with reduce any overhead and allow you to get much higher throughput.

Based on how you have your code, you might want to use something like statsd, You can then have it aggregate the metrics and then push into graphite.