April 9, 2012

Python - Getting Data Into Graphite - Code Examples

This post shows code examples in Python (2.7) for sending data to Graphite.

Once you have a Graphite server setup, with Carbon running/collecting, you need to send it data for graphing.

Basically, you write a program to collect numeric values and send them to Graphite's backend aggregator (Carbon).

To send data, you create a socket connection to the graphite/carbon server and send a message (string) in the format:

"metric_path value timestamp\n"
  • `metric_path`: arbitrary namespace containing substrings delimited by dots. The most general name is at the left and the most specific is at the right.
  • `value`: numeric value to store.
  • `timestamp`: epoch time.
  • messages must end with a trailing newline.
  • multiple messages maybe be batched and sent in a single socket operation. each message is delimited by a newline, with a trailing newline at the end of the message batch.

Example message:

"foo.bar.baz 42 74857843\n" 

Let's look at some (Python 2.7) code for sending data to graphite...


Here is a simple client that sends a single message to graphite.

Code:

#!/usr/bin/env python

import socket
import time


CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003

message = 'foo.bar.baz 42 %d\n' % int(time.time())

print 'sending message:\n%s' % message
sock = socket.socket()
sock.connect((CARBON_SERVER, CARBON_PORT))
sock.sendall(message)
sock.close()


Here is a command line client that sends a single message to graphite:

Usage:

$ python client-cli.py metric_path value

Code:

#!/usr/bin/env python

import argparse
import socket
import time


CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003


parser = argparse.ArgumentParser()
parser.add_argument('metric_path')
parser.add_argument('value')
args = parser.parse_args()


if __name__ == '__main__':
    timestamp = int(time.time())
    message = '%s %s %d\n' % (args.metric_path, args.value, timestamp)
    
    print 'sending message:\n%s' % message
    sock = socket.socket()
    sock.connect((CARBON_SERVER, CARBON_PORT))
    sock.sendall(message)
    sock.close()


Here is a client that collects load average (Linux-only) and sends a batch of 3 messages (1min/5min/15min loadavg) to graphite. It will run continuously in a loop until killed. (adjust the delay for faster/slower collection interval):

#!/usr/bin/env python
 
import platform
import socket
import time


CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003
DELAY = 15  # secs


def get_loadavgs():
    with open('/proc/loadavg') as f:
        return f.read().strip().split()[:3]


def send_msg(message):
    print 'sending message:\n%s' % message
    sock = socket.socket()
    sock.connect((CARBON_SERVER, CARBON_PORT))
    sock.sendall(message)
    sock.close()


if __name__ == '__main__':
    node = platform.node().replace('.', '-')
    while True:
        timestamp = int(time.time())
        loadavgs = get_loadavgs()
        lines = [
            'system.%s.loadavg_1min %s %d' % (node, loadavgs[0], timestamp),
            'system.%s.loadavg_5min %s %d' % (node, loadavgs[1], timestamp),
            'system.%s.loadavg_15min %s %d' % (node, loadavgs[2], timestamp)
        ]
        message = '\n'.join(lines) + '\n'
        send_msg(message)
        time.sleep(DELAY)


Resources:

2 comments:

Kamil Kisiel said...

The graphite docs are actually now here:

http://readthedocs.org/projects/graphite/

Also, for getting data in to graphite from Python you can just use Pickle:

----

#!/usr/bin/env python

import platform
import socket
import time
import pickle

CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003
DELAY = 15 # secs


def get_loadavgs():
with open('/proc/loadavg') as f:
return f.read().strip().split()[:3]


def send_msg(message):
payload = pickle.dumps(message)
header = struct.pack("!L", len(payload))
message = header + payload
sock = socket.socket()
sock.connect((CARBON_SERVER, CARBON_PORT))
sock.sendall(message)
sock.close()


if __name__ == '__main__':
node = platform.node().replace('.', '-')
while True:
timestamp = int(time.time())
loadavgs = get_loadavgs()
values = [
('system.%s.loadavg_1min' % node, (timestamp, loadavgs[0])),
('system.%s.loadavg_5min' % node, (timestamp, loadavgs[1])),
('system.%s.loadavg_15min' % node, (timestamp, loadavgs[2])),
]
send_msg(values)
time.sleep(DELAY)


----

Note that I wrote the code inline in the comment box, so there may be some mistakes :)

Jake said...

Hi Corey,

As we've been asked to possibly support IPv6 in the future, one small change I made to the socket creation/connection was to use:


sock = socket.create_connection((CARBON_SERVER, CARBON_PORT))

BTW, I've been reading your blog off and on for years, so thank you!