I’m currently at the tail end of collecting metrics for the current company I work for. We are using a combination of.

I built a python script, that leverages the awesome tool logtail in order to collect http status codes, without consuming too much cpu/ram.

I have tested this script against nginx logs and against 2 different formats (txt and json).

Download Script

Basic Logging

"GET /test/foo HTTP/1.0" 200 76288 "-" "Ruby"

Json Logging

I printed the json on multiple lines, but in the log, the json is all in 1 line.

{
    "timestamp": "2014-09-26T08:45:35-04:00",
    "fields": {
        "remote_addr": "127.0.0.1",
        "remote_user": "-",
        "body_bytes_sent": "4823",
        "request_time": "0.127",
        "status": "200",
        "request": "POST /rubyamf_gateway/ HTTP/1.0",
        "request_method": "POST",
        "http_referrer": "https://foo.bar.net/test/test",
        "http_user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36"
    }
}

The only dependency of this script is logtail which you can install easily on any *nix system.

  • (Debian/Ubuntu) apt-get install logtail
  • (CentOS/RedHat) yum install logtail

If you do not have logtail installed you will get the following error

{9:45}~/Linuxdynasty/scripts/sensu/metrics:master ✓ python nginx-status-code-metrics.py -h
Please install logtail

The following options are available

{9:45}~/Linuxdynasty/scripts/sensu/metrics:master ✓ ➭ python nginx-status-code-metrics.py -h
Options:
  -h, --help            show this help message and exit
  -d DIRECTORY, --directory=DIRECTORY
                        The directory where the access logs exist
  -f FILE, --file=FILE  name of the access log
  -t FILE_FORMAT, --file_format=FILE_FORMAT
                        json or txt
  -s SCHEME, --scheme=SCHEME
                        the metric naming scheme

This is how you would run the script

{9:45}~/Linuxdynasty/scripts/sensu/metrics:master ✓ ➭ python nginx-status-code-metrics.py -d /var/log/nginx -f access.json -t json
webserver01.codes.200 1080 1411736116.44
webserver01.codes.304 2 1411736116.44
webserver01.codes.404 19 1411736116.44

The end result will look like this. Grafana Screenshot

I just finished migrating from wordpress, and I love having the simplicity of just managing static pages and being able to edit them right from vim. I will be updating my old posts little by little, so if a download link is not valid than just check my GitHub Page

I have been so focused on vFense, that I have been negligent of this blog. vFense has come a long way from it’s inception a little over 20 months ago. We are still in the Beta, but we are continuously making improvements. As of right now, I’m currently the only active developer working on the next major release 0.8.0, which is a complete server rewrite. Examples of what is comming…

  • All functions, methods, and classes will be fully documented.
  • Vulnerability correlation for the following operating systems…
    • RedHat and clones
    • Ubuntu and clones
    • Windows
    • CVE/NVD
  • Moved to using APScheduler v3.1.0 for managing scheduled jobs
  • Logging of all administrative tasks into RethinkDB.
  • Remote command line tool for making API calls to vFense (Think of chef’s knife command)
  • Token based authentication for the agents
  • Separated WEB authentication from Agent authentication.

My team was assigned to create a redis slave status check to be ran under Zenoss. So while they are creating that check, I decided to google for redis checks written in python that work under Nagios and or Zenoss and none of the checks i found did exactly what I wanted.

So I decided to write my own check, that will grab every piece of data that the redis info() command was able to retrieve. Since this info is all in a python dictionary, I was able to get all the stats that were labeled as type int or as type float, which made my job that much easier.

So 1st I had to install 2 Redis instances on my local Ubuntu server at home. Now that i completed that, I had to make sure the slave was syncing off the master server. I used this link “http://redis.io/topics/replication” to help me configure redis replication. Now that all that stuff is out of the way, I wrote an easy_peasy python script to connect to redis and grab all of the performance stats. As well as verify if the instance is a master or a slave instance. If it is a slave instance, than it also verifies if it is syncing to the master or not. The script is using Redis-py that was installed using PIP.

Here is the script I wrote

godssoldier:python asanabria$ python ld_check_redis.py -d 127.0.0.1 -p 't35t_r3d15' -n 6379
OK Master Redis Server 127.0.0.1 is Running 2.4.4 | pubsub_channels=0 bgrewriteaof_in_progress=0
connected_slaves=1 uptime_in_days=0 lru_clock=1099413 last_save_time=1332199497 redis_git_sha1=0
loading=0 connected_clients=1 keyspace_misses=4 used_memory=939792 vm_enabled=0
used_cpu_user_children=0.000000 used_memory_peak=939776 total_commands_processed=10
latest_fork_usec=211 used_memory_rss=1286144 total_connections_received=8 pubsub_patterns=0
aof_enabled=0 used_cpu_sys=0.130000 used_cpu_sys_children=0.000000 blocked_clients=0
used_cpu_user=0.120000 client_biggest_input_buf=0 db0_keys=2 db0_expires=0 arch_bits=64
mem_fragmentation_ratio=1.370000 expired_keys=0 evicted_keys=0 bgsave_in_progress=0
client_longest_output_list=0 process_id=22007 uptime_in_seconds=401 changes_since_last_save=2
redis_git_dirty=0 keyspace_hits=1

godssoldier:python asanabria$ python ld_check_redis.py -d 127.0.0.1 -p 't35t_r3d15' -n 6390
OK Master 127.0.0.1 is up and Slave 127.0.0.1 is in sync | pubsub_channels=0 bgrewriteaof_in_progress=0
connected_slaves=0 uptime_in_days=0 lru_clock=1099413 last_save_time=1332199497 redis_git_sha1=0
loading=0 connected_clients=2 keyspace_misses=4 used_memory=939872 master_last_io_seconds_ago=8
vm_enabled=0 used_cpu_user_children=0.000000 used_memory_peak=931248 total_commands_processed=44
latest_fork_usec=0 used_memory_rss=1277952 total_connections_received=2 pubsub_patterns=0 aof_enabled=0
used_cpu_sys=0.130000 used_cpu_sys_children=0.000000 blocked_clients=0 used_cpu_user=0.070000
master_port=6379 client_biggest_input_buf=0 db0_keys=2 db0_expires=0 arch_bits=64
mem_fragmentation_ratio=1.360000 expired_keys=0 evicted_keys=0 bgsave_in_progress=0 client_longest_output_list=0
master_sync_in_progress=0 process_id=22010 uptime_in_seconds=398 changes_since_last_save=2 redis_git_dirty=0
keyspace_hits=0

CRITICAL Master 127.0.0.1 is down and Slave 127.0.0.1 is out of sync |
pubsub_channels=0 bgrewriteaof_in_progress=0 connected_slaves=0 master_link_down_since_seconds=1332199283
uptime_in_days=0 lru_clock=1099352 last_save_time=1332198992 redis_git_sha1=0 loading=0 connected_clients=1
keyspace_misses=0 used_memory=931040 master_last_io_seconds_ago=-1 vm_enabled=0 used_cpu_user_children=0.000000
used_memory_peak=931040 total_commands_processed=5 latest_fork_usec=0 used_memory_rss=1261568
total_connections_received=6 pubsub_patterns=0 aof_enabled=0 used_cpu_sys=0.080000
used_cpu_sys_children=0.000000 blocked_clients=0 used_cpu_user=0.030000 master_port=6379
client_biggest_input_buf=0 arch_bits=64 mem_fragmentation_ratio=1.360000 expired_keys=0 evicted_keys=0
 bgsave_in_progress=0 client_longest_output_list=0 master_sync_in_progress=0 process_id=21887
uptime_in_seconds=290 changes_since_last_save=0 redis_git_dirty=0 keyspace_hits=0

Since I do currently work for a streaming company, that would imply that we should have some type of monitoring for our RTSP streams ;-). You will 1st need to get the openRTSP command.

  • You will need to download the openRTSP command from http://www.live555.com/openRTSP/
  • Or if you are running Zenoss on top of Debian you can just run a apt-get install livemedia-utils

Once you do that, all you have to do next is to download my script and have a valid server to point at and a path to test... Example below.. {filelink=24}

./check_rtsp.py -d remote_server -p /iphone/2012 
OK /iphone/2012, test completed successfull against remote_server |status=0

or with stats...

./check_rtsp.py -d remote_server -p /iphone/2012 -s
OK /iphone/2012, test completed successfull against remote_server |status=0 num_packets_received=16 num_packets_lost=0 elapsed_measurement_time=3.000073 kBytes_received_total=16.309000 
measurement_sampling_interval_ms=1000 kbits_per_second_min=29.245485 kbits_per_second_ave=43.489608 
kbits_per_second_max=51.737449 packet_loss_percentage_min=0.000000 packet_loss_percentage_ave=0.000000 
packet_loss_percentage_max=0.000000 inter_packet_gap_ms_min=0.018000 inter_packet_gap_ms_ave=161.139313 
inter_packet_gap_ms_max=901.439000 subsession=video/H264 num_packets_received=61 num_packets_lost=0 
elapsed_measurement_time=3.000073 kBytes_received_total=27.630000 measurement_sampling_interval_ms=1000 
kbits_per_second_min=0.000000 kbits_per_second_ave=73.678207 kbits_per_second_max=122.923442 
packet_loss_percentage_min=0.000000 packet_loss_percentage_ave=0.000000 packet_loss_percentage_max=0.000000 
inter_packet_gap_ms_min=0.009000 inter_packet_gap_ms_ave=24.794672 inter_packet_gap_ms_max=528.923000

So you can trend the different stats that openRTSP provides in Zenoss..