Keith Pitt

Process Monitoring with Monit

Thursday, December 16, 2010

I’ve been using Bluepill for a while now, but when it came to setting up a new production server at work last week, I remembered the pain I had in getting it working in the first place.

I knew there were 2 other players in the process monitoring space, God and Monit. I read that God had a pretty bad memory leak a while ago - and to be honest, I didn’t check to see if it had be fixed. I had done a quick 10 minute sprint and researched monit, and I decided I’d try that first. It was surprisngly easy to setup, and I really liked the syntax - simple and to the point.

I monitor 4 services on my production machine, delayed_job, searchd (Sphinx), memcached and nginx. Below is my configuration file with some commentary.

Before we jump into the config, there are a couple of things you need to know:

• monit needs to run as sudo if its to run correctly
• In order to use the monit CLI, you have to have the web server enabled in your monitrc file.
• If things aren’t working like they should, check the monit log. Its defined on a “per monitrc” basis. In my config below, I’ve put it in /var/log/monit.log
• If you want to check the syntax of your monitc: “sudo monit -t” is what you want.
• Running monit from a non sudo user with no password (usefull in deploy scripts), you’ll need to add the following to your sudoers file (accessed via: “sudo visudo”) ”deploy” being the user you want to run monit from:
  
  “deploy  ALL= NOPASSWD: /usr/sbin/monit”

Note: I have only tested this on Ubuntu 10.10, but I’m confident it’ll “just work” on other *unix platforms.

# Check every 10 seconds
set daemon 10

# Log all monit activity to this file
set logfile /var/log/monit.log

# Here I am using sendgrid to send alter emails
set mailserver smtp.sendgrid.net
  username "your@username.com" password "yourpassword"
  with timeout 30 seconds

# Set the email you want to have alerts sent to
set alert your@email.com

# This is where we setup the web interface. The web interface is
# also required for the monit CLI to work. Replace username/password
# with login credentials for the web interface.
set httpd port 2812 and
  allow localhost
  allow username:password

# This is where the magic happens.

# Here we specify the pidfile to monitor, and define start/stop
# commands. Monit's PATH variable when it starts processes is practically
# empty, so your best bet is to reference via its full path.
check process delayed_job
  with pidfile /var/apps/myapp/shared/pids/delayed_job.pid
  start program = "/usr/bin/env PATH=/usr/local/bin:$PATH RAILS_ENV=production /var/apps/myapp/current/script/delayed_job start" as uid deploy and gid deploy
  stop program = "/usr/bin/env PATH=/usr/local/bin:$PATH RAILS_ENV=production /var/apps/myapp/current/script/delayed_job stop" as uid deploy and gid deploy

# Lets monitor searchd, which is used by thinking sphinx.
check process searchd
  with pidfile /var/apps/myapp/shared/pids/searchd.pid
  start program = "/usr/local/bin/searchd --pidfile --config /var/apps/myapp/current/config/production.sphinx.conf" as uid deploy and gid deploy
  stop program = "/usr/local/bin/searchd --stop --pidfile --config /var/apps/myapp/current/config/production.sphinx.conf" as uid deploy and gid deploy

# Memcached aswell.
check process memcached
  with pidfile /var/run/memcached.pid
  start program = "/etc/init.d/memcached start"
  stop program = "/etc/init.d/memcached stop"

# And for shits and giggles, nginx.
check process nginx
  with pidfile /var/run/nginx.pid
  start program = "/etc/init.d/nginx start"
  stop program = "/etc/init.d/nginx stop"