There remains some work to be done in order to automate the addition of servers to the pool, but can be achieved by a methodology similar to the one described in the paper Dynamically Scaling Web Applications in EC2 by Ramesh Ramajani (http://www.techmasala.com/2009/04/06/dynamically-scale-web-applications-in-amazon-ec2/). I believe this creates a cleaner way to manage the servers in the resource pool.
Mon Configuration
The following is a basic config file for a mon server using the NGINX alert that I have created. The name of the hostgroup should be the same as the name of the upstream server pool in your NGINX config.
/etc/mon/mon.cf:
#
# The mon.cf file
#
#
# global options
#
basedir = /usr/lib/mon
cfbasedir = /etc/mon
alertdir = /usr/lib/mon/alert.d
mondir = /usr/lib/mon/mon.d
dtlogfile = /var/log/mon.log
dtlogging = yes
histlength = 100
historicfile = /var/log/mon.hist
maxprocs = 20
startupalerts_on_reset = yes
syslog_facility = daemon
#
# group definitions (hostnames or IP addresses)
#
hostgroup backend 172.40.0.1 172.40.0.2
#
# Backend servers
#
watch backend
service http
interval 2s
monitor http.monitor
period wd {Sun-Sat}
alert nginx.alert
upalert nginx.alert
startupalert nginx.alert
NGINX Config
This script expects the configuration file to have in it an upstream definition similar to:
http {
upstream backend {
ip_hash;
server 172.40.0.1:80;
server 172.40.0.2:80;
}
server {
listen 80;
server_name www.domain.com;
location / {
proxy_pass http://backend;
}
}
}
The most important things to note with respect to this are that there must be a newline for each server as well as the closing brace in the upstream definition and it must be using the ip_hash module for the down parameter to be supported.
NGINX Alert for Mon
This is an NGINX alert for mon that will find and replace or add a line for a server configuration to an existing NGINX config file.
In addition to the mon paramaters as specified in the mon man page(http://mon.wiki.kernel.org/index.php/Mon_Manual) it supports some NGINX specific config options:
- -p NGINX pid file
- -c NGINX conf file
- -o additional parameters for NGINX upstream module (http://wiki.nginx.org/NginxHttpUpstreamModule)
- -g identifier for the upstream group of the server (required)
/usr/lib/mon/alert.d/nginx.alert
#!/usr/bin/perl -w
#
# a mon alert for nginx (http://wiki.nginx.org/)
#
# James Litton, jlitton@eng.compendiumblogware.com
#
# $Id: nginx.alert, v 1.0 2010/01/18 14:02:26 jlitton Exp $
#
# Copyright (C) 2010, Compendium Blogware
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
use strict;
use Getopt::Std;
use Tie::File;
my %opt;
getopts("s:g:h:t:l:u:T:O:c:p:o", %opt);
#
# the first line is summary information, adequate to send to a pager
# or email subject line
#
#
# the following lines normally contain more detailed information,
# but this is monitor-dependent
#
# see the "Alert Programs" section in mon(1) for an explanation
# of the options that are passed to the monitor script.
#
my $summary=<STDIN>;
if ( defined $summary ) { chomp $summary; };
my $nginx_conf='/etc/nginx/nginx.conf';
my $nginx_pid='/var/run/nginx.pid';
my $nginx_opts="";
my $port=80;
my $groupname;
if ( defined $opt{'c'} ) { $nginx_conf=$opt{'c'}; };
if ( defined $opt{'p'} ) { $nginx_pid=$opt{'p'}; };
if ( defined $opt{'o'} ) { $nginx_opts="$nginx_opts $opt{'o'}"; };
if ( defined $opt{'g'} ) { $groupname=$opt{'g'}; } else { die "No Groupname" };
my ($t, $wday, $mon, $day, $tm);
if (defined $opt{'t'})
{
$t = localtime($opt{'t'});
($wday,$mon,$day,$tm) = split (/s+/, $t);
}
my @alertservers;
my @allservers = split(/s+/, $opt{'h'});
if ( not $ENV{'MON_ALERTTYPE'} eq "startup" )
{
@alertservers = split(/s+/,$ENV{'MON_LAST_SUMMARY'});
}
my ($server, $oldline, $newline);
my $changed=0;
my $in_group=0;
my $lineno=-1;
my (@config, @deletelines);
tie @config, 'Tie::File', "$nginx_conf" or
die "Could not open nginx config file: $nginx_conf for write";
for (@config)
{
++$lineno;
if ( $in_group )
{
$oldline = $_;
if (m/^s*{/)
{
next;
} elsif (m/^s*ip_hash/) {
next;
} elsif (m/^s*#/) {
next;
} elsif (m/^s*}/) {
foreach $server (@allservers)
{
if ($ENV{'MON_ALERTTYPE'} eq "failure" and
grep {$_ eq $server} @alertservers)
{
$newline .= "server $server:$port $nginx_opts down\n";
}else {
$newline .= "server $server:$port $nginx_opts\n";
}
}
$newline .= $oldline;
$_ = $newline;
$changed=1;
last;
} else {
my $found=0;
for my $i (0..$#allservers)
{
$server = pop(@allservers);
$_ = $oldline;
if (m/server.*$server.*/)
{
if ($ENV{'MON_ALERTTYPE'} eq "failure" and
grep {$_ eq $server} @alertservers)
{
$_ = "server $server:$port $nginx_opts down\n;
}
$found=1;
last;
} else {
unshift(@allservers, $server);
}
}
# This line does not match a known required line or
# any of the known servers, remove it
if ( not $found )
{
push ( @deletelines, $lineno );
}
}
}
if ( m/upstream $groupname/ )
{
$in_group=1;
}
}
for my $line (reverse sort @deletelines)
{
splice @config, $line, 1;
}
untie @config;
if ( $changed )
{
my $pid = `cat $nginx_pid`;
kill 1, $pid or die "Failed to restart NGINX";
}
Usage
Create
Adding a server to the resource pool can be done simply by adding it to the hostgroup in the mon config and sending a SIGHUP to mon
Read
In order to determine the current state of the configuration, you can either check your current NGINX config or you can set up mon alerts for status changes: /etc/mon/mon.cf:
#
# The mon.cf file
#
#
# global options
#
basedir = /usr/lib/mon
cfbasedir = /etc/mon
alertdir = /usr/lib/mon/alert.d
mondir = /usr/lib/mon/mon.d
dtlogfile = /var/log/mon.log
dtlogging = yes
histlength = 100
historicfile = /var/log/mon.hist
maxprocs = 20
startupalerts_on_reset = yes
syslog_facility = daemon
#
# group definitions (hostnames or IP addresses)
#
hostgroup backend 172.40.0.1 172.40.0.2
#
# Backend servers
#
watch backend
service http
interval 2s
monitor http.monitor
period wd {Sun-Sat}
alert nginx.alert
upalert nginx.alert
startupalert nginx.alert
alert mail.alert jlitton@eng.compendiumblogware.com
upalert mail.alert jlitton@eng.compendiumblogware.com
alertafter 1
alertevery 10m strict
Additionally, you should consider the stub status module for NGINX(http://wiki.nginx.org/NginxHttpStubStatusModule)
Update
That's the point of the nginx.alert script
Delete
Removing a server from the resource pool can be done simply by removing it to the hostgroup in the mon config and sending a SIGHUP to mon
Additional Notes
In the setup that I use, my nginx.conf file is managed by puppet so I take advantage of the NGINX include syntax similar to:
/etc/nginx/nginx.conf
include /etc/nginx/backend.conf
include /etc/nginx/images.conf
server {
listen 80;
server_name www.domain.com;
location / {
proxy_pass http://backend;
}
location /images/ {
proxy_pass http://images;
}
}
/etc/nginx/backend.conf
upstream backend {
ip_hash;
server 172.40.0.1:80;
server 172.40.0.2:80;
}
/etc/nginx/images.conf
upstream images {
ip_hash;
server 172.40.0.1:80;
server 172.40.0.2:80;
}
/etc/mon/mon.cf:
#
# The mon.cf file
#
#
# global options
#
basedir = /usr/lib/mon
cfbasedir = /etc/mon
alertdir = /usr/lib/mon/alert.d
mondir = /usr/lib/mon/mon.d
dtlogfile = /var/log/mon.log
dtlogging = yes
histlength = 100
historicfile = /var/log/mon.hist
maxprocs = 20
startupalerts_on_reset = yes
syslog_facility = daemon
#
# group definitions (hostnames or IP addresses)
#
hostgroup backend 172.40.0.1 172.40.0.2
hostgroup images img1.example.com img2.example.com
#
# Backend servers
#
watch backend
service http
interval 2s
monitor http.monitor
period wd {Sun-Sat}
alert nginx.alert -c /etc/nginx/backend.conf
upalert nginx.alert -c /etc/nginx/backend.conf
startup nginx.alert -c /etc/nginx/backend.conf
alert mail.alert jlitton@eng.compendiumblogware.com
upalert mail.alert jlitton@eng.compendiumblogware.com
alertafter 1
alertevery 10m strict
#
# Images servers
#
watch images
service http
interval 2s
monitor http.monitor
period wd {Sun-Sat}
alert nginx.alert -c /etc/nginx/images.conf
upalert nginx.alert -c /etc/nginx/images.conf
startupalert nginx.alert -c /etc/nginx/images.conf
alert mail.alert jlitton@eng.compendiumblogware.com
upalert mail.alert jlitton@eng.compendiumblogware.com
alertafter 1
alertevery 10m strict
Comments for Using NGINX as a Reverse Proxy/Load Balancer with Mon
Leave a comment