Munin contributed plugins

# The 'byprojects' family

Those plugins are used to monitor different projects or vhost (i.e. either different log files or uing regular expression as filters) on the same web server.

## munin_byprojects_access

Count the number of hits per projects/vhost.  

![byproject_access](https://www.mantor.org/~northox/misc/munin-plugins/nginx_byprojects_access1-month.png "byproject_access")

## munin_byprojects_bandwidth

Count the total bandwidth used by each projects/vhost. [Logtail] (https://www.fourmilab.ch/webtools/logtail/) is required.

![byproject_bandwidth](https://www.mantor.org/~northox/misc/munin-plugins/apache_byprojects_bandwidth-month.png "byproject_bandwidth")

## munin_byprojects_inout_bandwidth

Counts the in/out bandwidth used by each projects/vhost. [Logtail] (https://www.fourmilab.ch/webtools/logtail/) is required.

![byproject_inout_bandwidth](https://www.mantor.org/~northox/misc/munin-plugins/apache_byprojects_inout_bandwidth-month.png "byproject_inout_bandwidth")

## Installation

Installation is pretty straight forward. First you need to configure the plugin:

Identify the file which will be used by logtail to identify it's position in the log and the path to logtail:

      $statepath = '/usr/local/var/munin/plugin-state'; # directory where logtail will save the state

      $logtail = '/usr/local/bin/logtail';

Multiple logs can be used for the same project/vhost and a regular expression (regex) can be used as a filter:

      %logs = (

      'prod' => ('/home/prod/log/access.log'),

      'test' => (

                  ('/var/log/httpd/access.log', '"[A-Z]+ /test/'),

                  '/home/test/log/access.log'

                )

      );

In the previous example the prod project graph will be using everything in /home/prod/log/access.log. The test project will be using eveything in /home/test/log/access.log and stuff that match '"[A-Z] /test/' in /var/log/httpd/access.log (e.g. "GET /test/).

Then link the file just as any other plugins.

#!/usr/bin/perl -w

#

# byprojects_access

#

# Perl script to monitor access *byprojects* (e.g. vhost) from multiple files and/or regex.

#

# Danny Fullerton <northox@mantor.org> 

# Mantor Organization <www.mantor.org>

# This work is licensed under a Creative Commons Attribution 3.0 Unported License.

#

# You need logtail (https://www.fourmilab.ch/webtools/logtail/)

#

# Log can be gather from multiple sources by simply specifying multiple log filename and/or an array with two

# elements: log filename and a regex.

#   - 'prod' => array('/home/prod/log/access.log'),

#     Prod graph will be using everything in /home/prod/log/access.log

#   - 'test' => array(array('/var/log/httpd/access.log', '"[A-Z]+ /test/'), '/home/test/log/access.log')

#     Test graph will be using eveything in /home/test/log/access.log and stuff that match '"[A-Z] /test/' in 

#     /var/log/httpd/access.log such as '"GET /test/'

$server = 'Apache';

$statepath = '/usr/local/var/munin/plugin-state'; # where logtail will save the state

$logtail = '/usr/local/bin/logtail';

%logs = (

'prod' => ('/home/prod/log/access.log'),

'test' => (

            ('/var/log/httpd/access.log', '"[A-Z]+ /test/'),

            '/home/test/log/access.log'

          )

);

###########

if(defined($ARGV[0])) {

  if ($ARGV[0] eq 'autoconf') {

    print "yes\n";

    exit(0);

  } elsif ($ARGV[0] eq 'config') {

    $order = '';

    while (($client, $files) = each(%logs)) { $order .= $client.' ' }

    print "graph_order $order\n";

    print "graph_title $server access byprojects\n";

    print "graph_total Total\n";

    print "graph_vlabel Access by \${graph_period}\n";

    print "graph_category $server\n";

    print "graph_info This graph show $server access by various projects.\n";

    while (($client, $files) = each(%logs)) {

      print $client.".label $client\n";

      print $client.".type DERIVE\n";

      print $client.".min 0\n";

    }

    exit(0);

  }

}

while (($client, $files) = each(%logs)) {

  $x = $i = 0;

  foreach $file ($files) {

    $regex = '';

    $state = $statepath.'/'.$client.$x.'_access.state';

    if(ref($file) eq 'ARRAY') { ($file, $regex) = @file }

    open(my $lt, "$logtail -f $file -o $state |") or die "Can't open $logtail : $!";

    while (<$lt>) {

      $buf = $_;

      if($buf eq '') { next }

      if(!defined($regex) || $buf =~ m/$regex/) {

        $i++;

      }

    }

    close($lt);

    $x++;

  }

  print $client.".value $i\n";

}

#!/usr/bin/perl -w

#

# byprojects_bandwidth

#

# Perl script to monitor total bandwidth *byprojects* (e.g. vhost) from multiple files and/or regex.

#

# Danny Fullerton <northox@mantor.org> 

# Mantor Organization <www.mantor.org>

#

# You need logtail (https://www.fourmilab.ch/webtools/logtail/) and

#

# mod_logio apache module (https://httpd.apache.org/docs/2.0/mod/mod_logio.html).

# Your logformat should look like this "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O"

# where %I is input and %O is output.

#

# Log can be gather from multiple sources by simply specifying multiple log filename and/or an array with two

# elements: log filename and a regex.

#   - 'prod' => array('/home/prod/log/access.log'),

#     Prod graph will be using everything in /home/prod/log/access.log

#   - 'test' => array(array('/var/log/httpd/access.log', '"[A-Z]+ /test/'), '/home/test/log/access.log')

#     Test graph will be using eveything in /home/test/log/access.log and stuff that match '"[A-Z] /test/' in 

#     /var/log/httpd/access.log such as '"GET /test/'

$statepath = '/usr/local/var/munin/plugin-state'; # directory where logtail will save the state

$logtail = '/usr/local/bin/logtail';

$server = 'Apache';

%logs = (

'prod' => ('/home/prod/log/access.log'),

'test' => (

            ('/var/log/httpd/access.log', '"[A-Z]+ /test/'),

            '/home/test/log/access.log'

          )

);

###########

if(defined($ARGV[0])) {

  if ($ARGV[0] eq 'autoconf') {

    print "yes\n";

    exit(0);

  } elsif ($ARGV[0] eq 'config') {

    $order = '';

    while (($client, $files) = each(%logs)) { $order .= $client.' ' }

    print "graph_order $order\n";

    print "graph_title $server total bandwidth byprojects\n";

    print "graph_total Total\n";

    print "graph_vlabel Bits\n";

    print "graph_category $server\n";

    print "graph_info This graph show $server total bandwidth used by various projects.\n";

    while (($client, $files) = each(%logs)) {

      print $client.".label $client\n";

      print $client.".type GAUGE\n";

    }

    exit(0);

  }

}

while (($client, $files) = each(%logs)) {

  $i = $o = $x = 0;

  foreach $file ($files) {

    $regex = '';

    $state = $statepath.'/'.$client.$x.'_totalbandwidth.state';

    if(ref($file) eq 'ARRAY') { ($file, $regex) = @file }

    open(my $lt, "$logtail -f $file -o $state |") or die "Can't open $logtail : $!";

    while (<$lt>) {

      $buf = $_;

      if($buf eq '') { next }

      if(!defined($regex) || $buf =~ m/$regex/) {

        if($buf =~ m/(\d+) (\d+)$/) {

          $i += $1;

          $o += $2;

        }

      }

    }

    close($lt);

    $x++;

  }

  $x = $i + $o;

  print $client.".value $x\n";

}

#!/usr/bin/perl -w

#

# byprojects_inout_bandwidth

#

# Perl script to monitor in/out bandwidth *byprojects* (e.g. vhost) from multiple files and/or regex.

#

# Danny Fullerton <northox@mantor.org> 

# Mantor Organization <www.mantor.org>

#

# You need logtail (https://www.fourmilab.ch/webtools/logtail/) and 

#

# Apache:

# mod_logio apache module (https://httpd.apache.org/docs/2.0/mod/mod_logio.html).

# Your logformat should look like this "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O"

# where %I is input and %O is output.

#

# Log can be gather from multiple sources by simply specifying multiple log filename and/or an array with two

# elements: log filename and a regex.

#   - 'prod' => array('/home/prod/log/access.log'),

#     Prod graph will be using everything in /home/prod/log/access.log

#   - 'test' => array(array('/var/log/httpd/access.log', '"[A-Z]+ /test/'), '/home/test/log/access.log')

#     Test graph will be using eveything in /home/test/log/access.log and stuff that match '"[A-Z] /test/' in 

#     /var/log/httpd/access.log such as '"GET /test/'

$server = 'Apache';

$statepath = '/usr/local/var/munin/plugin-state'; # directory where logtail will save the state

$logtail = '/usr/local/bin/logtail';

%logs = (

'prod' => ('/home/prod/log/access.log'),

'test' => (

            ('/var/log/httpd/access.log', '"[A-Z]+ /test/'),

            '/home/test/log/access.log'

          )

);

###########

if(defined($ARGV[0])) {

  if ($ARGV[0] eq 'autoconf') {

    print "yes\n";

    exit(0);

  } elsif ($ARGV[0] eq 'config') {

    print "graph_title $server in/out bandwidth byprojects\n";

    print "graph_args --base 1000\n";

    print "graph_vlabel bits per \${graph_period} in (-) / out (+)\n";

    print "graph_category $server\n";

    print "graph_info This graph show $server in/out bandwidth used by various projects.\n";

    while (($client, $files) = each(%logs)) {

      print "i".$client.".label $client\n";

      print "i".$client.".type GAUGE\n";

      print "i".$client.".graph no\n";

      print "i".$client.".cdef i".$client.",8,*\n";

      print "o".$client.".label $client\n";

      print "o".$client.".type GAUGE\n";

      print "o".$client.".negative i".$client."\n";

      print "o".$client.".cdef o".$client.",8,*\n";

    }

    exit(0);

  }

}

while (($client, $files) = each(%logs)) {

  $i = $o = $x = 0;

  foreach $file ($files) {

    $regex = '';

    $state = $statepath.'/'.$client.$x.'_inoutbandwidth.state';

    if(ref($file) eq 'ARRAY') { ($file, $regex) = @file }

    open(my $lt, "$logtail -f $file -o $state |") or die "Can't open $logtail : $!";

    while (<$lt>) {

      $buf = $_;

      if($buf eq '') { next }

      if(!defined($regex) || $buf =~ m/$regex/) {

        if($buf =~ m/(\d+) (\d+)$/) {

          $i += $1;

          $o += $2;

        }

      }

    }

    close($lt);

    $x++;

  }

  print "i".$client.".value $i\n";

  print "o".$client.".value $o\n";

}

# The 'byprojects' family

Those plugins are used to monitor different projects or vhost (i.e. either different log files or uing regular expression as filters) on the same web server.

## munin_byprojects_access

Count the number of hits per projects/vhost.  

![byproject_access](https://www.mantor.org/~northox/misc/munin-plugins/nginx_byprojects_access1-month.png "byproject_access")

## munin_byprojects_bandwidth

Count the total bandwidth used by each projects/vhost. [Logtail] (https://www.fourmilab.ch/webtools/logtail/) is required.

![byproject_bandwidth](https://www.mantor.org/~northox/misc/munin-plugins/apache_byprojects_bandwidth-month.png "byproject_bandwidth")

## munin_byprojects_inout_bandwidth

Counts the in/out bandwidth used by each projects/vhost. [Logtail] (https://www.fourmilab.ch/webtools/logtail/) is required.

![byproject_inout_bandwidth](https://www.mantor.org/~northox/misc/munin-plugins/apache_byprojects_inout_bandwidth-month.png "byproject_inout_bandwidth")

## Installation

Installation is pretty straight forward. First you need to configure the plugin:

Identify the file which will be used by logtail to identify it's position in the log and the path to logtail:

      $statepath = '/usr/local/var/munin/plugin-state'; # directory where logtail will save the state

      $logtail = '/usr/local/bin/logtail';

Multiple logs can be used for the same project/vhost and a regular expression (regex) can be used as a filter:

      %logs = (

      'prod' => ('/home/prod/log/access.log'),

      'test' => (

                  ('/var/log/httpd/access.log', '"[A-Z]+ /test/'),

                  '/home/test/log/access.log'

                )

      );

In the previous example the prod project graph will be using everything in /home/prod/log/access.log. The test project will be using eveything in /home/test/log/access.log and stuff that match '"[A-Z] /test/' in /var/log/httpd/access.log (e.g. "GET /test/).

Then link the file just as any other plugins.

#!/usr/bin/perl -w

#

# byprojects_access

#

# Perl script to monitor access *byprojects* (e.g. vhost) from multiple files and/or regex.

#

# Danny Fullerton <northox@mantor.org> 

# Mantor Organization <www.mantor.org>

# This work is licensed under a Creative Commons Attribution 3.0 Unported License.

#

# You need logtail (https://www.fourmilab.ch/webtools/logtail/)

#

# Log can be gather from multiple sources by simply specifying multiple log filename and/or an array with two

# elements: log filename and a regex.

#   - 'prod' => array('/home/prod/log/access.log'),

#     Prod graph will be using everything in /home/prod/log/access.log

#   - 'test' => array(array('/var/log/httpd/access.log', '"[A-Z]+ /test/'), '/home/test/log/access.log')

#     Test graph will be using eveything in /home/test/log/access.log and stuff that match '"[A-Z] /test/' in 

#     /var/log/httpd/access.log such as '"GET /test/'

$server = 'Nginx';

$statepath = '/usr/local/var/munin/plugin-state'; # where logtail will save the state

$logtail = '/usr/local/bin/logtail';

%logs = (

'prod' => ('/home/prod/log/access.log'),

'test' => (

            ('/var/log/httpd/access.log', '"[A-Z]+ /test/'),

            '/home/test/log/access.log'

          )

);

###########

if(defined($ARGV[0])) {

  if ($ARGV[0] eq 'autoconf') {

    print "yes\n";

    exit(0);

  } elsif ($ARGV[0] eq 'config') {

    $order = '';

    while (($client, $files) = each(%logs)) { $order .= $client.' ' }

    print "graph_order $order\n";

    print "graph_title $server access byprojects\n";

    print "graph_total Total\n";

    print "graph_vlabel Access by \${graph_period}\n";

    print "graph_category $server\n";

    print "graph_info This graph show $server access by various projects.\n";

    while (($client, $files) = each(%logs)) {

      print $client.".label $client\n";

      print $client.".type DERIVE\n";

      print $client.".min 0\n";

    }

    exit(0);

  }

}

while (($client, $files) = each(%logs)) {

  $x = $i = 0;

  foreach $file ($files) {

    $regex = '';

    $state = $statepath.'/'.$client.$x.'_access.state';

    if(ref($file) eq 'ARRAY') { ($file, $regex) = @file }

    open(my $lt, "$logtail -f $file -o $state |") or die "Can't open $logtail : $!";

    while (<$lt>) {

      $buf = $_;

      if($buf eq '') { next }

      if(!defined($regex) || $buf =~ m/$regex/) {

        $i++;

      }

    }

    close($lt);

    $x++;

  }

  print $client.".value $i\n";

}

#!/usr/bin/perl -w

#

# byprojects_bandwidth

#

# Perl script to monitor total bandwidth *byprojects* (e.g. vhost) from multiple files and/or regex.

#

# Danny Fullerton <northox@mantor.org> 

# Mantor Organization <www.mantor.org>

#

# You need logtail (https://www.fourmilab.ch/webtools/logtail/) and

#

# Your nginx configuration should look like this (i.e. $request_length $body_bytes_sent at the end):

#   log_format main  '$remote_addr - $remote_user $time_local "$request" '

#                    '$status $body_bytes_sent "$http_referer" '

#                    '"$http_user_agent" $request_length $body_bytes_sent';

#

# Log can be gather from multiple sources by simply specifying multiple log filename and/or an array with two

# elements: log filename and a regex.

#   - 'prod' => array('/home/prod/log/access.log'),

#     Prod graph will be using everything in /home/prod/log/access.log

#   - 'test' => array(array('/var/log/httpd/access.log', '"[A-Z]+ /test/'), '/home/test/log/access.log')

#     Test graph will be using eveything in /home/test/log/access.log and stuff that match '"[A-Z] /test/' in 

#     /var/log/httpd/access.log such as '"GET /test/'

$statepath = '/usr/local/var/munin/plugin-state'; # directory where logtail will save the state

$logtail = '/usr/local/bin/logtail';

$server = 'Nginx';

%logs = (

'prod' => ('/home/prod/log/access.log'),

'test' => (

            ('/var/log/httpd/access.log', '"[A-Z]+ /test/'),

            '/home/test/log/access.log'

          )

);

###########

if(defined($ARGV[0])) {

  if ($ARGV[0] eq 'autoconf') {

    print "yes\n";

    exit(0);

  } elsif ($ARGV[0] eq 'config') {

    $order = '';

    while (($client, $files) = each(%logs)) { $order .= $client.' ' }

    print "graph_order $order\n";

    print "graph_title $server total bandwidth byprojects\n";

    print "graph_total Total\n";

    print "graph_vlabel Bits\n";

    print "graph_category $server\n";

    print "graph_info This graph show $server total bandwidth used by various projects.\n";

    while (($client, $files) = each(%logs)) {

      print $client.".label $client\n";

      print $client.".type GAUGE\n";

    }

    exit(0);

  }

}

while (($client, $files) = each(%logs)) {

  $i = $o = $x = 0;

  foreach $file ($files) {

    $regex = '';

    $state = $statepath.'/'.$client.$x.'_totalbandwidth.state';

    if(ref($file) eq 'ARRAY') { ($file, $regex) = @file }

    open(my $lt, "$logtail -f $file -o $state |") or die "Can't open $logtail : $!";

    while (<$lt>) {

      $buf = $_;

      if($buf eq '') { next }

      if(!defined($regex) || $buf =~ m/$regex/) {

        if($buf =~ m/(\d+) (\d+)$/) {

          $i += $1;

          $o += $2;

        }

      }

    }

    close($lt);

    $x++;

  }

  $x = $i + $o;

  print $client.".value $x\n";

}

#!/usr/bin/perl -w

#

# byprojects_inout_bandwidth

#

# Perl script to monitor in/out bandwidth *byprojects* (e.g. vhost) from multiple files and/or regex.

#

# Danny Fullerton <northox@mantor.org> 

# Mantor Organization <www.mantor.org>

#

# You need logtail (https://www.fourmilab.ch/webtools/logtail/) and 

#

# Your nginx configuration should look like this (i.e. $request_length $body_bytes_sent at the end):

#   log_format main  '$remote_addr - $remote_user $time_local "$request" '

#                    '$status $body_bytes_sent "$http_referer" '

#                    '"$http_user_agent" $request_length $body_bytes_sent';

#

# Log can be gather from multiple sources by simply specifying multiple log filename and/or an array with two

# elements: log filename and a regex.

#   - 'prod' => array('/home/prod/log/access.log'),

#     Prod graph will be using everything in /home/prod/log/access.log

#   - 'test' => array(array('/var/log/httpd/access.log', '"[A-Z]+ /test/'), '/home/test/log/access.log')

#     Test graph will be using eveything in /home/test/log/access.log and stuff that match '"[A-Z] /test/' in 

#     /var/log/httpd/access.log such as '"GET /test/'

$server = 'Nginx';

$statepath = '/usr/local/var/munin/plugin-state'; # directory where logtail will save the state

$logtail = '/usr/local/bin/logtail';

%logs = (

'prod' => ('/home/prod/log/access.log'),

'test' => (

            ('/var/log/httpd/access.log', '"[A-Z]+ /test/'),

            '/home/test/log/access.log'

          )

);

###########

if(defined($ARGV[0])) {

  if ($ARGV[0] eq 'autoconf') {

    print "yes\n";

    exit(0);

  } elsif ($ARGV[0] eq 'config') {

    print "graph_title $server in/out bandwidth byprojects\n";

    print "graph_args --base 1000\n";

    print "graph_vlabel bits per \${graph_period} in (-) / out (+)\n";

    print "graph_category $server\n";

    print "graph_info This graph show $server in/out bandwidth used by various projects.\n";

    while (($client, $files) = each(%logs)) {

      print "i".$client.".label $client\n";

      print "i".$client.".type GAUGE\n";

      print "i".$client.".graph no\n";

      print "i".$client.".cdef i".$client.",8,*\n";

      print "o".$client.".label $client\n";

      print "o".$client.".type GAUGE\n";

      print "o".$client.".negative i".$client."\n";

      print "o".$client.".cdef o".$client.",8,*\n";

    }

    exit(0);

  }

}

while (($client, $files) = each(%logs)) {

  $i = $o = $x = 0;

  foreach $file ($files) {

    $regex = '';

    $state = $statepath.'/'.$client.$x.'_inoutbandwidth.state';

    if(ref($file) eq 'ARRAY') { ($file, $regex) = @file }

    open(my $lt, "$logtail -f $file -o $state |") or die "Can't open $logtail : $!";

    while (<$lt>) {

      $buf = $_;

      if($buf eq '') { next }

      if(!defined($regex) || $buf =~ m/$regex/) {

        if($buf =~ m/(\d+) (\d+)$/) {

          $i += $1;

          $o += $2;

        }

      }

    }

    close($lt);

    $x++;

  }

  print "i".$client.".value $i\n";

  print "o".$client.".value $o\n";

}