Pggraph

README -- Mar 13, 2008 -- Aurynn Shaw -- ashaw@commandprompt.com

h2. Python Dependencies

Make sure before you begin that you've got the following Python packages installed via apt-get:

* python-imaging

* python-paramiko

* python-psycopg

* python-psycopg2

* python-libreport

Report Lab note: you will need to compile the rl_addon renderPM lib from source (the installed python class is a stub). It is available via SVN at svn co http://www.reportlab.co.uk/svn/public/reportlab/trunk

h2. Pre-Installation Setup & Notes

* Create "pggraph_clients"  database, with "clients" table, for the installer script to INSERT to.

* Create "pggraph" database, install SQL average functions from .sql file in install path.

* The Python Path will need to be configured to point to the _main installation files (the directory you are installing from) so that fonts and common classes are available.

* Do NOT create the desired installation directory before you run the script, it is part of the script's function.

h2. At A Glance Installation

* Use /var/www/pggraph/_main/install.py. Consult this document for specifics.

* symlink busy_dml_report.py into your new install dir.

* Run the forwarder, /var/www/python/_main/forward.py /path/to/your/new/install

* Check if you need to set up a bastion install

*** If you do, ssh to the bastion and run

***** screen

***** inside screen: ssh -L7750:localhost:5432 cmd@the_actual_db_server

***** ctrl-a ctrl-d, detach from screen. It keeps on truckin'.

* Run python /var/www/pggraph/_main/bridge.py /path/to/your/new/install/config.ini

* Wait until it returns. Now run it again.

* check http://www.cmdalert.com/pggraph/your_install/

* Make sure there's graphs.

* Go edit the Apache config, copy and paste the existing format for PG-backed password auth.

*** The file is 001-something in /etc/apache2/sites-enabled

* Using the existing format, dump and add the database users from the client 

   Trac to serve as the auth for pggraph.

* Reload apache.

* Add the cronjob, based on the other cronjobs for pggraph.

* Fini.

h2. Introduction

If youre reading this document, congratulations! Youve managed to 

successfully install the server software for PGGraph! Youve done an amazing

thing, and you ought to be proud of yourself.

This document will be discussing the next step, setting up client systems for

pggraph, the configuration weirdness and gotchas that you need to know.

The specific sections in this document are as follows:

* The Installer

* Server preparation

* Running it the first time

* Cronjob

* Cursing

h2. The Installer

Step one, setting up a client instance.

Given PGGraphs curious evolutionary path, the process of installing a client

was once much more difficult than it is now, and thus, I created the 

Installer.

The installer is run from the source directory of all PGGGraph scripts - in

the case of cmdalert.com, /var/www/pggraph/_main/

The software is executed thusly:

<pre>

  $ python ./install.py

</pre>

It will then ask you a number of questions specifically relating to the 

install process.

For clarity, the installer is reproduced here to give you a basic walkthrough

on what values are expected and a basic concept of how PGGraph works.

For clarity, the || || characters have been added to delineate between my 

comments and what will be shown by the installer.

<pre>

    Installation directory?  || Requires absolute path: This is the exact path to the install. /var/www/pggraph/some_dir. || 

    Relative WWW path?  || Usually pggraph/prefix: pggraph/some_dir. Controls HTML generation. || 

    Full text title?  || Usually customer name: Page title, in the databse || 

    Now for some database configuration..

    Local pggraph database name? [pggraph]  || Don't change this, unless you're putting their data in a new DB || 

    Table prefix?  || some prefix, like aweber or vltlgx or something. To keep tables from colliding. || 

    Local database port? r5432  || It's on 5432. || 

    Local pggraph host? usually localhost [localhost]  || It's always localhost. Except when it isn't. || 

    Local username? usually pggraph [pggraph]  || It's pggraph. || 

    Password? Can be blank [8sm4LA2]  || It's this password. || 

    Remote DB username?  || Username we connect to the remote DB with. cmd, postgres, or other. || 

    Remote DB password?  || As described || 

    Remote DB host? [localhost]  || Almost always localhost. || 

    Local port to bind to remote PG port? r7555  || Always a non-5432 local port. Be sure this is numeric and not "None" as it may default if this is the first run. || 

    Some pggraph settings..

    Shortest display interval, in minutes? r10  || How frequently we're updating the database. || 

    Maximum number of images per page? r10  || Suffers from an off-by-one error, so enter 1 more than you want to display. || 

    Delay before continuing PGGraph startup r7  || Number of seconds after initializing a new tunnel via forward, that the script waits before attempting to connect. || 

    So far, so good. Now for the SSH configuration for the PG tunnel.

    SSH User?  || SSH user name. almost always cmd || 

    SSH Password?  || Deprecated in favour of || 

    Authentication keyfile? [/home/pggraph/.ssh/id_dsa]  || the default admin ssh key. Good almost everywhere. || 

    Remote server?  || remote server IP || 

    Port to SSH to? r22  || Remote port to SSH to || 

    Bind to what port? r5432  || Remote port to bind to - not always 5432, in cases of bastions. || 

    Bind to which remote address? [localhost]  || Again, not always, in case of bastions. || 

    Use a twin tunnel mechanism? Leave blank if you dont know currently buggy:  || Don't use this. It was a prior attempt at making bastions work. || 

        Address for 2nd server?  || Only shows up if you select twin tunnel. So don't. :) || 

</pre>

Any question with a [] following it indicates that pressing enter will give 

you the default value.

Once you've answered all the installer's questions, it will go off and create

the appropriate directories, create symlinks for all the necessary scripts and

libraries, as well as write out the configuration data.

At this point, you're about halfway done.

The first step (as of Mar 17, 2008), is to symlink busy_dml_report.py into the

freshly created directory. It's a recent addition and is not yet supported by

the installer script.

h2. Server Preparation

h3. No Bastion

The next step is, the remote server information needs to be verified. This is 

always going to be in the customers' Trac wiki, and easily found.

To test the tunneling setup:

<pre>

   $ python /var/www/pggraph/forward.py /var/www/pggraph/new_server

   Initializing tunnel..

   $ psql -d template1 -U cmd -h localhost -p <the port you gave it>

</pre>

If you connect, success! You can move on to the next phase in the installation.

If you don't, you can use the error message you got in helping you track down

why. The most common errors I've had are specific to pg_hba settings being

incorrect, and the cmd user not existing.

h3. Bastions

A fact of life, some of our customers don't trust people on the internet. It's

hard to not blame them for that, but it does make our life a lot harder when

it comes to running pggraph for them.

To set up a bastion host for pggraph, you're going to need to do a little bit

of trickery.

The specific process I've followed is:

* Set up forward.py as normal on cmdalert.

* ssh into the bastion server

* Using screen, run

<pre>

       ssh -L7750:localhost:5432 cmd@the_actual_db_server

</pre>

* Detach from screen, leaving the ssh tunnel open.

* Now that you're back on the bastion server, see if you can telnet to port 

   7750. 

* If you can, disconnect and attempt to connect via psql from cmdalert.

*** Retrace your steps in the event that you can't connect.

*** Most error messages are going to be caused by incorrect settings in

     pg_hba.conf.

You will need to check on your bastion-side tunnel from time to time, as 

they are significantly more fragile than the standard pggraph installation,

and require manual intervention to recreate.

h2. Running PGGraph

At this point, you're close to finished the PGGraph installation. The next 

step is to run the data-collecting script and test for proper graph 

generation.

This is fairly easily done, simply

<pre>

python ../_main/bridge.py ./config.ini

</pre>

from the newly installed PGGraph client path.

This will attempt to connect across the tunnel and collect the data that is

used to build the graphs.

At this point, you should open a browser window to the newly created path (as

mentioned in the installer), usually http://cmdalert.com/pggraph/your_client/

The first graph is a good indicator whether or not data is being correctly

collected, and its presence shows that you can move on to the next step.

If it fails to display, first you should re-run the data collector, and test 

the graph again. A lot of the architecture of PGGraph is based around deltas

of dataset - (dataset - 1 time unit), so only a single database entry may 

cause unintended failure.

If that fails, verify that your connection chain is intact. Failing that, feel

free to email me and ask for help.

h2. Cron

In order for PGGraph to continue to collect data about its client systems, the

data collector needs to be run on a regular basis. Thus, comes the cronjob.

Adding the cronjob is simple: as the pggraph user, add a line similar to the 

following to your crontab, and you're done.

<pre>

    */10 * * * * /usr/bin/python /var/www/pggraph/_main/bridge.py /var/www/pggraph/your_install_dir/config.ini

    or

    5,15,25,35,45,55 * * * * /usr/bin/python /var/www/pggraph/_main/bridge.py /var/www/pggraph/your_install_dir/config.ini

</pre>

Write out the crontab, and pggraph is now collecting data on a regular cycle.

h2. Apache

h3. Python

Use CGI Python, not mod_python. Example configuration:

<pre>

<Directory "/var/www/pggraph/*/">

Options +ExecCGI -Indexes -MultiViews +SymLinksIfOwnerMatch

[[DirectoryIndex]]

[[AddHandler]] cgi-script .py

</Directory>

</pre>

h3. Access

As pggraph contains a lot of very detailed information on the structure

of the clients' database, it's generally a bad idea for anyone who can type in

a URL to be able to access it.

From that, we come to require an authentication system to keep fingers and 

eyeballs from where they don't belong.

As we're using a Pg-backed authentication system in Apache, and as it's 

already fully set up, I won't go into any detail regarding installation or

configuration issues.

To set up the directory-specific authentication, you should copy and paste

the following configuration and modify it as needed to fit the new PGGraph

installation.

The file that needs to be altered is /etc/apache2/sites-enabled/default-ssl

Please back up this file prior to modification.

<pre>

<Location /pggraph/your_install>

        [[AuthName]] "Pggraph authentication for: Your Installation"

        [[AuthType]] Basic

        Auth_PG_host localhost

        Auth_PG_port 5432

        Auth_PG_user pggraph

        Auth_PG_pwd 8sm4LA2

        Auth_PG_authoritative on

        Auth_PG_database pggraph_users

        Auth_PG_pwd_table client_users

        Auth_PG_hash_type MD5

        Auth_PG_uid_field user_name

        Auth_PG_pwd_field user_passwd

        Auth_PG_log_table client_access_log

        Auth_PG_log_uname_field login

        Auth_PG_log_date_field date

        Auth_PG_log_uri_field   request

        Auth_PG_log_addrs_field  ip_address

        [[AuthBasicAuthoritative]] Off

        Require valid-user

</Location>

</pre>

Reload Apache via /etc/init.d/apache2 reload, and your new 

pggraph installation is now completed.