Project

General

Profile

ClientPgGraph2Install » History » Version 6

Aurynn Shaw, 05/15/2008 07:42 AM

1 1 Aurynn Shaw
README -- Mar 13, 2008 -- Aurynn Shaw -- ashaw@commandprompt.com
2
3 5 John Worsley
== Python Dependencies ==
4
5
Make sure before you begin that you've got the following Python packages installed via apt-get:
6
7
 * python-imaging
8
 * python-paramiko
9
 * python-psycopg
10
 * python-psycopg2
11
 * python-libreport
12
13
Report Lab note: you will need to compile the rl_addon renderPM lib from source (the installed python class is a stub). It is available via SVN at svn co http://www.reportlab.co.uk/svn/public/reportlab/trunk
14
15
== Pre-Installation Setup & Notes ==
16
 * Create "pggraph_clients"  database, with "clients" table, for the installer script to INSERT to.
17
 * Create "pggraph" database, install SQL average functions from .sql file in install path.
18
 * The Python Path will need to be configured to point to the _main installation files (the directory you are installing from) so that fonts and common classes are available.
19
 * Do NOT create the desired installation directory before you run the script, it is part of the script's function.
20
21 1 Aurynn Shaw
== At A Glance Installation ==
22
23 6 Aurynn Shaw
 * Use /var/www/pggraph/_main/install.py. Consult this document for specifics.
24 1 Aurynn Shaw
 * symlink busy_dml_report.py into your new install dir.
25
 * Run the forwarder, /var/www/python/_main/forward.py /path/to/your/new/install
26
 * Check if you need to set up a bastion install
27
   * If you do, ssh to the bastion and run
28
     * screen
29
     * inside screen: ssh -L7750:localhost:5432 cmd@the_actual_db_server
30
     * ctrl-a ctrl-d, detach from screen. It keeps on truckin'.
31
 * Run python /var/www/pggraph/_main/bridge.py /path/to/your/new/install/config.ini
32
 * Wait until it returns. Now run it again.
33
 * check http://www.cmdalert.com/pggraph/your_install/
34
 * Make sure there's graphs.
35
 * Go edit the Apache config, copy and paste the existing format for PG-backed password auth.
36
   * The file is 001-something in /etc/apache2/sites-enabled
37
 * Using the existing format, dump and add the database users from the client 
38
   Trac to serve as the auth for pggraph.
39
 * Reload apache.
40
 * Add the cronjob, based on the other cronjobs for pggraph.
41
 * Fini.
42
     
43
== Introduction ==
44
45
If youre reading this document, congratulations! Youve managed to 
46
successfully install the server software for PGGraph! Youve done an amazing
47
thing, and you ought to be proud of yourself.
48
49
This document will be discussing the next step, setting up client systems for
50
pggraph, the configuration weirdness and gotchas that you need to know.
51
52
The specific sections in this document are as follows:
53
54
 * The Installer
55
 * Server preparation
56
 * Running it the first time
57
 * Cronjob
58
 * Cursing
59
 
60
== The Installer ==
61
62
Step one, setting up a client instance.
63
Given PGGraphs curious evolutionary path, the process of installing a client
64
was once much more difficult than it is now, and thus, I created the 
65
Installer.
66
67
The installer is run from the source directory of all PGGGraph scripts - in
68
the case of cmdalert.com, /var/www/pggraph/_main/
69
70
The software is executed thusly:
71 3 Aurynn Shaw
{{{
72
  $ python ./install.py
73
}}}
74 1 Aurynn Shaw
It will then ask you a number of questions specifically relating to the 
75
install process.
76
For clarity, the installer is reproduced here to give you a basic walkthrough
77
on what values are expected and a basic concept of how PGGraph works.
78
79 4 Aurynn Shaw
For clarity, the || || characters have been added to delineate between my 
80
comments and what will be shown by the installer.
81
82 1 Aurynn Shaw
{{{
83
    
84
    Installation directory?  || Requires absolute path: This is the exact path to the install. /var/www/pggraph/some_dir. || 
85
    Relative WWW path?  || Usually pggraph/prefix: pggraph/some_dir. Controls HTML generation. || 
86
    Full text title?  || Usually customer name: Page title, in the databse || 
87
    Now for some database configuration..
88
89
    Local pggraph database name? [pggraph]  || Don't change this, unless you're putting their data in a new DB || 
90
    Table prefix?  || some prefix, like aweber or vltlgx or something. To keep tables from colliding. || 
91
    Local database port? [5432]  || It's on 5432. || 
92
    Local pggraph host? usually localhost [localhost]  || It's always localhost. Except when it isn't. || 
93
    
94
    Local username? usually pggraph [pggraph]  || It's pggraph. || 
95
    Password? Can be blank [8sm4LA2]  || It's this password. || 
96
97
    Remote DB username?  || Username we connect to the remote DB with. cmd, postgres, or other. || 
98
    Remote DB password?  || As described || 
99
    Remote DB host? [localhost]  || Almost always localhost. || 
100 5 John Worsley
    Local port to bind to remote PG port? [7555]  || Always a non-5432 local port. Be sure this is numeric and not "None" as it may default if this is the first run. || 
101 1 Aurynn Shaw
102
    Some pggraph settings..
103
104
    Shortest display interval, in minutes? [10]  || How frequently we're updating the database. || 
105
    Maximum number of images per page? [10]  || Suffers from an off-by-one error, so enter 1 more than you want to display. || 
106
    Delay before continuing PGGraph startup [7]  || Number of seconds after initializing a new tunnel via forward, that the script waits before attempting to connect. || 
107
108
    So far, so good. Now for the SSH configuration for the PG tunnel.
109
110
    SSH User?  || SSH user name. almost always cmd || 
111
    SSH Password?  || Deprecated in favour of || 
112
    Authentication keyfile? [/home/pggraph/.ssh/id_dsa]  || the default admin ssh key. Good almost everywhere. || 
113
    Remote server?  || remote server IP || 
114
    Port to SSH to? [22]  || Remote port to SSH to || 
115
    Bind to what port? [5432]  || Remote port to bind to - not always 5432, in cases of bastions. || 
116
    Bind to which remote address? [localhost]  || Again, not always, in case of bastions. || 
117
    Use a twin tunnel mechanism? Leave blank if you dont know currently buggy:  || Don't use this. It was a prior attempt at making bastions work. || 
118
        Address for 2nd server?  || Only shows up if you select twin tunnel. So don't. :) || 
119
    
120
}}}
121
122
Any question with a [] following it indicates that pressing enter will give 
123
you the default value.
124
125
Once you've answered all the installer's questions, it will go off and create
126
the appropriate directories, create symlinks for all the necessary scripts and
127
libraries, as well as write out the configuration data.
128
129
At this point, you're about halfway done.
130
131
The first step (as of Mar 17, 2008), is to symlink busy_dml_report.py into the
132
freshly created directory. It's a recent addition and is not yet supported by
133
the installer script.
134
135
== Server Preparation ==
136
137
=== No Bastion ===
138
The next step is, the remote server information needs to be verified. This is 
139
always going to be in the customers' Trac wiki, and easily found.
140
141
To test the tunneling setup:
142
143
{{{
144
    
145
   $ python /var/www/pggraph/forward.py /var/www/new_server
146
   Initializing tunnel..
147
   $ psql -d template1 -U cmd -h localhost -p <the port you gave it>
148
}}}
149
150
If you connect, success! You can move on to the next phase in the installation.
151
152
If you don't, you can use the error message you got in helping you track down
153
why. The most common errors I've had are specific to pg_hba settings being
154
incorrect, and the cmd user not existing.
155
156
157
=== Bastions ===
158
159
A fact of life, some of our customers don't trust people on the internet. It's
160
hard to not blame them for that, but it does make our life a lot harder when
161
it comes to running pggraph for them.
162
163
To set up a bastion host for pggraph, you're going to need to do a little bit
164
of trickery.
165
166
The specific process I've followed is:
167
 * Set up forward.py as normal on cmdalert.
168
 * ssh into the bastion server
169
 * Using screen, run
170
   {{{
171
       ssh -L7750:localhost:5432 cmd@the_actual_db_server
172
   }}}
173
 * Detach from screen, leaving the ssh tunnel open.
174
 * Now that you're back on the bastion server, see if you can telnet to port 
175
   7750. 
176
 * If you can, disconnect and attempt to connect via psql from cmdalert.
177
   * Retrace your steps in the event that you can't connect.
178
   * Most error messages are going to be caused by incorrect settings in
179
     pg_hba.conf.
180
     
181
You will need to check on your bastion-side tunnel from time to time, as 
182
they are significantly more fragile than the standard pggraph installation,
183
and require manual intervention to recreate.
184
185
186
== Running PGGraph ==
187
188
At this point, you're close to finished the PGGraph installation. The next 
189
step is to run the data-collecting script and test for proper graph 
190
generation.
191
192
This is fairly easily done, simply
193
{{{
194
python ../_main/bridge.py ./config.ini
195
}}}
196
from the newly installed PGGraph client path.
197
198
This will attempt to connect across the tunnel and collect the data that is
199
used to build the graphs.
200
201
At this point, you should open a browser window to the newly created path (as
202
mentioned in the installer), usually http://cmdalert.com/pggraph/your_client/
203
The first graph is a good indicator whether or not data is being correctly
204
collected, and its presence shows that you can move on to the next step.
205
206
If it fails to display, first you should re-run the data collector, and test 
207
the graph again. A lot of the architecture of PGGraph is based around deltas
208
of dataset - (dataset - 1 time unit), so only a single database entry may 
209
cause unintended failure.
210
211
If that fails, verify that your connection chain is intact. Failing that, feel
212
free to email me and ask for help.
213
214
215
== Cron ==
216
217
In order for PGGraph to continue to collect data about its client systems, the
218
data collector needs to be run on a regular basis. Thus, comes the cronjob.
219
220
Adding the cronjob is simple: as the pggraph user, add a line similar to the 
221
following to your crontab, and you're done.
222
223
{{{
224
    */10 * * * * /usr/bin/python /var/www/pggraph/_main/bridge.py /var/www/pggraph/your_install_dir/config.ini
225
    or
226
    5,15,25,35,45,55 * * * * /usr/bin/python /var/www/pggraph/_main/bridge.py /var/www/pggraph/your_install_dir/config.ini
227
}}}
228
229
Write out the crontab, and pggraph is now collecting data on a regular cycle.
230
231
232
== Apache ==
233 5 John Worsley
234
=== Python ===
235
236
Use CGI Python, not mod_python. Example configuration:
237
238
{{{
239
<Directory "/var/www/pggraph/*/">
240
Options +ExecCGI -Indexes -MultiViews +SymLinksIfOwnerMatch
241
DirectoryIndex
242
AddHandler cgi-script .py
243
</Directory>
244
}}}
245
246
=== Access ===
247 1 Aurynn Shaw
248 2 Aurynn Shaw
As pggraph contains a lot of very detailed information on the structure
249 1 Aurynn Shaw
of the clients' database, it's generally a bad idea for anyone who can type in
250
a URL to be able to access it.
251
252
From that, we come to require an authentication system to keep fingers and 
253
eyeballs from where they don't belong.
254
255
As we're using a Pg-backed authentication system in Apache, and as it's 
256 2 Aurynn Shaw
already fully set up, I won't go into any detail regarding installation or
257
configuration issues.
258
259
To set up the directory-specific authentication, you should copy and paste
260
the following configuration and modify it as needed to fit the new PGGraph
261
installation.
262
263
The file that needs to be altered is /etc/apache2/sites-enabled/default-ssl
264
Please back up this file prior to modification.
265
266
{{{
267
<Location /pggraph/your_install>
268
269
        AuthName "Pggraph authentication for: Your Installation"
270
        AuthType Basic
271
        Auth_PG_host localhost
272
        Auth_PG_port 5432
273
        Auth_PG_user pggraph
274
        Auth_PG_pwd 8sm4LA2
275
        Auth_PG_authoritative on
276
        Auth_PG_database pggraph_users
277
        Auth_PG_pwd_table client_users
278
        Auth_PG_hash_type MD5
279
        Auth_PG_uid_field user_name
280
        Auth_PG_pwd_field user_passwd
281
        Auth_PG_log_table client_access_log
282
        Auth_PG_log_uname_field login
283
        Auth_PG_log_date_field date
284
        Auth_PG_log_uri_field   request
285
        Auth_PG_log_addrs_field  ip_address
286
        AuthBasicAuthoritative Off
287
        Require valid-user
288
</Location>
289
}}}
290
291
Reload Apache via /etc/init.d/apache2 reload, and your new 
292
pggraph installation is now completed.