- Installed cacti on the main monitoring server which we'll call slappy. I used the FreeBSD port of cacti. Slappy already had php/mysql/apache installed.
- Added a user snmp to slappy and then I generated keys using ssh-keygen for each of the servers that slappy would be monitoring.
- On each of the servers that slappy would be monitoring I installed net-snmp from the ports tree and configured it to run over tcp on 127.0.0.1 and then I added a user snmp with a nologin shell and without password authentication as I will just be using snmp to create a tunnel to the snmpd process that will be running on localhost.
- Back on slappy I su'ed to the snmp user and created a shell script that would set up the tunnels to each of the servers using a command like this:
ssh -i ~/.ssh/keys/hostname -f -N -L 16101:127.0.0.1:161 hostnameand then added the script as a cronjob.
- Finally I added all the servers to cacti using the basic built-in net-snmp support as well as a couple of qmail and mysql scripts.
So I now have a nice collection of graphs for traffic / disk space / processor, memory and mysql load.
i've been working on setting up a system of bridging ip-less packt filters with ipfilter (for logging, filtering, accounting, proxying, NAT) and ipfw (for bandwidth limiting). I also wanted to set up a private mointoring network with a thrid NIC in each box. Even though this link was privat I still wanted to keep everything on the wire encrypted.