I like to monitor our web servers by ssh'ing into the remote machine and watching "top", tailing log files, etc. Normally, I open a terminal, ssh into the remote machine, run the monitoring command (e.g. "top"), then repeat for the rest of the remote machines. Then I adjust the window sizes so I can see everything at once.
My window manager, wmii, is great for tiling a bunch of windows at once. It is also scriptable with Python, so I wrote a Python script to create my web server monitoring view. Below is my script. I also put a video on YouTube.
#!/usr/bin/env python import os import time NGINX_MONITOR_CMD = "tail --follow=name /var/log/nginx/cache.log | grep --color -E '(HIT|MISS|EXPIRED|STALE|UPDATING|\*\*\*)'" APACHE_MONITOR_CMD = "top" MYSQL_MONITOR_CMD = "mysqladmin extended -i10 -r | grep -i 'questions\|aborted_clients\|opened_tables\|slow_queries\|threads_created' " CMDS_COL1 = ['urxvt -title "Nginx 1" -e ssh -t us-ng1 "%s" &' % NGINX_MONITOR_CMD, 'urxvt -title "Nginx 2" -e ssh -t us-ng2 "%s" &' % NGINX_MONITOR_CMD, ] CMDS_COL2 = ['urxvt -title "Apache 1" -e ssh -t us-med1 "%s" &' % APACHE_MONITOR_CMD, 'urxvt -title "Apache 2" -e ssh -t us-med2 "%s" &' % APACHE_MONITOR_CMD, 'urxvt -title "Apache 3" -e ssh -t us-med3 "%s" &' % APACHE_MONITOR_CMD, ] CMDS_COL3 = ['urxvt -title "MySQL 1" -e ssh -t us-my1 "%s" &' % MYSQL_MONITOR_CMD, 'urxvt -title "MySQL 2" -e ssh -t us-my2 "%s" &' % MYSQL_MONITOR_CMD, ] COLUMNS = [CMDS_COL1, CMDS_COL2, CMDS_COL3] def create_windows(): for i, col in enumerate(COLUMNS): cindex = str(i+1) for cmd in col: os.system(cmd) time.sleep(1) os.system('wmiir xwrite /tag/sel/ctl send sel %s' % cindex) os.system('wmiir xwrite /tag/sel/ctl colmode %s default-max' % cindex) os.system('wmii.py 45.5 31.5 23') if __name__ == '__main__': create_windows()
Note 1: The script above uses another script I wrote previously,
wmii.py, to set the column widths.
Note 2: The remote server addresses are specified by the nicknames us-ng1, us-ng2, us-med1, etc. configured in my
~/.ssh/config file as described here.
Note 3 (on using ssh and top): I first tried doing
ssh host top, but this gave me a
TERM environment variable not set. error. I then tried
ssh host "export TERM=rxvt-unicode; top", but this gave me a
top: failed tty get error. The solution that worked for me was to use the
-t option with
ssh -t host top. This is what I used in the script above.
Note 4 (added 2010-03-05): I used "tail --follow=name" instead of "tail -f" so that tail will follow the log file even after it has been rotated. For more information, see the man page for tail.
Note 5 (added 2010-03-05): To prevent your ssh session from timing out, add the following 2 lines to your
~/.ssh/config file (via):
Host * ServerAliveInterval 60
- Notes on Fabric 2 and Python 3 — posted 2021-02-07
- How to expose a Flask local development server to the public using SSH remote port forwarding — posted 2013-02-12
- How to run a Django local development server on a remote machine and access it in your browser on your local machine using SSH port forwarding — posted 2012-10-23
- Notes on debugging ssh connection problems — posted 2011-08-31
- Fabric post-run processing Python decorator — posted 2010-11-06
- Class-based Fabric scripts via a Python metaprogramming hack — posted 2010-09-23