I'm still no closer to solving the problems I'm having with mod_mono. It means having to restart the apache server at least twice a day, leaving large gaps in the day where this blog is completely inaccessible.
Until such time as I can trace and resolve the mod_mono issue, I've decided to resort to an ugly hack. Yes, you guessed it: automatically restarting the server if the blog goes down.
There are native Linux solutions to this very problem. System Integrity Monitor is one of them, but I was unable to make it work for my mod_mono driven virtual host, so I decided to roll my own solution.
The idea here is to periodically poll the blog and determine whether there are any HTTP errors returned. If an error is returned, we simply restart apache and send an email off to ourselves notifying us of the action.
There are native Linux commands that could be compiled into a bash script to do all of this. But since I'm a windows programmer, it's way easier writing a simple .NET console application to do the work. In my implementation, I only need to know if the blog returns anything other than the requested page and a 200 response code.
Step 1: Create the monitoring application
In Visual Studio/SharpDevelop/MonoDevelop, create a new Console application (call it whatever you'd like) and insert the following code:
class Program {
static void Main(string[] args) {
if (args.Length > 0) {
try {
// create a web request to the page passed on the command line
HttpWebRequest myReq = (HttpWebRequest)WebRequest.Create(args[0]);
WebResponse response = myReq.GetResponse();
} catch (Exception ex) {
// an exception will be raised if anything but an OK response is sent
Console.WriteLine(ex.Message);
try {
// send an email to ourselves detailing the error
SmtpClient client = new SmtpClient("localhost");
client.Send(new MailMessage("server", "me@example.com", "Server Restarted", ex.Message));
} catch (Exception ex2) {
Console.WriteLine("[MAIL] " + ex2.Message);
}
// start the apache restart script
System.Diagnostics.Process.Start("/etc/init.d/apache2 restart");
}
} else
Console.WriteLine("No arguements specified");
}
}
Compile the code and upload to your Linux box.
You can test the application by passing a non-existing page. A 404 File Not Found error should be generated and your apache server should be restarted.
# mono apachemon.exe http://blog.ruski.co.za/blah.aspx
The remote server returned an error: (404) Not Found.
# * Forcing reload of apache 2.0 web server...
The next step is setting your application up to run automatically at predefined times. In Linux, the standard way of running scheduled tasks is via cron jobs.
While cron jobs appear to be simple on the outset, they unfortunately run in a different environment to commands executed on the shell command line. This can make getting them to run a bit of a PITA. I was unable to get the cron job to run the mono application directly. Instead I was forced to create a bash script that runs the mono app which is then called by the cron job.
Step 2: Create a bash script
Using your favorite text editor, create a new script file (if you create it in the /usr/local/bin folder, it will automatically be in your path).
#nano /usr/local/bin/checkblog.sh
Place the following lines in your new script file:
#!/bin/bash
/usr/local/bin/mono /home/apachemon.exe http://blog.ruski.co.za/default.aspx
As the environment variables are likely to be different for a cron job, you should always make sure to use the full paths to any scripts/applications you call.
If you thought that you could just run your script now, you'd be wrong. Linux won't execute a script file unless the file is marked as executable. Use chmod to allow execution of your bash script:
#chmod 777 /usr/local/bin/checkblog.sh
Now that the file has been marked as executable, you can test it by simply typing "checkblog.sh" on the command line.
Step 3: Register the cron job
Each user has his own list of cron jobs. Because of that it is important that you log in as the correct user when creating a cron job. It is possible to create cron jobs for users other than yourself, but it is a bit more involved than what I'm about to demonstrate.
To view the currently registered cron jobs for the current user, type the following:
crontab -l
To modify the list of cron jobs for the currently logged in user:
crontab -e
This will open the system default text editor from which you can modify the list of cron jobs. In the cron job file, each job is listed on its own line. Each command line is prefixed with a set of 5 numbers which specify when the job is to run. You can substitute a number with a star (*) which indicates that you would like to run the job at any of the available intervals.
# +---------------- minute (0 - 59)
# | +------------- hour (0 - 23)
# | | +---------- day of month (1 - 31)
# | | | +------- month (1 - 12)
# | | | | +---- day of week (0 - 7) (Sunday=0 or 7)
# | | | | | * * * * * command to be executed
As we would like our job to run every minute of every day, we simply use a set of 5 stars to indicate the schedule.
* * * * * sh /usr/local/bin/checkblog.sh
Now save the text file and your cron job should be ready for action.
Step 4: Debug your cron job
Due to the nature of the cron job environment, you're likely to run into a situation where your cron job isn't working. There is a fair amount of cron debugging information available on the web, but I'll run you through some of the methods I used to get this job running.
Firstly, make sure that the cron daemon is running:
ps aux | grep "cron" root
5242 0.0 0.3 2040 800 ? Ss Nov05 0:00 /usr/sbin/cron
You should see at least one line like the one above.
Secondly, check the system log to see if your job was actually run:
tail -f /var/log/syslog | grep /USR/SBIN/CRON
Nov 6 10:14:01 ruski /USR/SBIN/CRON[20457]: (root) CMD (sh /usr/local/bin/checkblog.sh)
You should see a line indicating the last time your job was run (of course you need to wait for your job's schedule to happen first!).