***Update***

Looking around I found this post that puts forward what is a much better solution. It suggests renaming django.fcgi to dispatch.fcgi because dreamhost has a policy of not killing things called dispatch.fcgi. I’ve updated the dreamhost wiki page on django to mention this.

What’s below is my old solution:

Primary problem

My mum’s site is set up using Django on Apache running in a shared hosting environment with Dreamhost as laid out in the official documentation and my own write-up. I was having a few problems with it. The problem was an intermittent, unpredictable error about a third of the time which lead to /internal_error.html being displayed. This means that Django wasn’t even starting or was starting and failing very early on because if it was starting it would display the 500.html template instead of /internal_error.html. I took a look at the error.log and sure enough there were errors in the following format for each failed request:
[time and date] [error] [client xxx.xxx.xxx.xxx] (104)Connection reset by peer: FastCGI: comm with server "/home/grimboy/example.com/django.fcgi" aborted: read failed
[time and date] [error] [client xxx.xxx.xxx.xxx] FastCGI: incomplete headers (0 bytes) received from server "/home/grimboy/example.com/django.fcgi"

Solution

After much frustration at these weird, unexplained errors I was just changing random stuff to see if anything increased reliability. In the fastcgi python script that does setup and then calls runfastcgi and lives inside the website root (I call it django.fcgi) I changed:
runfastcgi(method="threaded", daemonize="false")
to
runfastcgi(method="prefork", daemonize="false")
which seemed to mysteriously fix the errors.

Secondary problem

I have no idea about how fastcgi works really. So I don’t know why changing the method from threaded to prefork solved my problem. However, if I’m not completely misinformed fork() starts an additional process and threaded involves using, well, threads. Threads are more lightweight than processes so it should be preferable to use threads.

In conclusion, I am left with two questions:

• I have no idea why this fixes the original problem and plead ignorance. Anyone care to enlighten me?
• I don’t know if this problem is unique to me. It could be that the original problem is caused by something else I’m doing wrong. Has anyone else had this problem?