***Update***

Looking around I found this post that puts forward what is a much better solution. It suggests renaming django.fcgi to dispatch.fcgi because dreamhost has a policy of not killing things called dispatch.fcgi. I’ve updated the dreamhost wiki page on django to mention this.

What’s below is my old solution:

Primary problem

My mum’s site is set up using Django on Apache running in a shared hosting environment with Dreamhost as laid out in the official documentation and my own write-up. I was having a few problems with it. The problem was an intermittent, unpredictable error about a third of the time which lead to /internal_error.html being displayed. This means that Django wasn’t even starting or was starting and failing very early on because if it was starting it would display the 500.html template instead of /internal_error.html. I took a look at the error.log and sure enough there were errors in the following format for each failed request:
[time and date] [error] [client xxx.xxx.xxx.xxx] (104)Connection reset by peer: FastCGI: comm with server "/home/grimboy/example.com/django.fcgi" aborted: read failed
[time and date] [error] [client xxx.xxx.xxx.xxx] FastCGI: incomplete headers (0 bytes) received from server "/home/grimboy/example.com/django.fcgi"

Solution

After much frustration at these weird, unexplained errors I was just changing random stuff to see if anything increased reliability. In the fastcgi python script that does setup and then calls runfastcgi and lives inside the website root (I call it django.fcgi) I changed:
runfastcgi(method="threaded", daemonize="false")
to
runfastcgi(method="prefork", daemonize="false")
which seemed to mysteriously fix the errors.

Secondary problem

I have no idea about how fastcgi works really. So I don’t know why changing the method from threaded to prefork solved my problem. However, if I’m not completely misinformed fork() starts an additional process and threaded involves using, well, threads. Threads are more lightweight than processes so it should be preferable to use threads.

In conclusion, I am left with two questions:

• I have no idea why this fixes the original problem and plead ignorance. Anyone care to enlighten me?
• I don’t know if this problem is unique to me. It could be that the original problem is caused by something else I’m doing wrong. Has anyone else had this problem?

photo by lilit

I recently started using vim as my primary text editor. I’m no expert with it (yet) I’m just posting with some advice to other people who also want to do the same. But first, I’m going to tell you why vim is good.

• Fast in terms of loading
• Fast in terms of usage
• Customisability
• Expandability
• Portability
• Makes you feel all tingly when you do something in just a few keystrokes

Lots of people reject the idea of heavily keyboard and (human) memory reliant editors such as Emacs or vim because they have a steep learning curve. However, I’ve found vim’s learning curve to be pretty reasonable, and if you code professionally or even as a regular hobby then it’s worth taking the time to learn an editor that will speed you up.

photo by hulksjedi

There’s a nice little vim configuration thing called Cream. From what I understand this comes in two components:

1. A number of configurations and macros for normal vim
2. Cream itself (still vim underneath, but even milder, vim on tranquillisers)

I find this quite useful in that it makes normal vim behave a little bit more with its macros. However, Cream itself is too mild, it’s like using a normal text editor.

So, in conclusion, Cream is good. Download it and install it, but use vim configured by cream. Don’t use just cream itself. Well, actually you can - in fact it’s a nice gentle introduction to vim. If you just want a normal text editor then I recommend you use cream itself. I used cream itself for a while in “expert mode” before trying to customise it and failing. One thing that annoyed me in particular while coding python was that “expand tabs” worked for a single document, then claimed it was on but had to be turned off and on before it would start working again. I also wanted to be able to have the exact same configuration everywhere. This includes stuff like SSHing into Dreamhost and doing a hot-fix on a website or messing around on my old computer underneath the desk^TM that does odd jobs like backups, svn, trac and an IRC bot. I had trouble doing configuration for Cream itself and I didn’t feel in control. So really, Cream configured or not (g)vim itself was the only option for me.

photo by phitar

Anyway, let’s rewind a bit. If you want to learn vim use the tutor. Invoke the tutor by opening a terminal or a dos prompt or whatever and entering vimtutor in it (this assumes vim is in your path). Do the entire tutor, then do it again. This is the only way. You probably won’t find it difficult but you may very well find yourself thinking “this is pointless, weird and archaic” every now and then. I know I did. However, your brain will know when to use various keystrokes magically¹. Apart from the tutor I found this charming IRC style tutorial quite useful for a bit of light revision later on. On top of that there’s always the official vim tips repository which is useful for specific pieces of insight. This tip is particularly useful in a general sense as it is a compilation.

photo by tojosan

Ok, so now that you know this little editor that is fast in all senses of the word you’re probably wondering about one of its most loved features, its customisability. Firstly there is the .vimrc (or _vimrc on windows). This is something that I did mostly by example, there are a number of heavily commented .vimrc files out there so it’s fairly easy to just read through them until an option catches your eye. Here are two of my favourites. Obviously if you want to customise something in particular then you can search through help or fall back to teh interwabs.

Here’s mine:
" Kill all the tabs.
set ts=4
set sw=4
set et
set nu
set sr

” Purdy color scheme
colorscheme inkpot


photo by jm3

Plugins are the next thing. This is pretty obvious, just get some from the official vim plugin repository and shove them in ~/.vim/plugin/ (or C:\Program Files\Vim\vimfiles\plugin\ on windows) and shove colour schemes in ~/.vim/colors/. My favourite plugins at the moment are (keep in mind that some of these are geared toward python/html stuff):

• Colours Sample Pack - brings variety to the editing experience. Although I am quite settled on inkpot although matrix is good if someone else is in the room.
• Subversion (svn) Integration Plugin, update with stupid star trek name - Means I don’t have to switch into a terminal as often. (Well I’d probably use :!svn … if I didn’t use this)
• python.vim - Some python related menu commands. You need to shove the following in your ~/.vimrc au FileType python source ~/.vim/plugin/python.vim
• runscript.vim - Again, saves me doing a terminal or !python moo.py …
• Vim Taglist - Turns vim into a source code browser. Endlessly useful for large files.

photo by akash_k

Finally, I said at the beginning that one of my requirements was to have the same configuration across several machines. What I’ve done is made a vim repository in svn (I use version control for everything) which I check out on all the machines I want to have configured then svn co and ln -s bits of it it to various directories (on windows I just check out bits of the tree directly). I have the plugin and color directories in the repository. I also have a directory called common. When I said I was showing you my .vimrc I was actually showing a file called /common/vimrc in the vim repository. I then source it from all of my .vimrc files. On Ubuntu on my laptop my .vimrc looks like²:

source $VIMRUNTIME/vimrc_example.vim
source $VIMRUNTIME/mswin.vim
behave mswin

source $HOME/.vim/common/common.vim
set guifont=ProFontWindows\ 9
(Yes, I use windows shortcuts on linux)

photo by tengis

In conclusion, I’m very happy with vim. It has replace notepad++ on Windows and a different editor every month on Linux. I like controlling everything with the keyboard, a keyboard is ordered. A mouse is chaotic and can easily just wander off into Firefox onto a Wikipedia article about conspiracy theories, or something similarly stupid. Also I’ve never quite been satisfied with using dialogs to do stuff (e.g. open files, change font), gvim provides this option, but I can just as easily do this by typing if I already know what I want³. Also being able to execute terminal/dos commands directly from an editor is unspeakably convenient. If you’re learning vim and want advice or have a question, or if you are a vimmer already and want to correct me on anything I’ve said, then scratch the itch and leave a comment.

1. No, seriously.
2. I’m not lying this time.
3. I think using dialogs are like window shopping. Prompts are like entering a number into the computer thing in Argos⁴.
4. Nation specfic references and corporate endorsement all in one. I must be turning into an evil.

Dreamhost is one of the few large non-VPS shared hosting companies that is currently Django compatible. I’ve found that It hasn’t been too hard to set up either. This little guide assumes you’ve read this guide and is mainly just caveats and tips followed by a bit of personal experience on reliability.

MySQL

This is easy. First go to your dreamhost panel. Next, select goodies, then manage MySQL. Now create a database, I’d recommend Django as a name. Then, a hostname and user for the database, the hostname could be something like mysql.yourdomain.com, the user, whatever you want. Now in your settings.py change the DATABASE_* options to something like these:

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = 'django'
DATABASE_USER = 'user'
DATABASE_PASSWORD = 'pass'
DATABASE_HOST = 'mysql.yourdomain.com'
DATABASE_PORT = '3306'

Email

Under Mail > Manage Email in the dreamhost panel you need to set up an email with a mailbox. At first I was making the fatal error of only having one email that just redirects email. Then enter the info in settings.py

EMAIL_HOST = 'mail.yourdomain.com'
EMAIL_HOST_USER = 'addy@yourdomain.com'
EMAIL_HOST_PASSWORD = 'password'

Statistics

Dreamhost normally gives some rather nice statistics on all of your domains at /stats/. You lose this when you get Django to handle the whole domain. Lets take a wee look at the .htaccess’ file:

AddHandler fastcgi-script .fcgi  RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ django.fcgi/$1 [QSA,L]

Currently everything everything except for files it being handled by Django. If you have Django serve for the whole of your domain you lose /stats/ as this isn’t a file. We want to make another exception, especially for /stats/. We want to do the same thing for failed authentication. We can put an extra rule before the Django one like so.

RewriteCond %{REQUEST_URI} ^/stats/(.*)$ [OR]
RewriteCond %{REQUEST_URI} ^/failed_auth.html$
RewriteRule ^.*$ - [L] 

This rule tells Apache to just keep these two URLs as they are and not do anything with Django.

Keeping in sync with the development version

Often a website will have more than one revision. Maybe you’ll find there are some bugs in your website. Maybe you’ll want to add some new features. My recommendation here is to start a subversion repository for your project. Actually doing this is out of the scope of this guide (but there’s an excellent book) but here are some tips.

• Have a trunk and then periodically copy it to a tag and do a checkout of that tag at the dreamhost end.
• Keep the media in a directory within the source control and make a symbolic link to it from the actual website directory. Have uploads as a real directory under the domain directory and shove all the uploads there, otherwise you’ll get loads of clutter in your working copy.
• Tell subversion to ignore settings.py (as well as *.pyc) so that you can set up individual settings for your development and production environments.

Making backups

Dreamhost backup all of your stuff if bad stuff were to go down over there. However, personally, I feel safer having backed up stuff myself as well.

SSH into dreamhost and make a file called db_dump in ~/ and have it contain

mysqldump --opt -u user -ppass -h mysql.yourdomain.com django > ~/db_backups/django.sql

as well as a variation on that line non-django databases you might have. Run chmod 755 db_dump; mkdir db_backups then crontab -e, now enter something like:

MAILTO="your@email.com"

00 09 * * * /home/you/backup_dbs

Note that I’ve chosen 9:00 daily. Dreamhost is at -700. I am at either 000 or +100 depending on day light savings. This means that I am either 7 or 8 hours ahead. This means that while this script is dumping the database it is 4 or 5 in sunny York. Do the same calculations for yourself if you’re in a non-dreamhost timezone.

Do this next bit on a local machine. If you’re using windows start by downloading wget and putting it in your path. Make a file called backup_dh (or backup_dh.bat in windows) in somewhere and have it contain something like

cd dh_backup
wget ftp://user:pass@yourdomain.com/db_backups/*

cd yoursite1_uploads
wget ftp://user:pass@yourdomain.com/yoursite1.com/uploads/*
cd ..
cd yoursite2_uploads
wget ftp://user:pass@yourdomain.com/yoursite2.com/uploads/*
cd ..

Run chmod 755 backup_dh (*nix only), mkdir dh_backup; cd dh_backup; mkdir yoursite1_uploads; mkdir yoursite2_uploads.

Then on *nix crontab -e, enter (or append) something like:

  30 17 *   *   *     /home/you/backup_dh

On windows Start > Control Panel > Scheduled Tasks > Add Scheduled Task > Next > Browse… > look for backup_dh.bat > Select daily > Choose a time - for me it’s 17:30 > Enter your username and password > Finish.

Notice that this downloading is scheduled to happen at 5:30 in sunny York, 1/2 to 1 1/2 hours since dreamhost did its database dump,, calculate a similar thing yourself.

Optimising so dreamhost don’t kill you

Officially you’re safe with dreamhost if you use under 60 cpu minutes (that’s 3600 seconds). Anything over that will be evaluated on a case by case situation. You should take a look at http://yourdomain.com/stats/resources/yourunixname.sa.analysed.0 and see if you are getting too near that. If you are, don’t despair, luckily Django comes with a built cache framework. A few pointers:

• Make sure whatever you do, to turn on CACHE_MIDDLEWARE_ANONYMOUS_ONLY, otherwise the admin will start playing silly buggers.
• I would choose local memory caching for any smallish sites. Otherwise use db caching (which, from what I understand should always be just as fast as file-system caching on dreamhost as both files and database is stored on machines other than that running the Apache instance).

Also have a dive into python has an amazing chapter on performance tuning.

Thoughts

So far I’ve been fine, but I really haven’t got that much traffic yet (not even on my mum’s fantastic writing site). However, I’m sure that will all change when I launch my new web application idea thing. So I’ll report back on this later.

Finally…

I’m sure I’ve made some mistakes here so please comment and correct me. Also please comment if you’re having trouble setting up Django on dreamhost.