Lance McNearney
lance@mcnearney.net

In case anyone else runs into this issue - if you're using the Memcached module for PHP and start mysteriously receiving the following result code:

RES_PROTOCOL_ERROR - 8

You may want to check that the key you're using doesn't contain any spaces in it, like the following (which was my verbose way of creating a cache key for a service call's results):

Categories->GetProducts(12345, 25, true);

If it does, it seems that they break the ASCII protocol when setting or getting the key (confirmed by turning "very verbose logging" on for the memcached server). I would have assumed that the value was hashed on the client before being sent. It's an easy fix though - just hash the key with something like md5.

It is documented in the Memcached's project documentation (although it's kind of obscure):

Avoid User Input

It's very easy to compromise memcached if you use arbitrary user input for keys. The ASCII protocol uses spaces and newlines. Ensure that neither show up your keys, live long and prosper. Binary protocol does not have this issue.

I've been experimenting with different caching methods ever since reading this post on the performance differences between no caching, built-in Django caching, memcached, static files, and Varnish. Varnish and static files were the fastest by a large margin.

Since I'd like to avoid adding another proxy layer to my setup (already using both Nginx and Apache), I looked into the caching capabilities built-in to Nginx. It seems that originally if you wanted to use caching with Nginx you used ncache. This is no longer the case since that project has been built into the Nginx core.

To add support for purging a cached item, you need to install the ngx_cache_purge module. Unfortunately, installing an Nginx module isn't as easy as installing an Apache module. It requires a re-compile and when you're used to just installing your software through apt-get, that can be a little intimidating. I also didn't want to lose the special setup that the package installer provided including using the /sites-available/ and /sites-enabled/ folders, the init script, etc.

Here's a quick guide on re-compiling while still remaining compatible with the default package:

Install Nginx through aptitude

This is really easy (and you probably already did this):

apt-get install nginx

Install compile tools

A few of the compile libraries needed by Nginx aren't installed by default. To install them, use the following command:

aptitude -y install build-essential libc6 libpcre3 libpcre3-dev libpcrecpp0 libssl0.9.8 libssl-dev zlib1g zlib1g-dev lsb-base

Download and extract the source

Download and extract the source of both the newest stable Nginx and the cache_purge module:

cd /usr/src/
wget http://www.nginx.org/download/nginx-0.7.65.tar.gz
wget http://labs.frickle.com/files/ngx_cache_purge-1.0.tar.gz
tar -xvf nginx-0.7.65.tar.gz
tar -xvf ngx_cache_purge-1.0.tar.gz
cd nginx-0.7.65/

Configure compile options

You can view the packaged Nginx's compile options by using:

nginx -V

For my server, I kept generally the same options but added the cache_purge module to the end using --add-module:

./configure --sbin-path=/usr/sbin --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug --with-http_stub_status_module --with-http_flv_module --with-http_ssl_module --with-http_dav_module --with-ipv6 --add-module=/usr/src/ngx_cache_purge-1.0

This command will write out a summary of the configured options.

Compile Nginx

Compiling Nginx can be done with this command:

make && make install

A lot of text will scroll by during the compilation. In my case, I hadn't stopped my Nginx service before compiling and an error occurred when the install process attempted to copy the new Nginx executable over the current one. To fix this, I had to manually stop, copy and restart Nginx:

/etc/init.d/nginx stop
cp objs/nginx /usr/sbin/nginx
/etc/init.d/nginx start

If everything worked, you can view the current version and compile information of the running Nginx by using:

nginx -V

And the final result should be:

nginx version: nginx/0.7.65
built by gcc 4.4.1 (Ubuntu 4.4.1-4ubuntu8)
TLS SNI support enabled
configure arguments: --sbin-path=/usr/sbin --conf-path=/etc/nginx/nginx.conf 
--error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid 
--lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log 
--http-client-body-temp-path=/var/lib/nginx/body 
--http-proxy-temp-path=/var/lib/nginx/proxy 
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug 
--with-http_stub_status_module --with-http_flv_module 
--with-http_ssl_module --with-http_dav_module --with-ipv6 
--add-module=/usr/src/ngx_cache_purge-1.0

This post is a continuation of the first post on creating a gravatar template tag.

My test script was fairly straight forward to get working as a template tag. The only gotcha was not realizing that any values passed into the template tag are simple strings and that it was my job to resolve any variables being passed in. Here's an example:

Template:

{% gravatar comment.user_email 64 as gravatar %}

Tag code:

class GravatarNode(Node):   
    def __init__(self, email, size, varname):
        self.email = email

    def render(self, context):
        return self.email

@register.tag
def gravatar(parser, token):
    bits = token.contents.split()
    return GravatarNode(bits[1], bits[2], bits[4])

This template tag will render:

comment.user_email

Since I was hashing this value and passing it to the gravatar service it took me a little while to figure out what was happening.

To get it to resolve the actual value of the comment.user_email variable, you have to wrap the string variable in a template.Variable() and then call the resolve(context) method of the variable inside the tag's render() method:

class GravatarNode(Node):   
    def __init__(self, email, size, varname):
        self.email = template.Variable(email)

    def render(self, context):
        return self.email.resolve(context)

@register.tag
def gravatar(parser, token):
    bits = token.contents.split()
    return GravatarNode(bits[1], bits[2], bits[4])

Now it will return the comment user's e-mail address.

Here's the completed code for the template tag. You can see how it's used in a template based on the main tag's comments:

from django import template
from django.template import Library, Node, TemplateSyntaxError
import urllib, hashlib, httplib

register = Library()

GRAVATAR_DOMAIN = 'gravatar.com'
GRAVATAR_PATH = '/avatar/'

class GravatarNode(Node):   
    def __init__(self, email, size, varname):
        self.email = template.Variable(email)
        self.size = int(size)
        self.varname = varname

    def render(self, context):
        hash = hashlib.md5(self.email.resolve(context)).hexdigest()
        context[self.varname] = get_gravatar(hash, self.size)
        return ''

def is_default(hash):
    '''
    Returns whether the specified gravatar is returning the default icon.
    '''
    try:
        query = urllib.urlencode({
            'gravatar_id': hash,
            's': 1, # Smallest size available
            'default': '/' # Causes a re-direct when the gravatar is missing
        })
        full_path = '%s?%s' % (GRAVATAR_PATH, query)

        # Create connection and test for 302 redirect
        conn = httplib.HTTPConnection(GRAVATAR_DOMAIN)
        conn.request('HEAD', full_path)
        response = conn.getresponse()

        return response.status == 302
    except:
        return True

def get_gravatar(hash, size):
    '''
    Returns an object with the gravatar information.
    '''
    query = urllib.urlencode({
        'gravatar_id': hash,
        's': size
    })

    return {
        'src': 'http://%s%s?%s' % (GRAVATAR_DOMAIN, GRAVATAR_PATH, query),
        'width': size,
        'height': size,
        'is_default': is_default(hash)
    }

@register.tag
def gravatar(parser, token):
    '''
    This tag is used for rendering a gravatar icon.

    You can also check that the gravatar exists by checking the is_default property.

    Usage::
        {% gravatar email_address size as variable_name %}

    Example::
        {% gravatar user.email 64 as icon %}
        {% if not icon.is_default %}
            <img src="{{ icon.src }}" width="{{ icon.width }}" 
            height="{{ icon.height}}" alt=""/>
        {% endif %}
    '''
    bits = token.contents.split()
    if len(bits) != 5:
        raise TemplateSyntaxError, 'gravatar takes exactly four arguments'
    if bits[3] != 'as':
        raise TemplateSyntaxError, 'third argument must be "as"'
    return GravatarNode(bits[1], bits[2], bits[4])

You can download full template tag here. Since I'm still fairly new to Django and Python I would appreciate any feedback anyone has to offer. Please add a comment or contact me directly. Thanks and enjoy!

I spent the last couple of days implementing Django's comment framework for my blog. Since making a comment requires the user's e-mail address, I wanted to display the user's gravatar icon next to their name (but only if they have a gravatar setup).

How Gravatar Works

Gravatar is a free service that lets a user associate a graphic with their e-mail address. The only thing required for sites to implement the service is to create a md5 hash of the user's e-mail address. You then make a request to retrieve that user's icon:

http://gravatar.com/avatar/?gravatar_id=1440826ae9908cf7c415dbfd3a66b240&s=64

The above request returns my gravatar icon at 64x64 pixels (the little graphic to the left).

Existing Django/Python Implementations

Searching for existing implementations of Gravatar template tags turned up the following two results:

• Django-gravatar
• Implementing Gravatar in Django

The first project is aimed at extending Django's User model. It also renders the gravatar as an img tag without providing any context variables for me to integrate into my own template HTML.

The blog post by Drew Engelson was a much better implementation. The only piece missing was that I wanted to check to see if a gravatar exists for an e-mail address instead of just providing a default image.

Since neither one of these solutions supported checking if a gravatar exists, I set out to create my own template tag.

Does a Gravatar Exist?

The gravatar service will return its own "default" image for users who don't have a gravatar setup. You can also pass a "default" parameter to the service with your request and it will issue a 302 re-direct to that address instead of returning the service's default image.

My goal was to issue a simple HEAD request to the service with a default parameter and if it issued a 302 re-direct response I knew the gravatar didn't exist for the specified e-mail address. Unfortunately, the urllib library in Python doesn't support HEAD requests and it also automatically follows re-directs to the new address (and doesn't tell you that it followed the re-direct). Because of this, I needed to use the lower level httplib library.

Here's a simple test script that implements the functionality I wanted to achieve:

#!/usr/bin/env python
import sys
import urllib, hashlib, httplib

GRAVATAR_DOMAIN = 'gravatar.com'
GRAVATAR_PATH = '/avatar/'

def main():
    email = sys.argv[1]
    hash = hashlib.md5(email).hexdigest()
    print email + ' = ' + hash

    query = urllib.urlencode({
        'gravatar_id': hash,
        's': 1, # Smallest size available
        'default': '/' # Causes a re-direct when the gravatar is missing
    })
    full_path = '%s?%s' % (GRAVATAR_PATH, query)
    print full_path

    # Create connection and test for 302 redirect
    conn = httplib.HTTPConnection(GRAVATAR_DOMAIN)
    conn.request('HEAD', full_path)
    response = conn.getresponse()
    print response.status

    if response.status == 302:
        print 'No gravatar :('
    else:
        print 'Gravatar found :)'

if __name__ == '__main__':
    main()

You can download the full gravatar test script if you're interested.

With my e-mail address passed to the script it found a valid gravatar:

$ gravatar.py lance@mcnearney.net
lance@mcnearney.net = 1440826ae9908cf7c415dbfd3a66b240
/avatar/?default=%2F&s=1&gravatar_id=1440826ae9908cf7c415dbfd3a66b240
200
Gravatar found :)

And testing with a bogus e-mail address returned the following:

$ gravatar.py bogus@example.com
bogus@example.com = 42d33ed1a79b32726b43ade0460c7ff8
/avatar/?default=%2F&s=1&gravatar_id=42d33ed1a79b32726b43ade0460c7ff8
302
No gravatar :(

Now I just needed to turn this example code into a Django template tag...

Read part two of this post for the template tag implementation.