Every developer I know loves the cron jobs, I can think of no development more rewarding than creating an application that can feed itself. On the flip side, processing a cron does create some prickly issues. Over the past couple months I’ve hit the same issue twice, the first time was during the development of weathertronic, where the pair of task queues that govern the forecast data import required a naming convention to key into the memcache so that the rendering side never had to worry about stale data. I hit the same issue again while putting together a police blotter app for the city of Keene, NH, only this time I really wanted to preserve caching at the template output level. The cache decorator provided is great, but there isn’t any built-in mechanism to “call it off” when you know the underlying data has changed – it was a bit too blunt for my purposes. After googling a bit, I found some snippets on clearing the django cache, but again a bit too blunt an object as there are other entries in the cache that had no need to update. Ultimately, I ended up wrapping the django cache with a bit of record keeping. I figure I’d put it out there for anyone looking to solve a similar issue..
The way to set it up is to add the below script somewhere in the python path, I created an appcache module, then at the rendering function called from the url dispatcher load it up with the namespace as the app name and your off.
from appcache import AppCacheManager
blotter_cache = AppCacheManager('blotter')
cache_response = blotter_cache.get([args/kwargs from dispatcher])
if cache_response is not None:
# intensive processing and/or http activity here
blotter_cache.set([args/kwargs from dispatcher])
In the cron all that’s left is to clear the namespace:
blotter_cache = AppCacheManager('blotter')
blotter_cache.delete()
That’s it…
from django.core.cache import cache
#********************************************************************************************************
class AppCacheManager():
"""
this is a simple keyword driven cache, it can be managed and cleared on
a per app/namespace basis. Created for the situation where a cron should flush all
http responses in cache and start over.
"""
def __init__(self, namespace,ttl=3600):
self.namespace = namespace
self.ttl = ttl
def _get_cache_list(self):
cm = cache.get(self.namespace)
return [] if cm is None else cm
def _add_cache_list_key(self,key):
cl = self.cache_list
cl.append(key)
# use set to dedupe
cache.set(self.namespace, list(set(cl)))
cache_list = property(_get_cache_list)
def _key(self, *args, **kwargs):
items = kwargs.items()
items.sort(key=lambda k: k[0])
bits = ['appcache',self.namespace]
bits.extend([str(a) for a in args])
for k, v in items:
bits.append('='.join([str(k),str(v)]))
return '_'.join(bits)
def delete(self):
for c in self.cache_list:
cache.delete(c)
cache.set(self.namespace, [])
def get(self, *args, **kwargs):
key = self._key(*args, **kwargs)
return cache.get(key)
def set(self, data, *args, **kwargs):
key = self._key(*args, **kwargs)
cache.set(key, data, self.ttl)
self._add_cache_list_key(key)