I just kind of figured it’d be relatively painless to extract a few calendar entries. That was my first and second mistake. Want an “id”? Get ready for fun – depending on your context an id can mean a url, a uri, or a 26 character string – and I’m talking about a reference to the same entry. Is consistent id representation not api design 101, day one material? Anyone else needlessly sinking time into this?

I’ve started looking into means of connecting external google calendars into keeners.org, and found it took longer than anticipated just to get the appropriate tools in place to get started. Importing google calendars, which I had assumed to be well documented is really lacking. The goal was to not only capture occurrences of events but to capture the recurrence rrules and datetimes that define them. I downloaded the python gdata package (gdata + atom) and began hacking at a public calendar. It turns out there’s no apparent mechanism in gdata to parse the iCalendar data itself. After considering parsing out the ical data with regular expressions, I thought better and  started looking around. There’s a vobject package that seems to do the trick.

So.. Here’s some basic noodling with the two libraries, I’ve found that there’s plenty of documentation on connecting to and collecting google calendars, but little on the actual import of the data – so here’s the missing link, it requires the gdata and vobject python packages listed above.

from gdata.calendar import service
import vobject

cs = service.CalendarService()
cs.email = 'anygoogleaccount@example.com'
cs.password = 'google_password'
cs.ProgrammaticLogin()

calendar_uri = '/calendar/feeds/5hqt8fv2s3smdn4rnua5pd6hbc%40group.calendar.google.com/public/full'
feed = cs.GetCalendarEventFeed(uri=calendar_uri)

print feed.entry
# [<gdata.calendar.CalendarEventEntry object at 0x03390550>,
#    <gdata.calendar.CalendarEventEntry object at 0x03390590>,
#    <gdata.calendar.CalendarEventEntry object at 0x033909B0>]

recurrence_text = feed.entry[0].recurrence.text
recurrence = vobject.readOne(recurrence_text)
recurrence.prettyPrint()

#    DTEND: 20091213T210000
#    params for  DTEND:
#       TZID [u'America/New_York']
#    DTSTART: 20091213T200000
#    params for  DTSTART:
#       TZID [u'America/New_York']
#    RRULE: FREQ=DAILY;WKST=SU
#    VTIMEZONE
#       TZID: America/New_York
#       DAYLIGHT
#          DTSTART: 19700308T020000
#          TZOFFSETFROM: -0500
#          TZNAME: EDT
#          TZOFFSETTO: -0400
#          RRULE: FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
#
#       STANDARD
#          DTSTART: 19701101T020000
#          TZOFFSETFROM: -0400
#          TZNAME: EST
#          TZOFFSETTO: -0500
#          RRULE: FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
#
#       X-LIC-LOCATION: America/New_York

dt_start = recurrence.contents['dtstart'][0]
print dt_start.prettyPrint()
# DTSTART: 20091213T200000
# params for  DTSTART:
#    TZID [u'America/New_York']
print dt_start.params
# {u'TZID': [u'America/New_York']}
print dt_start.params['TZID'][0]
# u'America/New_York'

print recurrence.contents['rrule'][0].value
# u'FREQ=DAILY;WKST=SU'
print recurrence.contents['dtstart'][0].value
# u'20091213T200000'

I was hitting a wall setting up a few urls where I was using optional kwargs in the url. I imagine this is a common scenario where content can live in more than one place, especially if the url itself is holding down some context to maintain an anonymous view. Here’s a quick example:

(r’^blotter/(?P<filter_name>(categories|arrests|personnel))/$’, render_blotter, {}, ‘blotter’),
(r’^blotter/(?P<filter_name>(categories|arrests|personnel))/(?P<filter_value>([\w-]+))/$’, render_blotter, {}, ‘blotter’),

The two routes are dispatched to the same named function ‘blotter’, but have a unique signature depending on what arguments are passed. A more common example might be a YYYY/, YYYY/MM, YYYY/MM/DD/ situation where they should share a common render mapping. The url dispatcher can handle this fine, and for the most part it can be handled in the application logic without much of a headache by removing kwargs equal to None before revers() is called to retrieve the url. The crux of the issue is really the url tag in the templates which is not happy unless it gets an exact match of arguments.

A better explanation is available here. In any case, I’ve put a templatetag together to handle optional kwargs. Hopefully someone will find it useful.

"""
     exactly the same as from django.template.defaulttags.url EXCEPT kwargs equal to None are removed
     this allows a bit more flexibility than the use of {% url %} where nesting is rested on optional
     base kw arguments.

     see http://code.djangoproject.com/ticket/9176
"""

from django import template
from django.template.defaulttags import URLNode, url

register = template.Library()

class URLNodeOptional(URLNode):

    """
    identical to django.template.defaulttags.URLNode
    but removes kwargs equal to None before resolving the url
    """

    def render(self, context):
        for k, v in self.kwargs.items():
            if v.resolve(context) is None:
                self.kwargs.pop(k)
        return super(URLNodeOptional, self).render(context)

def url_optional(parser, token): 

    """
     creates the default URLNode, then routes it to the Optional resolver with the same properties
     by first creating the URLNode, the parsing stays in django core where it belongs.
    """ 

    urlnode = url(parser, token)
    return URLNodeOptional(urlnode.view_name, urlnode.args, urlnode.kwargs, urlnode.asvar)

url_optional = register.tag(url_optional)

A quick note on google flash maps. Somewhere in the vicinity of a few thousand markers, initialization time starts to slow down perceptibly; this is not unexpected mind you.. In the maps application I was working on, a  minority of pages passed that mark – enough to look into options, but not enough to rethink “the metaphor”. It turns out getting more headroom in flash maps is really easy. The first thing to do is to create a custom marker to replace the google pin (a movie clip asset will do) using the MarkerOptions icon property. Now the important move – cache the clip as a bitmap (marker_icon.cacheAsBitmap = true;), and pass it into the overlay as the icon. Finally make sure shadow rendering is off (MarkerOptions hasShadow property).

Caching a custom marker as a bitmap and disabling shadow isn’t going solve all your issues if you’re looking to put too many more markers in the map, but it can at least triple the marker threshold before the map becomes unusable, which if you’re dealing with a few wild fringe cases can come in handy..

Every developer I know loves the cron jobs, I can think of no development more rewarding than creating an application that can feed itself. On the flip side, processing a cron does create some prickly issues. Over the past couple months I’ve hit the same issue twice, the first time was during the development of weathertronic, where the pair of task queues that govern the forecast data import required a naming convention to key into the memcache so that the rendering side never had to worry about stale data. I hit the same issue again while putting together a police blotter app for the city of Keene, NH, only this time I really wanted to preserve caching at the template output level. The cache decorator provided is great, but there isn’t any built-in mechanism to “call it off” when you know the underlying data has changed – it was a bit too blunt for my purposes.  After googling a bit, I found some snippets on clearing the django cache, but again a bit too blunt an object as there are other entries in the cache that had no need to update. Ultimately, I ended up wrapping the django cache with a bit of record keeping. I figure I’d put it out there for anyone looking to solve a similar issue..

The way to set it up is to add the below script somewhere in the python path, I created an appcache module, then at the rendering function called from the url dispatcher load it up with the namespace as the app name and your off.

from appcache import AppCacheManager
blotter_cache = AppCacheManager('blotter')
cache_response = blotter_cache.get([args/kwargs from dispatcher])
if cache_response is not None:
    # intensive processing and/or http activity here
    blotter_cache.set([args/kwargs from dispatcher])

In the cron all that’s left is to clear the namespace:

blotter_cache = AppCacheManager('blotter')
blotter_cache.delete()

That’s it…

from django.core.cache import cache

#********************************************************************************************************

class AppCacheManager():

    """
    this is a simple keyword driven cache, it can be managed and cleared on
    a per app/namespace basis. Created for the situation where a cron should flush all
    http responses in cache and start over.    
    """

    def __init__(self, namespace,ttl=3600):
        self.namespace = namespace
        self.ttl = ttl

    def _get_cache_list(self):
        cm = cache.get(self.namespace)
        return [] if cm is None else cm

    def _add_cache_list_key(self,key):        
        cl = self.cache_list
        cl.append(key)
        # use set to dedupe
        cache.set(self.namespace, list(set(cl)))

    cache_list = property(_get_cache_list)

    def _key(self, *args, **kwargs):
        items = kwargs.items()
        items.sort(key=lambda k: k[0])
        bits = ['appcache',self.namespace]
        bits.extend([str(a) for a in args])
        for k, v in items:
            bits.append('='.join([str(k),str(v)]))        
        return '_'.join(bits)

    def delete(self):
        for c in self.cache_list:
            cache.delete(c)
        cache.set(self.namespace, [])

    def get(self, *args, **kwargs):
        key = self._key(*args, **kwargs)
        return cache.get(key)        

    def set(self, data, *args, **kwargs):
        key = self._key(*args, **kwargs)
        cache.set(key, data, self.ttl)        
        self._add_cache_list_key(key)