I’m building a set of open-source cloud data-stores in December, blogging all the way.

In our last episode, we added a Redis front-end to the datastore, supporting get and set. The vision is that we can use different protocols to talk to our key-value store.

Redis supports a lot more commands than just get & set! I spent much of today implementing other commands: append, delete, exists, increment and increment-by, decrement and decrement-by.

To be able to do that and have some confidence in it, I created some unit tests for Redis, using the jedis driver. I found and fixed some more bugs, as well! And a big refactor is that I’m using the command design pattern for performing each mutating operation: given that Redis requires so many operations, it doesn’t scale to put them all the logic into one big switch statement any more.

I had hoped to implement distributed locks with compare-and-swap via the Redis protocol, but it turns out that Redis doesn’t implement compare-and-swap. The mailing list thread is just confused / wrong, so I’m hoping that someone will revisit this at some stage and it can go into the official protocol. If I want compare-and-swap, I’ll have to use another protocol, I guess. Memcache is an option, although the way it implements compare-and-swap is a little unusual as well (it only supports swapping based on a version id, not based on the value itself). Maybe our own RESTful protocol is the way to go!

I haven’t yet implemented any of the features that make Redis unique: in particular Redis supports values that are themselves data-structures (lists, sets and sorted-sets). I’m thinking through how best to support this in a generic way. One option would be to extend the key; for a list for example we could store key=(a,b,c) as three entries in our BTree: key.1=a key.2=b and key.3=c (metaphorically speaking). Another option would be to encode the list into the value, so that any value could be “typed”; we’d probably end up with something like the COM Variant type. We could also store data structures in a separate page in our system, in a data-structure specifically designed for lists/sets/sorted sets (i.e. not a BTree).

I need to think this one over. I find the best approach with these difficult ‘philosophical’ problems is (1) to work on something completely different, like implementing a distributed lock system, and (2) to work on the related problems, like a MongoDB / DynamoDB inspired store. Sounds like a plan for the weekend!