Caching with Memcached, Express and NodeJS

In a previous post I discussed how you can use Redis to store NodeJS session data. While in many cases Redis is a better product than Memcached, Memcached has one big advantage is that it has least-recently-used eviction. Basically, when the table is full, older data will be removed based on the LRU algorithm.

With NodeJS, there is a great npm package called memcache.

To access the package I created a simple wrapper called cache.js.  Like with all programming languages, some abstract can be a good thing.

var memcache = require(‘memcache’);

var client = undefined;
exports.connect = function() {
client = new memcache.Client();
client.connect();
return client;
}

exports.get = function(key, callback) {
client.get(key, callback);
};
exports.set = function(key, value) {
client.set(key, value, function(error, result){}, 10);
};

The implementation is then very simple:
1) Call cache.connect(); after app.listen(3000);
2) Within your route, you provide the key and have the result value returned in the callback. If the item is undefined then perform the required logic, set the value and continue.

app.get(‘/’, function(req, res){
cache.get(‘key’, function(error, result){
console.log(result);

if(result == null) {
result = microtime();
cache.set(‘key’, result);
}

res.render(‘index’, {title: “Hello Express at ” + result});
});

Simple. Dirty.

Sadly, every time I looked at the above code I felt dirty.

As such, I wondered. How else could you do this?

One approach I thought could be interesting is to take advantage of the middleware pipeline nature of Express and NodeJS. Now the logic and responsibility is separated into their own functions.

function getCache(req, res, next) {
cache.get(req.route.path, function(error, result){
req.params.cachedItem = result;
next();
});
};

function processIndexIfNotInCache(req, res, next) {
if(req.params.cachedItem == undefined) {
req.params.cachedItem = microtime();
cache.set(req.route.path, req.params.cachedItem);
}
next();
}

app.get(‘/’, getCache, processIndexIfNotInCache, function(req, res){
res.render(‘index’, {title: “Hello Express at ” + req.params.cachedItem});
});

However, while getCache is generic and reusable, processIndexIfNotInCache is a bit of a mouthful and you’ll soon start seeing repeating logic.

Dirty in a different way

While we’ve improved the code in some aspects, it’s still not great. In my final attempt, I was left with two generic methods and a hash which pointed the point to where the cacheable logic lived.

function processCache(req, res, next) {
cache.get(req.route.path, function(error, result){
if(result == undefined)
result = executeAndStore(req.route.path);

req.params.cachedItem = result;
next();
});
};

function executeAndStore(path) {
var result = routes[path]();
cache.set(path, result);
return result;
}

var routes = {‘/’: microtime }
app.get(‘/’, processCache, function(req, res){
res.render(‘index’, {title: “Hello Express at ” + req.params.cachedItem});
});

There are still a few refactoring to go, but we have started to start separating the logic of caching from the logic of the method and the actual rendering itself.

A sample, along with many others, can be found at https://github.com/BenHall/memcached_express

I would love to see other examples of how people would solve this – ping me @Ben_Hall

Experiment: Deploying C# / Mono on Heroku

Over the past couple of days I’ve been trying to get Mono running on top of Heroku.

I’m pleased to report that I have successfully deployed Nancy (a C# web framework) on top of Heroku.

It’s far from a production ready solution and is completely unsupported by Heroku but it’s an interesting experiment. More technical details in future posts but here’s some proof.

Server response headers:
$ curl -i http://deep-moon-1452.herokuapp.com/
HTTP/1.1 200 OK
Content-Type: text/html
Date: Tue, 03 Jan 2012 16:33:02 GMT
Nancy-Version: 0.9.0.0
Server: Mono-HTTPAPI/1.0

Screenshot:

This is the output when you push your git repository to Heroku. The source code is built on Heroku.

$ git push heroku master
Counting objects: 9, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 514 bytes, done.
Total 5 (delta 3), reused 0 (delta 0)


—–> Heroku receiving push
—–> Fetching custom buildpack… done
—–> Mono app detected
—–> Fetching Mono binaries
tar: Ignoring unknown extended header keyword `SCHILY.dev’
tar: Ignoring unknown extended header keyword `SCHILY.ino’
tar: Ignoring unknown extended header keyword `SCHILY.nlink’
tar: Ignoring unknown extended header keyword `SCHILY.dev’
tar: Ignoring unknown extended header keyword `SCHILY.ino’
tar: Ignoring unknown extended header keyword `SCHILY.nlink’
tar: Ignoring unknown extended header keyword `SCHILY.dev’
tar: Ignoring unknown extended header keyword `SCHILY.ino’
tar: Ignoring unknown extended header keyword `SCHILY.nlink’
tar: Ignoring unknown extended header keyword `SCHILY.dev’
tar: Ignoring unknown extended header keyword `SCHILY.ino’
tar: Ignoring unknown extended header keyword `SCHILY.nlink’
tar: Ignoring unknown extended header keyword `SCHILY.dev’
tar: Ignoring unknown extended header keyword `SCHILY.ino’
tar: Ignoring unknown extended header keyword `SCHILY.nlink’
tar: Ignoring unknown extended header keyword `SCHILY.dev’
tar: Ignoring unknown extended header keyword `SCHILY.ino’
tar: Ignoring unknown extended header keyword `SCHILY.nlink’
tar: Ignoring unknown extended header keyword `SCHILY.dev’
tar: Ignoring unknown extended header keyword `SCHILY.ino’
tar: Ignoring unknown extended header keyword `SCHILY.nlink’
tar: Ignoring unknown extended header keyword `SCHILY.dev’
tar: Ignoring unknown extended header keyword `SCHILY.ino’
tar: Ignoring unknown extended header keyword `SCHILY.nlink’
tar: Ignoring unknown extended header keyword `SCHILY.dev’
tar: Ignoring unknown extended header keyword `SCHILY.ino’
tar: Ignoring unknown extended header keyword `SCHILY.nlink’
—–> Vendoring mono 2.10.8
—–> building via /tmp/mono-GssQ/bin/mono /tmp/mono-GssQ/lib/mono/4.0/xbuild.exe /tmp/build_19y6xv4nr43ic/Nancy.Demo.Hosting.Self.sln
XBuild Engine Version 2.10.8.0
Mono, Version 2.10.8.0
Copyright (C) Marek Sieradzki 2005-2008, Novell 2008-2011.


Build started 01/03/2012 16:28:24.
__________________________________________________
Project “/tmp/build_19y6xv4nr43ic/Nancy.Demo.Hosting.Self.sln” (default target(s)):
Target ValidateSolutionConfiguration:
Building solution configuration “Debug|Mixed Platforms”.
Target Build:
Project “/tmp/build_19y6xv4nr43ic/src/Nancy.Demo.Hosting.Self.csproj” (default target(s)):
Target PrepareForBuild:
Configuration: Debug Platform: x86
Created directory “bin/”
Created directory “obj/x86/Debug/”
Target CopyFilesMarkedCopyLocal:
Copying file from ‘/tmp/build_19y6xv4nr43ic/lib/Nancy.dll’ to ‘/tmp/build_19y6xv4nr43ic/src/bin/Nancy.dll’
Copying file from ‘/tmp/build_19y6xv4nr43ic/lib/Nancy.Hosting.Self.dll’ to ‘/tmp/build_19y6xv4nr43ic/src/bin/Nancy.Hosting.Self.dll’
Target GenerateSatelliteAssemblies:
No input files were specified for target GenerateSatelliteAssemblies, skipping.
Target CoreCompile:
Tool /tmp/mono-GssQ/bin/dmcs execution started with arguments: /noconfig /debug:full /debug+ /optimize- /out:obj/x86/Debug/Nancy.Demo.Hosting.Self.exe Program.cs TestModule.cs /target:exe /define:”DEBUG;TRACE” /main:Nancy.Demo.Hosting.Self.Program /platform:x86 /reference:/tmp/mono-GssQ/lib/mono/4.0/System.dll /reference:../lib/Nancy.dll /reference:../lib/Nancy.Hosting.Self.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Xml.Linq.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Data.DataSetExtensions.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/Microsoft.CSharp.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Data.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Xml.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Core.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/mscorlib.dll /warn:4
Target _CopyDeployFilesToOutputDirectoryAlways:
Creating directory ‘/tmp/build_19y6xv4nr43ic/src/bin/Views’
Copying file from ‘/tmp/build_19y6xv4nr43ic/src/Views/staticview.html’ to ‘/tmp/build_19y6xv4nr43ic/src/bin/Views/staticview.html’
Target _CopyAppConfigFile:
Copying file from ‘/tmp/build_19y6xv4nr43ic/src/app.config’ to ‘/tmp/build_19y6xv4nr43ic/src/bin/Nancy.Demo.Hosting.Self.exe.config’
Target DeployOutputFiles:
Copying file from ‘/tmp/build_19y6xv4nr43ic/src/obj/x86/Debug/Nancy.Demo.Hosting.Self.exe.mdb’ to ‘/tmp/build_19y6xv4nr43ic/src/bin/Nancy.Demo.Hosting.Self.exe.mdb’
Copying file from ‘/tmp/build_19y6xv4nr43ic/src/obj/x86/Debug/Nancy.Demo.Hosting.Self.exe’ to ‘/tmp/build_19y6xv4nr43ic/src/bin/Nancy.Demo.Hosting.Self.exe’
Done building project “/tmp/build_19y6xv4nr43ic/src/Nancy.Demo.Hosting.Self.csproj”.
Done building project “/tmp/build_19y6xv4nr43ic/Nancy.Demo.Hosting.Self.sln”.


Build succeeded.
0 Warning(s)
0 Error(s)


Time Elapsed 00:00:01.0730530
—–> Discovering process types
Procfile declares types -> local, web
—–> Compiled slug size is 78.3MB
—–> Launching… done, v18
http://deep-moon-1452.herokuapp.com deployed to Heroku


To [email protected]:deep-moon-1452.git
4d9e07b..6ce44e4 master -> master

2011 – An Unexpected Journey

When I started this post I wanted to reflect on the interesting year I’ve had. I wanted to talk about how launching a start-up had taught me so much about business, technology and myself. I wanted to go on and cover how it has also meant making sacrifices, such as missing Glastonbury even when I had a ticket and cutting back on spending, blogging, open source and conference speaking – the things I really enjoy. Finally, I wanted to end with how start-up life has sometimes meant working in isolation and at times being demotivated and stressed with how everything was going.

However, while it’s important for me to look back I also want to focus on looking forward which can be summed up by this tweet:

#2012 Release more. Blog more. Drink more. Earn more.
— Ben Hall (@Ben_Hall) January 1, 2012

To get the ball rolling, interested in hiring me? Send me an email at [email protected]