Rolling Averages in Redis

I’m working on a system that consumes a bunch of readings from a sensor and I thought how nice it would be if I could get a rolling average into Redis. As I’m going to be consuming quite a few of these pieces of data I’d rather not fetch the current value from redis, add to it and send it back. I was thinking about just storing the aggregate value and a counter in a hash set in Redis and then dividing one by the other when I needed the true value. You can set multiple hash values at the same time with HMSET and you can increment a value using a float inside a hash using HINCRBYFLOAT but there is no way to combine the two. I was complaining that there is no HMINCRBYFLOAT command in Redis on twitter when Itamar Haber suggested writing my own in Lua.

I did not know it but apparently you can write your own functions that plug into Redis and can become commands. Nifty! I managed to dig up a quick Lua tutorial that listed the syntax and I got to work. Instead of a HMINCRBYFLOAT function I thought I could just shift the entire rolling average into Redis.

	local currentval = redis.call('get', ARGV[1])
	local currentcount = redis.call('get', ARGV[1] .. '.count')
	redis.call('incr', ARGV[1] .. '.count')
	currentval = (currentval * (currentcount/(currentcount + 1))) + (ARGV[2]/(currentcount+1))
	redis.call('set', ARGV[1], currentval)
	return currentval

view raw

rollingAVG.lua

hosted with ❤ by GitHub

This script gets the current value of the field as well as a counter of the number of records that have been entered into this field. By convention I’m calling this counter key.count. I increment the counter and use it to weight the old and new values.

I’m using the excellent StackExchange.Redis client so to test out my function I created a test that looked like

	[Fact]
	public void EvalTest()
	{
	var redisConnection = GetRedisConnection();
	redisConnection.StringSet("foo", 77);
	redisConnection.StringSet("foo.count", 1);
	var result = redisConnection.ScriptEvaluate(@"local currentval = redis.call('get', ARGV[1])
	local currentcount = redis.call('get', ARGV[1] .. '.count')
	redis.call('incr', ARGV[1] .. '.count')
	currentval = (currentval * (currentcount/(currentcount + 1))) + (ARGV[2]/(currentcount+1))
	redis.call('set', ARGV[1], currentval)
	return currentval", null, new RedisValue[] { "foo", 11 }, CommandFlags.None);
	((int)result).Should().Be.EqualTo(44);
	}

view raw

evaltest.cs

hosted with ❤ by GitHub

The test passed perfectly even against the Redis hosted on Azure. This script saves me a database trip for every value I take in from the sensors. On an average day this could reduce the number of requests to Redis by a couple of million.

The script is a bit large to transmit to the server each time. However if you take the SHA1 digest of the script and pass that in instead then Redis will use the cached version of the script that matches the given SHA1. You can calculate the SHA1 like so

	public byte[] GetScriptHash(string script)
	{
	SHA1 sha = new SHA1CryptoServiceProvider();
	return sha.ComputeHash(System.Text.Encoding.UTF8.GetBytes(script));
	}

view raw

hashscript.cs

hosted with ❤ by GitHub

The full test looks like

	[Fact]
	public void EvalTest()
	{
	var redisConnection = GetRedisConnection();
	redisConnection.StringSet("foo", 77);
	redisConnection.StringSet("foo.count", 1);
	string script = @"local currentval = redis.call('get', ARGV[1])
	local currentcount = redis.call('get', ARGV[1] .. '.count')
	redis.call('incr', ARGV[1] .. '.count')
	currentval = (currentval * (currentcount/(currentcount + 1))) + (ARGV[2]/(currentcount+1))
	redis.call('set', ARGV[1], currentval)
	return currentval";
	var result = redisConnection.ScriptEvaluate(script, null, new RedisValue[] { "foo", 11 }, CommandFlags.None);
	((int)result).Should().Be.EqualTo(44);
	result = redisConnection.ScriptEvaluate(GetScriptHash(script), null, new RedisValue[] { "foo", 11 }, CommandFlags.None);
	((int)result).Should().Be.EqualTo(33);
	}

view raw

fulltest.cs

hosted with ❤ by GitHub