Multi-user real-time heatmaps – a possible approach

What is this article about?

This article contains further thoughts and improvements on my recently released real-time heatmap example with Canvas and Javascript. After some feedback from the community I was wondering if there were some useful features which could make the example more inspiring.
Please keep in mind that my experiment is still no product, I cut off some essential things such as storing or managing the heatmap data on the serverside to reduce the effort. The demo is in alpha mode; if anything doesn’t work please drop me a comment, I’ll fix it as soon as possible.

The demo

Due to some performance issues and low resources on my server there are some limitations to the demo:
– A heatmap represents a channel. The maximum amount of users in a channel is 5.
– There are about 100 channels available (auto cleanup)
– I haven’t tested the demo with a huge amount of connected users yet, I’m sorry if my server isn’t able to handle it.
Have a look at it right here: Multi-user real-time heatmap

I’m sorry if there is no slot available for you, you may want to try the demo in a few minutes or days again ;-)

My thoughts and concept

Well, I wanted to push my heatmap example some steps further and thought about some of the latest technologies and how I could combine them for a more useful approach on a real-time heatmap. (the heatmap data is still not saved on the serverside, so if you expect a full working heatmap product this is probably the wrong place for you).
My first thought concerned multi-user collaboration, therefore I came to the Ajax Push Engine (short: APE) which makes it possible to send and receive data from/to many users efficiently. The heatmap should collect data and send it to the server in a short interval, the server should broadcast these messages to all users who are viewing the heatmap. In production environment the data collecting process should be seperated from the heatmap generation, n clients are tracking their data and send it to m admins who then could look at the data in real-time, but this should be clear. I thought about storing the data of each heatmap on the serverside in a data-matrix with the size of the region being observed. When a client sends some coordinates to the server, the server iterates through the data and increases the affected values by a gradient-like value. If the data-matrix reaches a value x over a defined maximum, the server should decrease all values by the difference between the maximum and x. When the serverside data-matrix manipulation is done, the server should send a “cool-down” request to all users who are looking at a real-time heatmap hence the heatmap contains only relative data, which should be representative for a usability study. The cool-down request forces the local heatmap to recalculate its alpha values, this is a resource-intensive process therefore I thought of web workers which don’t block the document while executing javascript. The following image summarizes my approach:

Ajax Push Engine as message oriented middleware

APE is a nice solution for Ajax Push, containing a server and a javascript framework. It’s easy to write server modules and implement any kind of real-time data streaming to a web browser, without having to install anything on the client-side. You can use serverside javascript for creating your own commands and message handling. Unfortunately I read that the 1.0 version still has some memory leaks but I haven’t noticed anything yet. If you are interested in learning more about APE visit their website, it’s really cool. For my approach, I wrote a serverside module containing two commands: one command for managing the maximum amount of channels and user-joins, and another one for broadcasting the heatmap data to the specific channel. If you want to send coolDown requests you have to implement a third command which sends clients the signal to execute the coolDown function.

Web workers for cooling the heatmap down

As mentioned before, the cooldown process is resource-intensive, therefore I thought about doing it with web workers. The worker script contains the coloring function but the color information is recalculated for the whole heatmap with the addition that every alpha value gets decreased by a defined value. The coolDown function just works in Firefox, when I try to execute the Web Worker in Safari or Chrome I get a “Error: SECURITY_ERR: DOM Exception 18”, couldn’t figure out what the problem is.

Further required architectural redesign

The most important architectural improvement on the heatmap script was the function-queue. Instead of drawing the alpha gradients when the mousemove events fire, the new logic alpha drawing functions get pushed with its coordinates to the queue. The script permanently iterates through the queue and removes the functions after their execution.

Why is this the most important improvement?

It’s important because when the script communicates with APE and is constantly receiveing messages for drawing new coordinates it should not affect the drawing process of other messages. The drawing process manipulates a rectangle of pixels, if the script wasn’t executing the functions simultaneously, the rectangles could overlap and use wrong data for the calculation. (e.g. a function should increment a counter, start value is 0. The first function call increments the counter, but the second function call has the data from before the first function execution was done -> counter is 1 instead of 2)

The next step?

I wrote about storing and managing heatmap data in a data-matrix on the serverside in the description of my concept. I don’t know if this was a good concept for high performance in a multi-user real-time heatmap application and I didn’t try it because I haven’t had enough time, but if anybody wants to give it a shot I’d be open for support or hacking together. Just let me know