Modelling Process and Staleness

Some interesting talks at QCon

I went to some excellent talks last night at the LJC night at QCon. Gil Tene of Azul systems and Adrian Cockroft of Netflix both gave enlightening and useful talks. The talk "Modelling Process" by the creator of Clojure, Rich Hickey, was especially interesting to me as it touched on several important themes.

Rich talked (amongst other things) about how processes 'perceive' data values and never the actual value. He compared the way processor and memory caches work to the delay that a person gets when listening or watching a distant event due to sound and light waves taking time to travel.

I think this is a very important point. I would say that ALL data is stale as you only ever see a value at a point of time in the past. Now this may not matter at all for some data (the data may not actually be changing) or it can be incredibly important (real time prices for trading systems). How stale the data is before you process it will also vary greatly - from users sending in changes to their information via hand-written forms to electronic sensors with a direct connection.

Rich's point (and I hope I'm not misrepresenting him here) was that it's OK in most instances to send an immutable copy of required data to processes rather than insist that they always use the 'real' value in memory. This helps avoid locking and contention. In fact the consumers of the information may see a much more realistic value as they are not waiting behind 30 other consumers who are taking it in turns to lock, read and use.

Understanding your staleness requirements can lead to different architectural decisions.

His presentation is worth a read and I really liked his "Epochal Time Model".

About the author

Robert Annett Robert works in financial services and has spent many years creating and maintaining trading systems. He knows far more about low latency data systems and garbage collection than is good for anyone. He likes to think of himself as a pragmatist who loves technology but uses what's appropriate rather than what's cool.

When not pouring over data connections or tormenting interviewees with circular reference questions, Robert can be found locked in his shed with an impressive collection of woodworking tools.

E-mail : robert.annett at

Add a comment Send a TrackBack