Zeitgeist scalability & efficiency bootcamp results

As Trever blogged yesterday, the Zeigeist team has been busy with tweaking the DB and the engine. During that process tools and benchmarks have been developed to make the tweaking and testing more interesting. Trever will be blogging about that tomorrow so make sure to check his blog.

Our end goal is  trying to scale the engine to be able to handle a few billion events just as fast as it can handle a few hundred thousand. While we are not there yet we managed to have some pretty nice stable results for the first iteration. A lot of results show more than 100% speed enhancement. In other words a lot of queries from our standard benchmarks now consume more than 50% less time to execute. Here are some graphs of our benchmarks.

Green indicates the 0.9 release

Yellow indicates the new trunk

Most notable performance enhancement is querying Zeitgeist with a specified timeframe (from data x to date y).

 

Same queries with an open timeframe also improved

 

We also have a copy of the Synapse queries benchmarked

The queries here are typical queries used to extract info from Zeitgeist. So right now the team is really happy with the initial results. For Synapse on my local DB (over a year old), all my synapse queries perform under 0.08 seconds. We still can get more out of this. The trick here was improving our indexes and our sql query generator.

Next month we will be going through another iteration.