seif.codes

Log levels described as sh*t hitting the fan

2016-09-02T21:14:10+00:00

Log Level	Description
DEBUG	poop contains corn
TRACE	poop currently moving into colon
WARN	sh*t is approaching the fan
ERROR	sh*t has hit the fan
SEVERE	sh*t is spraying into the fan
CRITICAL	sh*t is spraying all over the room
ALERT	sh*t is piling up
FATAL	the room is flooding with sh*t
EMERGENCY	the fan has stopped spinning
VERBOSE	censored

By Chris Andrews, narrated by Devin Villegas

What We Talk About When We Talk About Zeitgeist

2016-06-05T19:10:00+00:00

There is a tangible confusion around as to what Zeitgeist is and what it isn’t; what it can do and what it can’t do. This is partly our own fault because we could have communicated this whole thing better, for instance we have some very outdated wiki pages lying around that you should probably stay away from until we updated them. In this post I aim to give a semi technical run down of the core Zeitgeist functionality and how we expose it for you to work with. This should hopefully clear out some confusion.

Events

The Zeitgeist daemon (also known as the engine) is a process that exposes an event logging framework as a DBus API. The structure of these events is that they have a block of metadata that describe the event itself (this is known simply as the event metadata) and another block of metadata that describes the subject, or subjects, that this event happened to (this part is known as the subject metadata). The metadata for the event looks like:

Timestamp – When did this happen. Milliseconds since the Unix Epoch. Note that we see events as single points in time, meaning that events don’t have a duration
Interpretation – Abstract interpretation of this event; what happened. Fx. “opened”, “saved”, “closed”, “send”, etc.
Manifestation – How the event happened. Fx. “user activity”, “notification”, or “scheduled activity”.
Actor – Who triggered it. This will typically point to the .desktop file of the acting application. It will most likely be an application, but it is not required to be so.
Payload – A free-form binary blob that you may attach to the event. This is specifically application specific and mainly intended to be a “back door” for people to do all sorts of funky hacks.

Each event has one or more subjects associated. For each subject we store:

URI – You guessed it! The URI of the subject
Interpretation – Abstract interpretation of the subject. This could be “Document”, “Image”, “Video”, “Email”, “Instant message”, “Contact”, anything.
Manifestation – How the subject is stored. This could be something like “File”, “Mailbox”, “Web page”.
Origin – A URI pointing to the origin or “patron” of the subject. For files this would be the parent folder. For YouTube videos it would be http://youtube.com
Mimetype – The format of the datastream representing the subject. Fx. text/plain, application/xml.
Text – Textual information added to the subject. This is not applicable for for types of subjects.
Storage – Identifier for the storage medium this subject resides on. We use this to make it possible for queries that return only events for subjects that are “available now”. Fx. some clients don’t want to show events for files that are stored on you USB pen drive when it is not connected.

Ontology – Or Data Description

In reality the metadata fields we store don’t contain simple strings like “Document” for the subject interpretation. It’s a bit more complex than that – sorry! We store a URI pointing to a formal definition of something categorized as a Document. This formal categorization is called an ontology if you want a word to confuse your friends with. We are fortunate enough that someone already wrote such a spec, namely the Nepomuk Ontology. So instead of just “Document” we store the string ”http://www.semanticdesktop.org/ontologies/2007/03/22/nfo/#Document“.

Since Tracker also uses the Nepomuk ontologies you may take these formal classification strings and plug them directly into Tracker to find everything that Tracker considers a document.

We will also have an ontology for the event metadata as this is not covered by Nepomuk. We are actively working on this.

Getting Data Out – Querying the Event Log

We employ a template based query API for searching our log data. You send us a list of event templates you want to look for and how you want it sorted and we give you the results. So if you want to find all “open” events on subjects of type “Document” simply create an Event object, set the interpretaion to “open”-event and add a subject to the event template with the interpretation set to “document”. All other fields should be left blank. Send this template to us and we will give you the matches.

The list of event templates is collected into a big OR-query to imbue the consumers with more power.

Getting Data Into Zeitgeist

There are really no limits to what kind of events we could store. If you have a spare mobile with a in-built accelerometer and glue it to your front door then you could send an event over bluetooth to your desktop each time your front door opens. Probably there are better use cases?

The point is that the usefulness of Zeitgeist stands and falls with the events that you push into us. We can store anything that you can model using the structures I outlined above. I am pretty certain that people will not agree on the kinds of data they want logged, but we are ready for anything :-)

Normal users would of course not need to think about getting their data into Zeitgeist. What developers need to know is that we have a simple DBus API to insert events (surprisingly called InsertEvents). It is called InsertEvents and not AppendEvents or something like that for a reason. Namely that you are allowed to insert events that are in the past. This is useful if you want to import your Firefox history or what ever. If you try to log an event twice the engine will throw an error at you, so no need to worry about dupes.

Ok. I think that about wraps up what I intended to say for now. Hope it’s useful to at least one person out there! :-)

Skizze: Behind the Scenes of Alpha 2

2016-03-07T19:40:57+00:00

Based on the feedback we got for our initial alpha release, we worked on improving Skizze and moving the project forward.

To recap, Skizze is a sketch data store to deal with all problems around counting and sketching using probabilistic data-structures.

My old time hacking buddy Neil Patel, who is also Xamarin Insights Technical Lead and Architect, blogged about the latest release, and also provided some background on why Skizze exists, and how to get started.

This second alpha focuses mainly on improving development and operating experience. It is an early alpha so don’t expect much, but hopefully now it is easier to run and experiment with. In this post I wanted to highlight a couple of features and ideas we have for Skizze:

Domains

As Neil mentioned in his post, we have a clear definition of Domains now:

One of the first pieces of feedback we got with the initial release was: "My data needs all four Sketches and I'm sending the values four times to the server!". Which is a fair point, in most cases the stream of values you have, you almost always want all four kinds of questions answered. Domains help solve this problem. When you create a Domain, behind the scenes, Skizze creates all four sketches of the same name. From then on, as you add values to that domain, they automatically get multiplexed into each Sketch that belongs to it.

Here is a small Mockup of how flow looks like.

Persistence

One of the main new features is the introduction of an initial functioning persistence functionality, which is done via AOF (Append Only File) inspired by Redis:

In basic terms, append-only log files keep a record of data changes that occur by writing each change to the end of the file. In doing this, anyone could recover the entire dataset by replaying the append-only log from the beginning to the end -
— Redis-Labs: 4.1.2 Append-only file persistence </blockquote> In our case we record all incoming (write) requests (serialised protobuf) to the AOF. This is all done in a go routine. Upon restart we replay them again. Currently this might be a bit slow, which is why we intend to add support for snapshotting to allow quicker startup/restoring. While we do intend to add clustering in the future, for now we are focusing on making [Skizze](https://github.com/skizzehq/skizze) work reliably on one machine. All feedback or support on implementing better persistence is welcome. ### Initial throughput benchmarks We also focused on improving the throughput complexity/speed. Thanks to [Damian Gryski](https://github.com/dgryski/)'s tips, this was achieved by using: * [faster hashing algorithms](https://github.com/dgryski/go-farm) * [faster random number generators](https://github.com/dgryski/go-pcgr) * [smarter multi-hashing tricks](http://www.eecs.harvard.edu/~michaelm/postscripts/rsa2008.pdf). And although often when working in a high-throughput environment, beefier machines are expected, benchmarking on a standard low tier VM offering should be a good reference point. To benchmark our throughput speed we used the lowest Digital Ocean Tier (5$/month) </td>

Memory 512MB Ram

Disk 20GB SSD

OS Ubuntu 14.04

CPU Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz (1Core)

Our benchmark scenario is 1 Domain with 4 sketches, CML, BloomFilter, Top-K (tracking the top 100) and HyperLogLog++ (disregards the expected capacity). The error rate is set to 1% with an expected 1000000 values capacity. The domain is subject to **10 million** insertions (**unique: 349902**) in a **Zipf (1.1)** distribution. The following are our insertions benchmark results.

batch size elapsed time values per second

1000 34s 285714

10000 29s 333333

100000 26s 370370

We will be attempting to improve the throughput with every release. I will follow up with a blog post on the "querying" accuracy, speed and resource consumption soon. ### What's next? There is still a lot to be done, and we appreciate every kind of support. For the next alpha #3 release we are focusing on: * **Adding support for Snapshots**: have snapshots of the sketches dumped to disk (while clearing the AOF), so upon restart the sketches are loaded from disk, and the delta is replayed from AOF * **Threshold functionality**: providing 100% accuracy as long as capacity under specific threshold.. * **Improving AOF**: increase AOF read and write speed. * **Fix bugs**: there are always bugs to fix. :D For alpha #n we are playing with the following ideas: * **Histograms**: by automatically partitioning sketches and domains in intervals we can answer frequency, cardinality and memberships queries via merging over the sketches for a given time range. In some cases like the Count-Min-Sketch we will be using [Hokusai](http://www.auai.org/uai2012/papers/231.pdf) concept for histograms. * **Metrics**: unlike domains which can only deal with "strings" metrics will be similar but dealing with "numbers". We want to introduce t-digest, running means, running medians, etc... ### Shout outs! A big thank you to this release's contributors: * [Martin Pinto](http://github.com/martinpinto) * [Arne Bahlo](https://arne.me) * [Manuel Barkhau](http://github.com/mbarkhau) Please feel free to open an [issue](https://github.com/skizzehq/skizze/issues), or join us for a chat in our channel on the [Gophers Slack](https://blog.gopheracademy.com/gophers-slack-community/) on **#skizze**.

Skizze progress and REPL

2016-01-18T17:41:43+00:00

Over the last 3 weeks, based on feedback we proceeded fledging out the concepts and the code behind Skizze. Neil Patel suggested the following:

So I’ve been thinking about the server API. I think we want to choose one thing and do it as well as possible, instead of having six ways to talk to the server. I think that helps to keep things sane and simple overall.

Thinking about usage, I can only really imagine Skizze in an environment like ours, which is high-throughput. I think that is it’s ‘home’ and we should be optimising for that all day long.

Taking that into account, I believe we have two options:

We go the gRPC route, provide .proto files and let people use the existing gRPC tooling to build support for their favourite language. That means we can happily give Ruby/Node/C#/etc devs a real way to get started up with Skizze almost immediately, piggy-backing on the gRPC docs etc.
We absorb the Redis Protocol. It does everything we need, is very lean, and we can (mostly) easily adapt it for what we need to do. The downside is that to get support from other libs, there will have to be actual libraries for every language. This could slow adoption, or it might be easy enough if people can reuse existing REDIS code. It’s hard to tell how that would end up.

gRPC is interesting because it’s built already for distributed systems, across bad networks, and obviously is bi-directional etc. Without us having to spend time on the protocol, gRPC let’s us easily add features that require streaming. Like, imagine a client being able to listen for changes in count/size and be notified instantly. That’s something that gRPC is built for right now.

I think gRPC is a bit verbose, but I think it’ll pay off for ease of third-party lib support and as things grow.

The CLI could easily be built to work with gRPC, including adding support for streaming stuff etc. Which could be pretty exciting. *** That being said, we gave Skizze a new home, where based on feedback we developed .proto files and started rewriting big chunks of the code.

We added a new wrapper called “domain” which represents a stream. It wraps around Count-Min-Log, Bloom Filter, Top-K and HyperLogLog++, so when feeding it values it feeds all the sketches. Later we intend to allow attaching and detaching sketches from “domains” (We need a better name).

We also implemented a gRPC API which should allow easy wrapper creation in other languages.

Special thanks go to Martin Pinto for helping out with unit tests and Soren Macbeth for thorough feedback and ideas about the “domain” concept. Take a look at our initial REPL work there:

Skizze - A probabilistic data-structures service and storage (Alpha)

2015-12-25T09:53:21+00:00

At my day job we deal with a lot of incoming data for our product, which requires us to be able to calculate histograms and other statistics on the data-stream as fast as possible.

One of the best tools for this is Redis, which will give you 100% accuracy in O(1) (except for its HyperLogLog implementation which is a probabilistic data-structure). All in all Redis does a great job. The problem with Redis for me personally is that, when using it for 100 of millions of counters, I could end up with Gigabytes of memory.

I also tend to use Top-K, which is not implemented in Redis but via Lua scripting can be built on top of the ZSet data-structure. The Top-K data-structure is used to keep track of the top “k” heavy hitters in a stream without having to keep track all “n” flows (k < n), with a O(1) complexity.

Anyhow, dealing with a massive amount of data the interest is most of the time in heavy hitters, that could be estimated while using less memory with an O(1) complexity for reading and writing (that is if you don’t care about a count being 124352435 or 124352011 because on the UI of an app you will be showing “over 124 Million”).

There are a lot of algorithms floating around and used to solve counting, frequency, membership and top-k problems, which in practice are implemented and used as part of a data-stream pipeline where stuff is counted, merged then stored.

I couldn’t find a one-stop-shop service to fire & forget my data at.

Basically in need of a solution where I can set up sketches to answer cardinality, frequency, membership as well as ranking queries about my data-stream (without having to reimplement the algorithm in a pipeline embedded in storm, spark, etc…) led to the development of Skizze (which is in alpha state).

##What is Skizze?

Skizze ([ˈskɪt͡sə]: german for sketch) is a probabilistic data-structures (sketch) service & store to deal with all problems around counting and sketching using probabilistic data-structures. (https://github.com/seiflotfy/skizze)

Unlike a Key-Value store, Skizze does not store values, but rather appends values to sketches, to solve frequency and cardinality queries in near O(1) time, with minimal memory footprint.

##Which data structures are supported? Currently the following data structures are supported:

HyperLogLog++ to query cardinality of values in the sketch.
Count-Min-Log Sketch to query frequency of values in the sketch.
Top-K to list the top k values in the sketch.
Bloom Filter query membership of a value in the sketch.
Dictionary to 100% accurately query membership and frequency of values in the sketch.

##What are upcoming data structures Soon we intend to implement/integrate the following sketches:

Count-min Sketch frequency counting with merging ability
Hokusai the exponential histogram extension to Count-min Sketch
Probabilistic Multiplicity Counting similar to Count-min Sketch
Self-learning Bitmaps cardinality counting like HLL++

##How to use? Skizze runs as a single service for now, and exposes a Restful API.

##Who helped out? I’d like to thank the following contributors who helped develop this project:

Manuel Barkhau (coding all over the place)
Martin Pinto-Bazurco (coding all over the place)
Damian Gryski (provided me with good reading material and reusable code)
Keith Ballinger (implementing the standard dictionary)

##What else? The project is in alpha state, and we intend to improve it in every possible way, e.g:

Benchmarks to test data structures against each other.
Referencing algorithms, instead of copying into the local source (once Go 1.6’s vendoring lands)
Storage currently Skizze writes to disc after n seconds or m operations. Soon I’d like to be able to write only dirty segments of the sketch to disc (in case a sketch is large e.g: 1 GB).

Feel free to open issues or help out with more specs development of the project on GitHub. All input appreciated.

Counting flows (Semi-evaluation of CMS, CML and PMC)

2015-08-31T22:25:58+00:00

Assume we have a stream of events coming in one at a time, and we need to count the frequency of the different types of events in the stream.

In other words: We are receiving fruits one at a time in no given order, and at any given time we need to be able to answer how many of a specific fruit did we receive.

The most naive implementation is a dictionary in the form of <string, int>, and is most accurate and suitable for streams with limited types of events.

Let us assume a unique item consists of 15 bytes and has a dedicated uint32 (4 bytes) counter assigned to it.

At 10 million unique items we end up using 19 MB which is a bit much, but on the plus side its as accurate as it gets.

But what if we don’t have the 19 MB. Or what if we have to keep track of several streams?

Maybe saving to a DB? Well when querying the DB upon request, something in the lines of:

SELECT count(event) WHERE event = ?)

The more items we add, the more resource intensive the query becomes.

Thankfully solutions come in the form of Probabalistic datastructures (sketches).

I won’t get into details but to solve this problem I semi-evaluated the following data structures:

Count-Min sketch (CMS) [1][2]
Count-Min-Log sketch (CML) [1][2]
Probabilistic Multiplicity Counting sketch (PMC) [1]

Test details:

For each sketch I linearly added a new flow with equivalently linear events. So the first flow got 1 event inserted. The second flow for 2 events inserted, all the way up to 10k-th flow with 10k events inserted.

flow 1: 1 event
flow 2: 2 events
...
flow 10000: 10000 events

All three data structures were configured to have a size of 217KB (exactly 1739712 bits).

A couple dozen runs yielded the following results (based on my unoptimized code esp. for PMC and CML)

CMS: 07s for 50005000 insertion (fill rate: 31%)
CML: 42s for 50005000 insertion (fill rate: 09%)
PMC: 18s for 50005000 insertion (fill rate: 54%)

CMS with ɛ: 0.0001, δ: 0.99 (code)

Observe the biased estimation of CMS. CMS will never underestimate. In our case looking at the top border of the diagram we can see the there was a lot of overestimation.

CML with ɛ: 0.000025, δ: 0.99 (16-bit counters) (code)

Just like CMS, CML is also biased and will never underestimate. However unlike CMS the top border of the diagram is less noisy. Yet accuracy seems to be decreasing for the high counting flows.

PMC with (256x32) virtual metrices (code)

Unlike the previous two sketches. This sketch is unbiased, so underestimations exist. Also the estimate flow count increases with the actual flow count (linearly bigger errors). The drawback here is that PMC gets filled up very quickly which means at some point it will just have everything overestimated. It is recommended to know what the max num of different flows will be beforehand.

Bringing it all together ![ALL] (https://dl.dropboxusercontent.com/u/7162902/all.png)

So what do you think. If you are familiar with these algorithms or can propose a different benchmarking scenario, please comment. I might be able to work on that on a weekend. The code was all written in Go, feel free to suggest optimizations of fix any bugs found (links above respective plots).

Ich bin ein Xamarin(er) ♥

2014-04-15T18:30:22+00:00

As of the beginning of the April I am a Xamarin (that is what Xamarin employees call themselves).

At Xummit I met the rest of the Xamarins and I had an incredible time there (dare I say magical ♥). I met old friends like Rodrigo Moya, Jason Smith, David Siegel, Cody Russell, Neil Patel, Connor Curran, Gord Allot and others, but also made new friends:

Zack Gramana: The right amount of crazy and creative. He is helping me with my new pet project.
Seth Rosetter: SF chilled out hacker with an ear for techno and extreme positive attitude, a delight to hang out with.
Mike Krüger: One of the friendliest people I got to meet and know with exactly my kind of humour.
Victoria Grothey: Incredibly nice person with lots of energy and always smiling.
Marek Safar: The most passionate beer expert I know I guess. Also rumour has it that either I am stalking him or he is stalking me.
Václav Vančura: An awesome designer who motivated me to start drawing again. Thanks for that. And many many more.

One thing I believe in, is that interpersonal relationships between co-workers is a must for a community or a company to be productive and successful. Xamarin promoted (and still promotes) this positive habit, achieved it and even more. The upbeat attitude and enthusiasm at Xamarin is infectious. Combined with the diversity in culture as well as stuff/tasks to do brings the best out of Xamarins. I will not forget the bus ride to the venue. 8 people with 7 different nationalities, but all happy and psyched about what they are doing and what others are doing ♥.

Since I joined Xamarin I started doing more Mono in my free time too. Currently I am porting

Synapse to Mac (since I loved the interface and some of the functionalities I couldn’t find in Alfred and Quicksilver). Here is a small very early sneak peak :)

I am loving Xamarin and all its stands for and brings to the table.

P.S: Hylke Bons has a fan base here at Xamarin :)

Marconi love :)

2014-03-24T02:23:44+00:00

I have lots of positive things to say about the OpenStack community. They managed to create a great mixture of professional yet agile and FLOSSy. Amazing CI/CD and devstack is just an awesome way to kickstart development on OpenStack.

Last month I discovered Marconi (https://wiki.openstack.org/wiki/Marconi)…

Basically its a new MQ written in pure Python and allos different backends. The awesome thing is you can run it without having to run the whole openstack stuff and it works… It is still a young project but the community around it its awesome ans their IRC #openstack-marconi is very responsive. These guys could use a hand so if you are a Python enthusiast or want to hack on something OpenStack related without needing to have the whole stack running, please show some support and try to help out. There is a lot of low hanging fruit (https://bugs.launchpad.net/openstack/marconi) :)

Sc(r)um

2014-03-22T20:01:40+00:00

I am entitled to my opinion and in my honest opinion: scrum sucks big time

Here is why (again my opinion).

Does not empower developers to become creative. Good way to sell that the developers are doing something (so that failure is not their fault) even if running around in circles ==> plausible deniability
Highly motivated developers are limited from jumping out of the “plan” to hack on other project related stuff (even doing so in free time is regarded wrong since you are not sticking to a plan). Thus limiting the personal success story and relying on motivation to come from team achievements. So being a weak link in a team just makes matters worse.
The “Daily Stand-up” routine is not productive in its defined structure of:
- What did I accomplish yesterday? Tried to finish a story. Ran out of time.
- What will I do today? Will try to finish this story.
- What obstacles are impeding my progress? This meeting, the circle jerk around legacy code and the dude demanding me to discuss every merge request before submitting it. (Just accept/deny with a comment - thank you ;) )
Stories over quality: Developers are keen to finish as much stories as possible which could to take a toll on the quality.
Not applicable on a distributed community of volunteers, or when dependency on open source community exist.
Time-boxed problem solving does not guarantee best possible outcome.
The commitment: One of the reason Scrum fails is its approach on commitment. “Deliver the maximum value of stories in a specific time frame”. As soon as the team can not agree on committing to a story all other stories with lower priority are ignored. And once done with those stories you pick the one out of the backlog that you couldn’t agree on (Since it has the highest priority) and waste time on it. You can’t skip this story since it has the highest priority. To me this is like sitting in an exam and being stuck on a problem. Instead of moving to the next problem with the hope of gathering as much points as possible, one decided to be stubborn and try fucking with that problem. That is just stupid. Now assume I solved the problem. Scrum now forces me to pick something from the backlog. And not anything. But the one with the highest priority and again waste time being stuck on that until the time ends. If would have adapted this style of tackling problems during school I would have never passed.

Don’t get me wrong Scrum can work don’t know how personally but I do believe it can work with the right team and people, however what are the chances of it succeeding if 2 or more of the following apply:

You don’t have full control of the stack (dependency on third party)
Your developers are not all on the same level of expertise and knowledge.
Shoehorned into an ongoing running process and code base.
Stories of the team are not focused on a specific topic but on the general delivery of the whole project.

I look at GNOME, Mozilla and the Kernel. They don’t follow this methodology and stuff happens and gets delivered. So much big companies can learn from FLOSS.

Moving on…

2014-03-20T23:57:43+00:00

The last year has been a rough year. I started working for Deutsche Telekom (with a bunch of other FLOSS developers) on OpenStack. What started off as developing new features upstream ended up being more or less devops stuff (mainly Puppet and orchestration) which is not really my style.

During that time I was working on GNOME Music, I enjoyed developing it a lot, the students and the team are just amazing but I was growing unhappy with the external micromanagement that I experienced which personally stressed me and made me feel unappreciated for what I think I can bring to the table. What started off as a hobby became a burden, so at GUADEC I decided to spend time away from GNOME. Strangely enough I started feeling better, I took things a step further and switched to Mac and noticed what a long road we have for the Linux desktop. I became more convinced that the focus of delivering an experience for the desktop or mobile market is misplaced and we missed both trains. The only way for GNOME to continue being relevant is to either start focusing on wearable experience or focus on marketing the technology first then the experience. Get GNOME tech on other platforms and wearables.

Anyhow being unhappy can take a toll on the mental and physical health, I became a bit depressed and negative person and my health was declining. I went through this a couple of years ago when my father passed away and found an escape contributing to GNOME and FLOSS. So with that some decision had to be taken.

Somehow I can’t see myself actively contributing to GNOME anymore and maybe its for the best. I’ll be around and will help out when I can if asked directly. But I won’t be running GNOME actively.

I am also taking time off contributing to Mozilla. However I do see myself coming back soon since I am missing it already. I really enjoy the compatibility team (where we use some GNOME technologies including GI) and the community building team. I will continue maintaining Zeitgeist but there will be no active feature development. After almost 6 years we all moved on and Zeitgeist is feature complete.

I quit my job. I am on a nice vacation for a bit and will be starting my new job sooner than expected. With a help of friends and my girl-friend I started getting my self-esteem back and all is better again.

Feeling better already :D (No depression)