pretix – behind the scenes

We recently deployed Loki to our infrastructure and in this blog post want to share the pitfalls and tips we discovered.

Hey there, we’re hiring! Right now, we’re still only looking for people fluent in German, so please excuse us for writing this in German.

Hey there, we’re hiring! Right now, we’re still only looking for people fluent in German, so please excuse us for writing this in German.

Hey there, we’re hiring! Right now, we’re still only looking for people fluent in German, so please excuse us for writing this in German.

It’s been a while since we shared the story of an incident with you, and that’s probably a good thing – most operational incidents we had in the past year were “boring” enough in nature to fix them easily. This time, we’ve got a story of a data loss, caused by pure and simple human error – and the story of how we recovered the data.

Hey there, we’re hiring! Right now, we’re still only looking for people fluent in German, so please excuse us for writing this in German.

Last night, there was a disruption in the network of the data center our servers run in. We do not yet know the source or exact nature of the disruption, but it probably caused increased latencies and error rates in the connection of our application servers to our database servers. This alone would not be a problem in itself, but it caused a series of other problems that caused a full outage of our system starting approximately at 6:57 CET. I was alerted about the issue (and woken up) at 07:02 CET. At 7:04 CET, the system was already available again, but three of our four application servers were still not working.

Hey there, we’re hiring! Right now, we’re still only looking for people fluent in German, so please excuse us for writing this in German.

pretix is a multi-tenant application: With one software installation, it can handle lots of companies and institutions selling tickets. In pretix, they are called organizers, but in the more general case, we usually speak of “tenants” in the software industry. Building pretix this way is a design choice, we could just as well have created a software that only handles one company and run the software many times on logically or physically seperate systems for every event organizer. We decided to go with multi-tenancy in the software many years ago for a number of reasons.

From the very beginning, privacy has been one of the most important design goals of pretix. Privacy is usually best and easiest achieved by just not collecting any unnecessary data that and we give our very best to live by that standard.

Both our documentation as well as our website contain a number of screenshots of our software. Taking these screenshots manually is really tedious, since you first need to populate your database with sensible test data, select a proper display resolution and then take screenshots separately for every language our software supports, so we can use them in the localized websites properly.

We’re looking for a Sales Manager. For now, we’re looking for someone who is fluent in German, so please excuse us for writing this post in German. Thank you!

pretix comes with a test suite of currently 2167 tests. When executing these tests, 81 % of pretix’ codebase is run. These tests are intended to verify that pretix is operating correctly and errors and regressions are spotted early. Having such a test suite saved us from a lot of major problems in the past and will hopefully continue to do so in the future.

In my last blogpost, I described our hosting setup for pretix.eu in detail and talked about the efforts we take to achieve a resistance against failing servers: The system should tolerate the failure of any single server at any given time and keep running. Well, up to now, this was only nearly true.

This night, pretix.eu had a planned outage for around 70 minutes to allow us to make a fundamental change to our service architecture. In this blogpost, we want to go into detail on what we changed and why. We hope this might be insightful for you if you run a similar service or if you are just curious about the challenges we experience along the way.