Nicolas Crocfer

All Python developer who want to run asynchronous tasks should know Celery. If you have already used it, you know how great it is ! But you also discovered how it can be complicated to follow the state of a complex workflow. Celery Director is a tool we created at OVH to fix this problem : using some concepts of Event Sourcing, Celery Director helps us to follow the whole lifecycle of our workflows. It allows us to check when a problem occurred and relaunch the whole DAG (or just a subpart if tasks are not completely idempotent). During this talk we will introduce you the different concepts of Celery Director then we'll make a demonstration of it.

All Python developer who want to run asynchronous tasks should know Celery. If you have already used it, you know how great it is ! But you also discovered how it can be complicated to follow the state of a complex workflow. Celery Director is a tool we created at OVH to fix this problem : using some concepts of Event Sourcing, Celery Director helps us to follow the whole lifecycle of our workflows. It allows us to check when a problem occurred and relaunch the whole DAG (or just a subpart if tasks are not completely idempotent). During this talk we will introduce you the different concepts of Celery Director then we'll make a demonstration of it.

DepC is an open source tool written in Python 3.5+ : https://github.com/ovh/depc

We use it to compute the Quality Of Service (QoS) of our IT infrastructure and customers (14k servers and 5M websites in the WebHosting department).

To be able to do this we use some open source technologies : - TimeSeries databases and Pandas / Numpy for the QoS computing, - Neo4j to store the graph of our dependencies (a customer depends on several servers and some services, like databases, themselves depends on servers. If a server fails, several customers QOS are impacted), - Kafka consumer to update in real time the graph in Neo4j, - Apache Airflow to schedule the jobs used to compute the QOS of the whole infrastructure (the graph in Neo4j contains ~10M nodes).

The goal of this conference is to show you how do we assemble these technologies to construct DepC in OVH.