conferences | speakers | series

Follow Your Celery Tasks

home

Follow Your Celery Tasks
FOSDEM 2020

All Python developer who want to run asynchronous tasks should know Celery. If you have already used it, you know how great it is ! But you also discovered how it can be complicated to follow the state of a complex workflow. Celery Director is a tool we created at OVH to fix this problem : using some concepts of Event Sourcing, Celery Director helps us to follow the whole lifecycle of our workflows. It allows us to check when a problem occurred and relaunch the whole DAG (or just a subpart if tasks are not completely idempotent). During this talk we will introduce you the different concepts of Celery Director then we'll make a demonstration of it.

All Python developer who want to run asynchronous tasks should know Celery. If you have already used it, you know how great it is ! But you also discovered how it can be complicated to follow the state of a complex workflow. Celery Director is a tool we created at OVH to fix this problem : using some concepts of Event Sourcing, Celery Director helps us to follow the whole lifecycle of our workflows. It allows us to check when a problem occurred and relaunch the whole DAG (or just a subpart if tasks are not completely idempotent). During this talk we will introduce you the different concepts of Celery Director then we'll make a demonstration of it.

Speakers: Nicolas Crocfer