All Python developer who want to run asynchronous tasks should know Celery. If you have already used it, you know how great it is ! But you also discovered how it can be complicated to follow the state of a complex workflow. Celery Director is a tool we created at OVH to fix this problem : using some concepts of Event Sourcing, Celery Director helps us to follow the whole lifecycle of our workflows. It allows us to check when a problem occurred and relaunch the whole DAG (or just a subpart if tasks are not completely idempotent). During this talk we will introduce you the different concepts of Celery Director then we'll make a demonstration of it.
All Python developer who want to run asynchronous tasks should know Celery. If you have already used it, you know how great it is ! But you also discovered how it can be complicated to follow the state of a complex workflow. Celery Director is a tool we created at OVH to fix this problem : using some concepts of Event Sourcing, Celery Director helps us to follow the whole lifecycle of our workflows. It allows us to check when a problem occurred and relaunch the whole DAG (or just a subpart if tasks are not completely idempotent). During this talk we will introduce you the different concepts of Celery Director then we'll make a demonstration of it.
Speakers: Nicolas Crocfer