Topics: partitioning of large calculations, computation networks, python, celery, rabbitmq, redis
The open source OpenQuake seismic calculation engine (http://openquake.org/) is developed and maintained by the GEM foundation (http://www.globalquakemodel.org/) which is a public-private partnership that drives a collaborative effort aimed at developing and deploying tools and resources for earthquake risk assessment worldwide.
Seismic hazard calculations can be massive making it necessary to partition and distribute them across a compute network. The OpenQuake engine (http://openquake.org/) partitions seismic calculation and uses
- celery
- rabbitmq
- redis
to push the tasks into the compute network and to collect the results.
There were a number of lessons learned while scaling up OpenQuake to perform seismic calculation for the whole of Europe and I will touch upon the most important ones.