The massive size of the data in large graph processing requires distributed processing. However, conventional frameworks for distributed graph processing, such as Pregel, use programming models that are well-suited for scalability but inconvenient for programming graph algorithms. In this talk, we demonstrate how to use Green-Marl, a Domain-Specific Language for graph analysis, to describe graph algorithms intuitively and extend its compiler to generate equivalent Pregel programs. Using the semantic information exposed by Green-Marl, the compiler applies the same kinds of transformation rules that programmers apply when manually implementing graph algorithms with Pregel. Our experiments show that the Pregel programs generated by Green-Marl compiler perform similarly to native Pregel implementations of the same algorithms. The compiler is even able to generate a Pregel implementation of a complicated graph algorithm whose native Pregel implementation is very challenging.
The massive size of the data in large graph processing requires distributed processing. However, conventional frameworks for distributed graph processing, such as Pregel, use programming models that are well-suited for scalability but inconvenient for programming graph algorithms. In this talk, we demonstrate how to use Green-Marl, a Domain-Specific Language for graph analysis, to describe graph algorithms intuitively and extend its compiler to generate equivalent Pregel programs. Using the semantic information exposed by Green-Marl, the compiler applies the same kinds of transformation rules that programmers apply when manually implementing graph algorithms with Pregel. Our experiments show that the Pregel programs generated by Green-Marl compiler perform similarly to native Pregel implementations of the same algorithms. The compiler is even able to generate a Pregel implementation of a complicated graph algorithm whose native Pregel implementation is very challenging.
The first paper describing Green-Marl was published at ASPLOS 2012. http://arsenalfc.stanford.edu/kogroup/papers/asplos2012.pdf We have extended our compiler to now generate Giraph code in addition to our c++ backend. The source code is available here: https://github.com/stanford-ppl/Green-Marl
Speakers: Jan Van Der Lugt