Kunft, Andreas; Stadler, Lukas; Bonetta, Daniele; Basca, Cosmin; Meiners, Jens; Breß, Sebastian; Rabl, Tilmann; Fumero, Juan José; Markl, Volker
Proceedings of the ACM Symposium on Cloud Computing, SoCC 2018, Carlsbad, CA, USA, October 11-13, 2018
To cope with today’s large scale of data, parallel dataflow enginessuch as Hadoop, and more recently Spark and Flink, have beenproposed. They offer scalability and performance, but require datascientists to develop analysis pipelines in unfamiliar programminglanguages and abstractions. To overcome this hurdle, dataflow en-gines have introduced some forms of multi-language integrations,e.g., for Python and R. However, this results in data exchange be-tween the dataflow engine and the integrated language runtime,which requires inter-process communication and causes high run-time overheads. In this paper, we present ScootR, a novel approachto execute R in dataflow systems. ScootR tightly integrates thedataflow and R language runtime by using the Truffle frameworkand the Graal compiler. As a result, ScootR executes R scripts di-rectly in the Flink data processing engine, without serialization andinter-process communication. Our experimental study reveals thatScootR outperforms state-of-the-art systems by up to an order ofmagnitude.