To minimize the amount of data-shuffling I/O that occurs between the pipeline stages of a
distributed dataparallel program, its procedural code must be optimized with full awareness
of the pipeline that it executes in. Unfortunately, neither pipeline optimizers nor traditional
compilers examine both the pipeline and procedural code of a data-parallel program so
programmers must either hand-optimize their program across pipeline stages or live with
poor performance. To resolve this tension between performance and programmability, this …

To minimize the amount of data-shuffling I/O that occurs between the pipeline stages of a
distributed data-parallel program, its procedural code must be optimized with full awareness
of the pipeline that it executes in. Unfortunately, neither pipeline optimizers nor traditional
compilers examine both the pipeline and procedural code of a data-parallel program so
programmers must either hand-optimize their program across pipeline stages or live with
poor performance. To resolve this tension between performance and programmability, this …