A coflow-based co-optimization framework for high-performance data analytics

L. Cheng, Y. Wang, Y. Pei, D.H.J. Epema

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureConferentiebijdrageAcademicpeer review

12 Citaten (Scopus)


Efficient execution of distributed database operators such as joining and aggregating is critical for the performance of big data analytics. With the increase of the compute speedup of modern CPUs, reducing the network communication time of these operators in large systems is becoming increasingly important, and also challenging current techniques. Significant performance improvements have been achieved by using state-of-the-art methods, such as reducing network traffic designed in the data management domain, and data flow scheduling in the data communications domain. However, the proposed techniques in both fields just view each other as a black box, and performance gains from a co-optimization perspective have not yet been explored.In this paper, based on current research in coflow scheduling, we propose a novel Coflow-based Co-optimization Framework (CCF), which can co-optimize application-level data movement and network-level data communications for distributed operators, and consequently contribute to their performance in large distributed environments. We present the detailed design and implementation of CCF, and conduct an experimental evaluation of CCF using large-scale simulations on large data joins. Our results demonstrate that CCF can always perform faster than current approaches on network communications in large-scale distributed scenarios.

Originele taal-2Engels
Titel2017 46th International Conference on Parallel Processing (ICPP), 14-17 August 2017, Bristol, United Kingdom
Plaats van productiePiscataway
UitgeverijInstitute of Electrical and Electronics Engineers
Aantal pagina's10
ISBN van elektronische versie978-1-5386-1042-8
ISBN van geprinte versie978-1-5386-1043-5
StatusGepubliceerd - 1 sep. 2017
Evenement46th International Conference on Parallel Processing, ICPP 2017 - Bristol, Verenigd Koninkrijk
Duur: 14 aug. 201717 aug. 2017


Congres46th International Conference on Parallel Processing, ICPP 2017
Land/RegioVerenigd Koninkrijk


Duik in de onderzoeksthema's van 'A coflow-based co-optimization framework for high-performance data analytics'. Samen vormen ze een unieke vingerafdruk.

Citeer dit