KOALA-F : a resource manager for scheduling frameworks in clusters

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

9 Citations (Scopus)

Abstract

Due to the diversity in the applications that run in clusters, many different application frameworks have been developed, such as MapReduce for data-intensive batch jobs and Spark for interactive data analytics. A framework is first deployed in a cluster, and then starts executing a large set of jobs that are submitted over time. When multiple such frameworks with time-varying resource demands are presentin a single cluster, static allocation of resources on a per-framework basis leads to low system utilization and resource fragmentation. In this paper, we present koala-f, a resource manager that dynamically provides resources to frameworks by employing a feedback loop to collecttheir possibly different performance metrics. Frameworks periodically -- not necessarily with the same frequency -- report the values of their performancemetrics to koala-f, which then either rebalances their resources individuallyagainst the idle-resource pool, or, when the latter is empty, rebalances their resources amongst them. We demonstrate the effectiveness of koala-f with experiments in a real system.
Original languageEnglish
Title of host publication2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, 16-19 May 2016, Cartagena, Columbia
Place of PublicationPiscataway
PublisherInstitute of Electrical and Electronics Engineers
Pages80-89
ISBN (Print)978-1-5090-2452-0
DOIs
Publication statusPublished - 16 May 2016
Event16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May 16-19, 2016, Cartagena, Colombia - Cartagena, Colombia
Duration: 16 May 201619 May 2016

Conference

Conference16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May 16-19, 2016, Cartagena, Colombia
Abbreviated titleCCGrid 2016
Country/TerritoryColombia
CityCartagena
Period16/05/1619/05/16

Fingerprint

Dive into the research topics of 'KOALA-F : a resource manager for scheduling frameworks in clusters'. Together they form a unique fingerprint.

Cite this