KOALA-F : a resource manager for scheduling frameworks in clusters

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureConferentiebijdrageAcademicpeer review

6 Citaties (Scopus)

Uittreksel

Due to the diversity in the applications that run in clusters, many different application frameworks have been developed, such as MapReduce for data-intensive batch jobs and Spark for interactive data analytics. A framework is first deployed in a cluster, and then starts executing a large set of jobs that are submitted over time. When multiple such frameworks with time-varying resource demands are presentin a single cluster, static allocation of resources on a per-framework basis leads to low system utilization and resource fragmentation. In this paper, we present koala-f, a resource manager that dynamically provides resources to frameworks by employing a feedback loop to collecttheir possibly different performance metrics. Frameworks periodically -- not necessarily with the same frequency -- report the values of their performancemetrics to koala-f, which then either rebalances their resources individuallyagainst the idle-resource pool, or, when the latter is empty, rebalances their resources amongst them. We demonstrate the effectiveness of koala-f with experiments in a real system.
TaalEngels
Titel2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, 16-19 May 2016, Cartagena, Columbia
Plaats van productiePiscataway
UitgeverijInstitute of Electrical and Electronics Engineers
Pagina's80-89
ISBN van geprinte versie978-1-5090-2452-0
DOI's
StatusGepubliceerd - 16 mei 2016
Evenement16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May 16-19, 2016, Cartagena, Colombia - Cartagena, Colombia
Duur: 16 mei 201619 mei 2016

Congres

Congres16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May 16-19, 2016, Cartagena, Colombia
Verkorte titelCCGrid 2016
LandColombia
StadCartagena
Periode16/05/1619/05/16

Vingerafdruk

Managers
Scheduling
Electric sparks
Feedback
Experiments

Citeer dit

Kuzmanovska, A., Mak, R. H., & Epema, D. (2016). KOALA-F : a resource manager for scheduling frameworks in clusters. In 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, 16-19 May 2016, Cartagena, Columbia (blz. 80-89). Piscataway: Institute of Electrical and Electronics Engineers. DOI: 10.1109/CCGrid.2016.60
Kuzmanovska, Aleksandra ; Mak, Rudolf H. ; Epema, Dick. / KOALA-F : a resource manager for scheduling frameworks in clusters. 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, 16-19 May 2016, Cartagena, Columbia. Piscataway : Institute of Electrical and Electronics Engineers, 2016. blz. 80-89
@inproceedings{eb6165c567ea44a8ab38eae6241eb889,
title = "KOALA-F : a resource manager for scheduling frameworks in clusters",
abstract = "Due to the diversity in the applications that run in clusters, many different application frameworks have been developed, such as MapReduce for data-intensive batch jobs and Spark for interactive data analytics. A framework is first deployed in a cluster, and then starts executing a large set of jobs that are submitted over time. When multiple such frameworks with time-varying resource demands are presentin a single cluster, static allocation of resources on a per-framework basis leads to low system utilization and resource fragmentation. In this paper, we present koala-f, a resource manager that dynamically provides resources to frameworks by employing a feedback loop to collecttheir possibly different performance metrics. Frameworks periodically -- not necessarily with the same frequency -- report the values of their performancemetrics to koala-f, which then either rebalances their resources individuallyagainst the idle-resource pool, or, when the latter is empty, rebalances their resources amongst them. We demonstrate the effectiveness of koala-f with experiments in a real system.",
author = "Aleksandra Kuzmanovska and Mak, {Rudolf H.} and Dick Epema",
year = "2016",
month = "5",
day = "16",
doi = "10.1109/CCGrid.2016.60",
language = "English",
isbn = "978-1-5090-2452-0",
pages = "80--89",
booktitle = "2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, 16-19 May 2016, Cartagena, Columbia",
publisher = "Institute of Electrical and Electronics Engineers",
address = "United States",

}

Kuzmanovska, A, Mak, RH & Epema, D 2016, KOALA-F : a resource manager for scheduling frameworks in clusters. in 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, 16-19 May 2016, Cartagena, Columbia. Institute of Electrical and Electronics Engineers, Piscataway, blz. 80-89, Cartagena, Colombia, 16/05/16. DOI: 10.1109/CCGrid.2016.60

KOALA-F : a resource manager for scheduling frameworks in clusters. / Kuzmanovska, Aleksandra; Mak, Rudolf H.; Epema, Dick.

2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, 16-19 May 2016, Cartagena, Columbia. Piscataway : Institute of Electrical and Electronics Engineers, 2016. blz. 80-89.

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureConferentiebijdrageAcademicpeer review

TY - GEN

T1 - KOALA-F : a resource manager for scheduling frameworks in clusters

AU - Kuzmanovska,Aleksandra

AU - Mak,Rudolf H.

AU - Epema,Dick

PY - 2016/5/16

Y1 - 2016/5/16

N2 - Due to the diversity in the applications that run in clusters, many different application frameworks have been developed, such as MapReduce for data-intensive batch jobs and Spark for interactive data analytics. A framework is first deployed in a cluster, and then starts executing a large set of jobs that are submitted over time. When multiple such frameworks with time-varying resource demands are presentin a single cluster, static allocation of resources on a per-framework basis leads to low system utilization and resource fragmentation. In this paper, we present koala-f, a resource manager that dynamically provides resources to frameworks by employing a feedback loop to collecttheir possibly different performance metrics. Frameworks periodically -- not necessarily with the same frequency -- report the values of their performancemetrics to koala-f, which then either rebalances their resources individuallyagainst the idle-resource pool, or, when the latter is empty, rebalances their resources amongst them. We demonstrate the effectiveness of koala-f with experiments in a real system.

AB - Due to the diversity in the applications that run in clusters, many different application frameworks have been developed, such as MapReduce for data-intensive batch jobs and Spark for interactive data analytics. A framework is first deployed in a cluster, and then starts executing a large set of jobs that are submitted over time. When multiple such frameworks with time-varying resource demands are presentin a single cluster, static allocation of resources on a per-framework basis leads to low system utilization and resource fragmentation. In this paper, we present koala-f, a resource manager that dynamically provides resources to frameworks by employing a feedback loop to collecttheir possibly different performance metrics. Frameworks periodically -- not necessarily with the same frequency -- report the values of their performancemetrics to koala-f, which then either rebalances their resources individuallyagainst the idle-resource pool, or, when the latter is empty, rebalances their resources amongst them. We demonstrate the effectiveness of koala-f with experiments in a real system.

U2 - 10.1109/CCGrid.2016.60

DO - 10.1109/CCGrid.2016.60

M3 - Conference contribution

SN - 978-1-5090-2452-0

SP - 80

EP - 89

BT - 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, 16-19 May 2016, Cartagena, Columbia

PB - Institute of Electrical and Electronics Engineers

CY - Piscataway

ER -

Kuzmanovska A, Mak RH, Epema D. KOALA-F : a resource manager for scheduling frameworks in clusters. In 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, 16-19 May 2016, Cartagena, Columbia. Piscataway: Institute of Electrical and Electronics Engineers. 2016. blz. 80-89. Beschikbaar vanaf, DOI: 10.1109/CCGrid.2016.60