Dynamically scheduling a component-based framework in clusters

A. Kuzmanovska, R.H. Mak, D.H.J. Epema

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

5 Citations (Scopus)
1 Downloads (Pure)

Abstract

In many clusters and data centers, application frameworks are used that offer programming models such as Dryad and MapReduce, and jobs submitted to the clusters or data centers may be targeted at specific instances of these frameworks, for example because of the presence of certain data. An important question that then arises is how to allocate resources to framework instances that may have highly fluctuating workloads over their lifetimes. Static resource allocation, a traditional approach for scheduling jobs, may result in inefficient resource allocation because of poor resource utilization during off-peak hours. We address this issue with a strategy for the dynamic deployment of a component-based framework by extending a resource manager responsible for scheduling jobs in multi-cluster environments. This extension allows scheduling multiple concurrent instances of the framework as long-running utility jobs that share computational resources of the cluster. In order to accommodate the fluctuating resource demands of frameworks, we consider two provisioning policies for dynamic resource allocation: OnDemand and Proactive provisioning. We evaluate the effectiveness of both policies by comparing them with static resource allocation on the das4 multi-cluster system. Our results show that dynamic resource allocation gives at least 30 % improvement over the static resource allocation with respect to both the utilization of the resources and the reject rate of the applications within the framework.
Original languageEnglish
Title of host publicationJob Scheduling Strategies for Parallel Processing (18th International Workshop, JSSPP 2014, Phoenix AZ, USA, May 23, 2014. Revised Selected Papers)
EditorsW. Cirne, N. Desai
Place of PublicationHeidelberg
PublisherSpringer
Pages129-146
ISBN (Print)978-3-319-15788-7
DOIs
Publication statusPublished - 2015
Eventconference; 18th International Workshop on Job Scheduling Strategies for Parallel Processing; 2014-05-23; 2014-05-23 -
Duration: 23 May 201423 May 2014

Publication series

NameLecture Notes in Computer Science
Volume8828
ISSN (Print)0302-9743

Conference

Conferenceconference; 18th International Workshop on Job Scheduling Strategies for Parallel Processing; 2014-05-23; 2014-05-23
Period23/05/1423/05/14
Other18th International Workshop on Job Scheduling Strategies for Parallel Processing

Fingerprint

Dive into the research topics of 'Dynamically scheduling a component-based framework in clusters'. Together they form a unique fingerprint.

Cite this