Performance analysis of dynamic workflow scheduling in multicluster grids

O.O. Sonmez, M.N. Yigitbasi, S. Abrishami, A. Iosup, D.H.J. Epema

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    25 Citations (Scopus)

    Abstract

    Scientists increasingly rely on the execution of workflows in grids to obtain results from complex mixtures of applications. However, the inherently dynamic nature of grid workflow scheduling, stemming from the unavailability of scheduling information and from resource contention among the (multiple) workflows and the non-workflow system load, may lead to poor or unpredictable performance. In this paper we present a comprehensive and realistic investigation of the performance of a wide range of dynamic workflow scheduling policies in multicluster grids. We first introduce a taxonomy of grid workflow scheduling policies that is based on the amount of dynamic information used in the scheduling process, and map to this taxonomy seven such policies across the full spectrum of information use. Then, we analyze the performance of these scheduling policies through simulations and experiments in a real multicluster grid. We find that there is no single grid workflow scheduling policy with good performance across all the investigated scenarios. We also find from our real system experiments that with demanding workloads, the limitations of the head-nodes of the grid clusters may lead to performance loss not expected from the simulation results. We show that task throttling, that is, limiting the per-workflow number of tasks dispatched to the system, prevents the head-nodes from becoming overloaded while largely preserving performance, at least for communication-intensive workflows.
    Original languageEnglish
    Title of host publicationProceedings of the 19th International Symposium on High Performance Distributed Computing (HPDC'10, Chicago IL, USA, June 20-25, 2010)
    Place of PublicationNew York NY
    PublisherAssociation for Computing Machinery, Inc
    Pages49-60
    ISBN (Print)978-1-60558-942-8
    Publication statusPublished - 2010

    Fingerprint Dive into the research topics of 'Performance analysis of dynamic workflow scheduling in multicluster grids'. Together they form a unique fingerprint.

  • Cite this

    Sonmez, O. O., Yigitbasi, M. N., Abrishami, S., Iosup, A., & Epema, D. H. J. (2010). Performance analysis of dynamic workflow scheduling in multicluster grids. In Proceedings of the 19th International Symposium on High Performance Distributed Computing (HPDC'10, Chicago IL, USA, June 20-25, 2010) (pp. 49-60). Association for Computing Machinery, Inc.