Job admission and resource allocation in distributed streaming systems

J.L. Wolf, N. Bansal, K. Hildrum, S. Parekh, D. Rajan, R. Wagle, K.L. Wu

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    4 Citations (Scopus)

    Abstract

    This paper describes a new and novel scheme for job admission and resource allocation employed by the SODA scheduler in System S. Capable of processing enormous quantities of streaming data, System S is a large-scale, distributed stream processing system designed to handle complex applications. The problem of scheduling in distributed, stream-based systems is quite unlike that in more traditional systems. And the requirements for System S, in particular, are more stringent than one might expect even in a "standard" stream-based design. For example, in System S, the offered load is expected to vastly exceed system capacity. So a careful job admission scheme is essential. The jobs in System S are essentially directed graphs, with software "processing elements" (PEs) as vertices and data streams as edges connecting the PEs. The jobs themselves are often heavily interconnected. Thus resource allocation of individual PEs must be done carefully in order to balance the flow. We describe the design of the SODA scheduler, with particular emphasis on the component, known as macroQ, which performs the job admission and resource allocation tasks. We demonstrate by experiments the natural trade-offs between job admission and resource allocation.
    Original languageEnglish
    Title of host publicationJob Scheduling Strategies for Parallel Processing (14th International Workshop, JSSPP 2009, Rome, Italy, May 29, 2009. Revised Papers)
    EditorsE. Frachtenberg, U. Schwiegelshohn
    Place of PublicationBerlin
    PublisherSpringer
    Pages169-189
    ISBN (Print)978-3-642-04632-2
    DOIs
    Publication statusPublished - 2009

    Publication series

    NameLecture Notes in Computer Science
    Volume5798
    ISSN (Print)0302-9743

    Fingerprint Dive into the research topics of 'Job admission and resource allocation in distributed streaming systems'. Together they form a unique fingerprint.

    Cite this