SODA : An optimizing scheduler for large-scale stream-based distributed computer systems

J.L. Wolf, N. Bansal, K. Hildrum, S. Parekh, D. Rajan, R. Wagle, K.L. Wu, L. Fleischer

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    77 Citations (Scopus)

    Abstract

    This paper describes the SODA scheduler for System S, a highly scalable distributed stream processing system. Unlike traditional batch applications, streaming applications are open-ended. The system cannot typically delay the processing of the data. The scheduler must be able to shift resource allocation dynamically in response to changes to resource availability, job arrivals and departures, incoming data rates and so on. The design assumptions of System S, in particular, pose additional scheduling challenges. SODA must deal with a highly complex optimization problem, which must be solved in real-time while maintaining scalability. SODA relies on a careful problem decomposition, and intelligent use of both heuristic and exact algorithms. We describe the design and functionality of SODA, outline the mathematical components, and describe experiments to show the performance of the scheduler.
    Original languageEnglish
    Title of host publicationMiddleware 2008 (ACM/IFIP/USENIX 9th International Middleware Conference, Leuven, Belgium, December 1-5, 2008. Proceedings)
    EditorsV. Issamy, R.E. Schantz
    Place of PublicationBerlin
    PublisherSpringer
    Pages306-325
    ISBN (Print)978-3-540-89855-9
    DOIs
    Publication statusPublished - 2008

    Publication series

    NameLecture Notes in Computer Science
    Volume5346
    ISSN (Print)0302-9743

    Fingerprint Dive into the research topics of 'SODA : An optimizing scheduler for large-scale stream-based distributed computer systems'. Together they form a unique fingerprint.

  • Cite this

    Wolf, J. L., Bansal, N., Hildrum, K., Parekh, S., Rajan, D., Wagle, R., Wu, K. L., & Fleischer, L. (2008). SODA : An optimizing scheduler for large-scale stream-based distributed computer systems. In V. Issamy, & R. E. Schantz (Eds.), Middleware 2008 (ACM/IFIP/USENIX 9th International Middleware Conference, Leuven, Belgium, December 1-5, 2008. Proceedings) (pp. 306-325). (Lecture Notes in Computer Science; Vol. 5346). Springer. https://doi.org/10.1007/978-3-540-89856-6_16