Towards Distributed Model Analytics with Apache Spark

Önder Babur; Loek Cleophas; Mark van den Brand

doi:10.5220/0006735407670772

Towards Distributed Model Analytics with Apache Spark

Önder Babur, Loek Cleophas, Mark van den Brand

Software Engineering and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

3 Citations (Scopus)

1 Downloads (Pure)

Abstract

The growing number of models and other related artefacts in model-driven engineering has recently led to the emergence of approaches and tools for analyzing and managing them on a large scale. The framework SAMOS applies techniques inspired by information retrieval and data mining to analyze large sets of models. As the data size and analysis complexity goes up, however, further scalability is needed. In this paper we extend SAMOS to operate on Apache Spark, a popular engine for distributed Big Data processing, by partitioning the data and parallelizing the comparison and analysis phase. We present preliminary studies using a cluster infrastructure and report the results for two datasets: one with 250 Ecore metamodels where we detail the performance gain with various settings, and a larger one of 7.3k metamodels with nearly one million model elements for further demonstrating scalability.

Original language	English
Title of host publication	MODELSWARD 2018 - Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development
Editors	Slimane Hammoudi, Luis Ferreira Pires, Bran Selic
Publisher	SciTePress Digital Library
Pages	767-772
Number of pages	6
ISBN (Electronic)	978-989-758-283-7
DOIs	https://doi.org/10.5220/0006735407670772
Publication status	Published - 2018
Event	6th International Conference on Model-Driven Engineering and Software Development, MODELSWARD 2018 - Funchal, Madeira, Portugal Duration: 22 Jan 2018 → 24 Jan 2018

Conference

Conference	6th International Conference on Model-Driven Engineering and Software Development, MODELSWARD 2018
Country/Territory	Portugal
City	Funchal, Madeira
Period	22/01/18 → 24/01/18

Funding

This work is supported by the 4TU.NIRICT Research Community Funding on Model Management and Analytics in the Netherlands. We also would like to thank SURF, the collaborative ICT organisation for Dutch education and research, for providing us with a computational infrastructure and support.

Keywords

Apache Spark
Big Data
Distributed Computing
Model Analytics
Model-Driven Engineering
Scalability

Access to Document

10.5220/0006735407670772

Cite this

@inproceedings{e7aad9864ee74a54b9e2598ce785e536,

title = "Towards Distributed Model Analytics with Apache Spark",

abstract = "The growing number of models and other related artefacts in model-driven engineering has recently led to the emergence of approaches and tools for analyzing and managing them on a large scale. The framework SAMOS applies techniques inspired by information retrieval and data mining to analyze large sets of models. As the data size and analysis complexity goes up, however, further scalability is needed. In this paper we extend SAMOS to operate on Apache Spark, a popular engine for distributed Big Data processing, by partitioning the data and parallelizing the comparison and analysis phase. We present preliminary studies using a cluster infrastructure and report the results for two datasets: one with 250 Ecore metamodels where we detail the performance gain with various settings, and a larger one of 7.3k metamodels with nearly one million model elements for further demonstrating scalability.",

keywords = "Apache Spark, Big Data, Distributed Computing, Model Analytics, Model-Driven Engineering, Scalability",

author = "{\"O}nder Babur and Loek Cleophas and {van den Brand}, Mark",

year = "2018",

doi = "10.5220/0006735407670772",

language = "English",

pages = "767--772",

editor = "Slimane Hammoudi and Pires, {Luis Ferreira} and Bran Selic",

booktitle = "MODELSWARD 2018 - Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development",

publisher = "SciTePress Digital Library",

note = "6th International Conference on Model-Driven Engineering and Software Development, MODELSWARD 2018 ; Conference date: 22-01-2018 Through 24-01-2018",

}

Babur, Ö , Cleophas, L & van den Brand, M 2018, Towards Distributed Model Analytics with Apache Spark. in S Hammoudi, LF Pires & B Selic (eds), MODELSWARD 2018 - Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development. SciTePress Digital Library, pp. 767-772, 6th International Conference on Model-Driven Engineering and Software Development, MODELSWARD 2018, Funchal, Madeira, Portugal, 22/01/18. https://doi.org/10.5220/0006735407670772

Towards Distributed Model Analytics with Apache Spark. / Babur, Önder ; Cleophas, Loek ; van den Brand, Mark.
MODELSWARD 2018 - Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development. ed. / Slimane Hammoudi; Luis Ferreira Pires; Bran Selic. SciTePress Digital Library, 2018. p. 767-772.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Towards Distributed Model Analytics with Apache Spark

AU - Babur, Önder

AU - Cleophas, Loek

AU - van den Brand, Mark

PY - 2018

Y1 - 2018

N2 - The growing number of models and other related artefacts in model-driven engineering has recently led to the emergence of approaches and tools for analyzing and managing them on a large scale. The framework SAMOS applies techniques inspired by information retrieval and data mining to analyze large sets of models. As the data size and analysis complexity goes up, however, further scalability is needed. In this paper we extend SAMOS to operate on Apache Spark, a popular engine for distributed Big Data processing, by partitioning the data and parallelizing the comparison and analysis phase. We present preliminary studies using a cluster infrastructure and report the results for two datasets: one with 250 Ecore metamodels where we detail the performance gain with various settings, and a larger one of 7.3k metamodels with nearly one million model elements for further demonstrating scalability.

AB - The growing number of models and other related artefacts in model-driven engineering has recently led to the emergence of approaches and tools for analyzing and managing them on a large scale. The framework SAMOS applies techniques inspired by information retrieval and data mining to analyze large sets of models. As the data size and analysis complexity goes up, however, further scalability is needed. In this paper we extend SAMOS to operate on Apache Spark, a popular engine for distributed Big Data processing, by partitioning the data and parallelizing the comparison and analysis phase. We present preliminary studies using a cluster infrastructure and report the results for two datasets: one with 250 Ecore metamodels where we detail the performance gain with various settings, and a larger one of 7.3k metamodels with nearly one million model elements for further demonstrating scalability.

KW - Apache Spark

KW - Big Data

KW - Distributed Computing

KW - Model Analytics

KW - Model-Driven Engineering

KW - Scalability

UR - http://www.scopus.com/inward/record.url?scp=85052021429&partnerID=8YFLogxK

U2 - 10.5220/0006735407670772

DO - 10.5220/0006735407670772

M3 - Conference contribution

AN - SCOPUS:85052021429

SP - 767

EP - 772

BT - MODELSWARD 2018 - Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development

A2 - Hammoudi, Slimane

A2 - Pires, Luis Ferreira

A2 - Selic, Bran

PB - SciTePress Digital Library

T2 - 6th International Conference on Model-Driven Engineering and Software Development, MODELSWARD 2018

Y2 - 22 January 2018 through 24 January 2018

ER -

Towards Distributed Model Analytics with Apache Spark

Abstract

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this