Optimization of large scale data analysis in LHCb

D. Remenska, R. Aaij, G. Raven, M. Merk, J.A. Templon, R.J. Bril

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review


Observation has lead to a conclusion that the physics analysis jobs run by LHCb physicists on a local computing farm (i.e. non-grid) require more efficient access to the data which resides on the Grid. Our experiments have shown that the I/O bound nature of the analysis jobs in combination with the latency due to the remote access protocols (e.g. rfio, dcap) cause a low CPU efficiency of these jobs. In addition to causing a low CPU efficiency, the remote access protocols give rise to high overhead (in terms of amount of data transferred). This paper gives an overview of the concept of pre-fetching and caching of input files in the proximity of the processing resources, which is exploited to cope with the I/O bound analysis jobs. The files are copied from Grid storage elements (using GridFTP), while concurrently performing computations, inspired from a similar idea used in the ATLAS experiment. The results illustrate that this file staging approach is relatively insensitive to the original location of the data, and a significant improvement can be achieved in terms of the CPU efficiency of an analysis job. Dealing with scalability of such a solution on the Grid environment is discussed briefly.
Original languageEnglish
Title of host publication18th International Conference on Computing in High Energy and Nuclear Physics (CHEP, Taipei, Taiwan, October 18-22, 2010)
Publication statusPublished - 2011

Publication series

NameJournal of Physics: Conference Series
ISSN (Print)1742-6588


Dive into the research topics of 'Optimization of large scale data analysis in LHCb'. Together they form a unique fingerprint.

Cite this