Optimization of large scale data analysis in LHCb

D. Remenska, R. Aaij, G. Raven, M. Merk, J.A. Templon, R.J. Bril

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureConferentiebijdrageAcademicpeer review


Observation has lead to a conclusion that the physics analysis jobs run by LHCb physicists on a local computing farm (i.e. non-grid) require more efficient access to the data which resides on the Grid. Our experiments have shown that the I/O bound nature of the analysis jobs in combination with the latency due to the remote access protocols (e.g. rfio, dcap) cause a low CPU efficiency of these jobs. In addition to causing a low CPU efficiency, the remote access protocols give rise to high overhead (in terms of amount of data transferred). This paper gives an overview of the concept of pre-fetching and caching of input files in the proximity of the processing resources, which is exploited to cope with the I/O bound analysis jobs. The files are copied from Grid storage elements (using GridFTP), while concurrently performing computations, inspired from a similar idea used in the ATLAS experiment. The results illustrate that this file staging approach is relatively insensitive to the original location of the data, and a significant improvement can be achieved in terms of the CPU efficiency of an analysis job. Dealing with scalability of such a solution on the Grid environment is discussed briefly.
Originele taal-2Engels
Titel18th International Conference on Computing in High Energy and Nuclear Physics (CHEP, Taipei, Taiwan, October 18-22, 2010)
StatusGepubliceerd - 2011

Publicatie series

NaamJournal of Physics: Conference Series
ISSN van geprinte versie1742-6588


Duik in de onderzoeksthema's van 'Optimization of large scale data analysis in LHCb'. Samen vormen ze een unieke vingerafdruk.

Citeer dit