The IceProd framework: distributed data processing for the IceCube neutrino observatory

Journal article

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, iden- tify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. IceProd is a distributed management system based on Python, XML-RPC and GridFTP. It is driven by a central database in order to coordinate and admin- ister production of simulations and processing of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, Condor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.

Authors: Aartsen, MG; Sarkar, S

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Elsevier
• Journal: Journal of Parallel and Distributed Computing
• Volume: 75
• Pages: 198–211-198–211
• Publication date: 2014-09-16
• Acceptance date: 2014-08-05
• DOI: 10.1016/j.jpdc.2014.08.001
• ISSN: 0743-7315
• Keywords: cs.DC; SBTMR