Categorizing research computing into several main groups and discussing which computational approaches are best suited to each group (HTC vs. HPC workflows)

Compact Muon Solenoid (CMS) is a particle physics experiment at CERN Large Hadron Collider (LHC). The talk presents an overview of how HTCondor is being used in the CMS grid computing infrastructure and highlights the scale we have managed to achieve availing various features provided by the software.

This talk will discuss how IceCube is using the Condor glideins in our simulation production system. We get to most of our distributed resources via Condor glide ins. For those sites which do not present a grid interface (a CE), we have recently developed a simple pilot factory which we can run from a local account at the site, as a cron. This allows us to seamlessly integrate remote sites into our condor pools in a very convenient way. We are running this system in several sites, including supercomputers from XSEDE where we are interested in running GPU jobs. We will describe the system and how it helped IceCube run jobs at multiple remote locations with very small operational overhead.

With the HTCondor-CE usage spreading out amongst Europe, this talk looks at what benefits a HTCondor-on-HTCondor deployment can bring, an in-depth of the CERN CE deployment, and how the HTCondor eco-system can change the grid for the better.

This talk will discuss how we are currently using HTCondor here at STScI in the operational system for processing and reprocessing Hubble Space Telescope (HST) data for our science users, some of the issues we are working on, and our plans for addressing them. We also will be using HTCondor in our data processing system for the James Webb Space Telescope (JWST), to launch in 2018, using similar patterns, and the differences in those plans compared with out HST architecture will also be presented.

The published scientific literature contains a large amount of data and information that has utility beyond the scope of the original investigation. For example, fossil occurrences are commonly described in the literature as part of local and regional field work, but literature-based syntheses of millions of fossil occurrences from around the world are required to generate an accurate history of life on Earth. Here we describe GeoDeepDive (GDD), a High Throughput cyberinfrastructure to support the reliable, scalable, and automated fetching of documents from content providers, the preprocessing of those documents by software tools that provide annotations for machine reading, and the indexing of those documents based on known vocabularies of scientific terms. The infrastructure currently contains more than 1.2 million documents (https://geodeepdive.org) from six different content providers and grows at a rate of 60K documents per week. Emphasis is currently placed on geoscience- and bioscience-related content, but the infrastructure is not domain specific. Software applications can be written by scientists to extract data and information from these documents using the GeoDeepDive application template and testing datasets. The GDD infrastructure supports the running of applications against the whole of the relevant document set and the continual updating of the result set as new relevant documents are acquired and processed. The high throughput computing capabilities of HTCondor are critical to the processing of documents and the deployment of new tools against the entire library as they are developed and as the collection of documents grows.

Creating a common submission infrastructure for the LHC participating University of California campuses.

The CERN tier-0 batch service caters for a diverse set of users, from the grid to “local”, and increasingly runs on a diverse infrastructure. We’ll report on how we’ve been using HTCondor to cater for our user groups, how recent HTCondor developments are helping us, and how we’re integrating cloud resources into out pool.

Batch monitoring at Fermilab and how other Condor users can leverage it to quickly set up detailed monitoring of their own pools.

We’ve all heard of the gangliad, soon to be the metricsd, and the python api. Can we throw away those ad-hoc scripts that turned into critical monitoring components and work towards a common solution that can be shared across deployments?