Flocks of a feather dock together: Using Docker and HTCondor to link high-throughput computing across the USGS Completed

USGS scientists often face computationally intensive tasks that require high-throughput computing capabilities. Several USGS facilities use HTCondor to run their computational pools but are not necessarily connected to the larger USGS pool. This project demonstrated how to connect HTCondor pools by flocking, or coordinating, within the USGS. In addition to flocking the Upper Midwest Environmental Science Center and the Wisconsin Water Science Center, we have flocked with the USGS Advanced Research Computing Yeti supercomputing cluster and other water science centers. We also developed tutorials on how to sandbox code using Docker within the USGS environment for use with high-throughput computing. A main accomplishment of the project was working through technical hurdles in the existing computing environment. Our results from this project helped the USGS Advanced Computing Cooperative flock HTCondor pools together and move towards an operational implementation.



Principal Investigator : Richard A Erickson

Co-Investigator : Sunnie (Grace) McCalla, Michael N Fienen, Jon J Amberg

Cooperator/Partner : Janice Gordon, Paul E Exter, Melvin Bower, Jonathan M Knudson



Note: This description is from the Community for Data Integration Fiscal Year 2017 Funded Project Report.