Completed Project: VC3: Virtual Clusters for Community Computation (2016-2019)

A traditional HPC computing facility provides a large amount of computing power but has a fixed environment designed to satisfy local needs. This makes it very challenging for users to deploy complex applications that span multiple sites and require specific application software, scheduling middleware, or sharing policies. This project addressed many of these challenges by making it possible for researchers to easily aggregate and share resources, install custom software environments, and deploy clustering frameworks across multiple HPC facilities through the concept of “virtual clusters”. We designed and implemented a prototype virtual cluster facility that enabled unprivileged users to create dynamic aggregations of computing power across multiple sites, deployed with custom middleware and complex software dependencies. This service is hosted at the University of Chicago and available through the site virtualclusters.org.

Related Publications

VC3: A Virtual Cluster Service for Community Computation

Lincoln Bryant, Jeremy Van, Benedikt Riedel, Robert Gardner, Jose Caballero, John Hover, Ben Tovar, and Douglas Thain

In PEARC: Practice and Experience in Advanced Research Computing, 2018

doi: 10.1145/3219104.3219125

Bib PDF

@inproceedings{vc3-pearc-2018,
  author = {Bryant, Lincoln and Van, Jeremy and Riedel, Benedikt and Gardner, Robert and Caballero, Jose and Hover, John and Tovar, Ben and Thain, Douglas},
  title = {{VC3: A Virtual Cluster Service for Community Computation}},
  booktitle = {{PEARC: Practice and Experience in Advanced Research Computing}},
  pages = {1-8},
  year = {2018},
  note = {{doi: 10.1145/3219104.3219125}},
  cclpaperid = {949},
  keywords = {vc3},
}

Automatic Dependency Management for Scientific Applications on Clusters

Benjamin Tovar, Nicholas Hazekamp, Nathaniel Kremer-Herman, and Douglas Thain

In IEEE International Conference on Cloud Engineering (IC2E) , 2018

doi: 10.1109/IC2E.2018.00026

Bib PDF

@inproceedings{tovar-ic2e-2018,
  author = {Tovar, Benjamin and Hazekamp, Nicholas and Kremer-Herman, Nathaniel and Thain, Douglas},
  title = {{Automatic Dependency Management for Scientific Applications on Clusters}},
  booktitle = {{IEEE International Conference on Cloud Engineering (IC2E) }},
  year = {2018},
  note = {{doi: 10.1109/IC2E.2018.00026}},
  cclpaperid = {945},
  keywords = {vc3},
}