Chirp is a user-level file system for collaboration across distributed systems such as clusters, clouds, and grids. Chirp allows ordinary users to discover, share, and access storage, whether within a single machine room or over a wide area network.
Chirp requires no special privileges. Unlike most standard file systems or storage services, Chirp does not require root access, kernel changes, special modules, or anything like that. It can be run by ordinary users to export ordinary file systems on any machine or port that you like.
Chirp is transparent. When used with Parrot or FUSE, Chirp servers can be transparently attached to existing ordinary applications -- like tcsh, vi, and perl – without any sort of kernel changes or special privileges. Chirp is designed to give maximum compatibility with standard Unix semantics.
Chirp is easy to deploy. Chirp is designed to be deployed with a minimum of fuss. One simple command starts a Chirp server or a Chirp client. There is no complex configuration, installation, or setup to mess up. It just works. This makes Chirp ideal for on-the-fly storage management in batch computing and grid computing environments.
@article{ccpe-confuga,author={Donnelly, Patrick and Thain, Douglas},title={{Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows}},journal={{Concurrency and Computation: Practice and Experience}},volume={29},number={4},year={2016},note={{doi: 10.1002/cpe.3834}},cclpaperid={929},keywords={makeflow, chirp, confuga},}
Data Locality Techniques in an Active Cluster Filesystem for Scientific Workflows
@thesis{pdonnelly-thesis,author={Donnelly, Patrick},title={{Data Locality Techniques in an Active Cluster Filesystem for Scientific Workflows}},editor={Thesis, Ph.D.},booktitle={{University of Notre Dame}},year={2016},cclpaperid={928},keywords={makeflow, chirp, confuga},}
Confuga: Scalable Data Intensive Computing for POSIX Workflows
Patrick Donnelly, Nicholas Hazekamp, and Douglas Thain
In IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, 2015
@inproceedings{confuga-ccgrid2015,author={Donnelly, Patrick and Hazekamp, Nicholas and Thain, Douglas},title={{Confuga: Scalable Data Intensive Computing for POSIX Workflows}},booktitle={{IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing}},pages={392-401},year={2015},note={{doi: 10.1109/CCGrid.2015.95}},cclpaperid={908},keywords={makeflow, chirp, confuga},}
Design of an Active Storage Cluster File System for DAG Workflows
Patrick Donnelly and Douglas Thain
In International Workshop on Data-Intensive Scalable Computing Systems, 2013
@inproceedings{confuga-discs2013,author={Donnelly, Patrick and Thain, Douglas},title={{Design of an Active Storage Cluster File System for DAG Workflows}},booktitle={{International Workshop on Data-Intensive Scalable Computing Systems}},pages={37-42},year={2013},note={{doi: 10.1145/2534645.2534656}},cclpaperid={900},keywords={chirp, confuga},}
ROARS: A Robust Object Archival System for Data Intensive Scientific Computing
Hoang Bui, Peter Bui, Patrick Flynn, and Douglas Thain
@article{roars-pddb-2012,author={Bui, Hoang and Bui, Peter and Flynn, Patrick and Thain, Douglas},title={{ROARS: A Robust Object Archival System for Data Intensive Scientific Computing}},journal={{Distributed and Parallel Databases}},volume={30},number={5-6},publisher={Springer},year={2012},note={{doi: 10.1007/s10619-012-7103-5}},cclpaperid={890},keywords={chirp},}
Fine-Grained Access Control in the Chirp Distributed File System
Patrick Donnelly and Douglas Thain
In IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, 2012
@inproceedings{chirp-tickets-ccgrid12,author={Donnelly, Patrick and Thain, Douglas},title={{Fine-Grained Access Control in the Chirp Distributed File System}},booktitle={{IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing}},year={2012},note={{doi: 10.1109/CCGrid.2012.128}},cclpaperid={101},keywords={parrot, chirp, filesystems, career, gridfs},}
Data Intensive Computing with Clustered Chirp Servers
Douglas Thain, Michael Albrecht, Hoang Bui, Peter Bui, Rory Carmichael, Scott Emrich, and Patrick Flynn
In Data Intensive Distributed Computing: Challenges and Solutions for Large Scale Information Management, 2012
@incollection{chirp-didc-chapter,author={Thain, Douglas and Albrecht, Michael and Bui, Hoang and Bui, Peter and Carmichael, Rory and Emrich, Scott and Flynn, Patrick},title={{Data Intensive Computing with Clustered Chirp Servers}},editor={Kosar, Tevfik},booktitle={{Data Intensive Distributed Computing: Challenges and Solutions for Large Scale Information Management}},pages={140-154},publisher={IGI},year={2012},note={{isbn: 9781615209712}},doi={10.4018/978-1-61520-971-2.ch009},cclpaperid={99},keywords={chirp, filesystems, career, gridfs}}
Attaching Cloud Storage to a Campus Grid Using Parrot, Chirp, and Hadoop
Patrick Donnelly, Peter Bui, and Douglas Thain
In IEEE International Conference on Cloud Computing Technology and Science, 2010
@inproceedings{chirp+parrot+hdfs,author={Donnelly, Patrick and Bui, Peter and Thain, Douglas},title={{Attaching Cloud Storage to a Campus Grid Using Parrot, Chirp, and Hadoop}},booktitle={{IEEE International Conference on Cloud Computing Technology and Science}},pages={488-495},year={2010},note={{doi: 10.1109/CloudCom.2010.74}},cclpaperid={90},keywords={parrot, chirp, filesystems, career, gridfs},}
ROARS: A Scalable Repository for Data Intensive Scientific Computing
Hoang Bui, Peter Bui, Patrick Flynn, and Douglas Thain
In The Third International Workshop on Data Intensive Distributed Computing at ACM HPDC 2010, 2010
@inproceedings{roars-didc10,author={Bui, Hoang and Bui, Peter and Flynn, Patrick and Thain, Douglas},title={{ROARS: A Scalable Repository for Data Intensive Scientific Computing}},booktitle={{The Third International Workshop on Data Intensive Distributed Computing at ACM HPDC 2010}},year={2010},note={{doi: 10.1145/1851476.1851587}},cclpaperid={85},keywords={chirp, filesystems, career, hecura, gridfs},}
Biocompute: Toward a Collaborative Workspace for Data Intensive Bio-Science
Rory Carmichael, Patrick Braga-Henebry, Douglas Thain, and Scott Emrich
In Workshop on Emerging Computational Methods for Life Sciences at ACM HPDC 2010, 2010
@inproceedings{biocompute-ecmls,author={Carmichael, Rory and Braga-Henebry, Patrick and Thain, Douglas and Emrich, Scott},title={{Biocompute: Toward a Collaborative Workspace for Data Intensive Bio-Science}},booktitle={{Workshop on Emerging Computational Methods for Life Sciences at ACM HPDC 2010}},pages={489-498},year={2010},note={{doi: 10.1145/1851476.1851547}},cclpaperid={91},keywords={chirp},}
Coordination of Access to Large-scale Datasets in Distributed Environments
T. Kosar, A. Hutanu, J. McLaren, and D. Thain
In Scientific Data Management: Challenges, Existing Technology, and Deployment, 2009
@incollection{kosar-databook,author={Kosar, T. and Hutanu, A. and McLaren, J. and Thain, D.},title={{Coordination of Access to Large-scale Datasets in Distributed Environments}},editor={Shoshani, A. and Rotem, D.},booktitle={{Scientific Data Management: Challenges, Existing Technology, and Deployment}},publisher={Chapman and Hall / CRC Press},year={2009},note={{isbn: 978-1420069808}},doi={10.1201/9781420069815-12},cclpaperid={8},keywords={chirp}}
Experience with BXGrid: A Data Repository and Computing Grid for Biometrics Research
Hoang Bui, Michael Kelly, Christopher Lyon, Mark Pasquier, Deborah Thomas, Patrick Flynn, and Douglas Thain
@article{bxgrid-jcc,author={Bui, Hoang and Kelly, Michael and Lyon, Christopher and Pasquier, Mark and Thomas, Deborah and Flynn, Patrick and Thain, Douglas},title={{Experience with BXGrid: A Data Repository and Computing Grid for Biometrics Research}},journal={{Journal of Cluster Computing}},volume={12},number={4},pages={373},year={2009},note={{doi: 10.1007/s10586-009-0098-7}},cclpaperid={1},keywords={chirp, filesystems, career, gridfs},}
Chirp: A Practical Global Filesystem for Cluster and Grid Computing
Douglas Thain, Christopher Moretti, and Jeffrey Hemmes
@article{chirp-jgc,author={Thain, Douglas and Moretti, Christopher and Hemmes, Jeffrey},title={{Chirp: A Practical Global Filesystem for Cluster and Grid Computing}},journal={{Journal of Grid Computing}},volume={7},number={1},pages={51-72},year={2009},note={{doi: 10.1007/s10723-008-9100-5}},cclpaperid={14},keywords={parrot, chirp, filesystems, career, gridfs},}
Biomolecular Committor Probability Calculation Enabled by Processing in Network Storage
Paul Brenner, Justin Wozniak, Douglas Thain Aaron Striegel, Jeffrey Peng, and Jesus Izaguirre
@article{pins-jpc,author={Brenner, Paul and Wozniak, Justin and Striegel, Douglas Thain Aaron and Peng, Jeffrey and Izaguirre, Jesus},title={{Biomolecular Committor Probability Calculation Enabled by Processing in Network Storage}},journal={{Journal of Parallel Computing}},volume={34},number={11},pages={652-660},year={2008},note={{doi: 10.1016/j.parco.2008.08.001}},doi={10.1016/j.parco.2008.08.001},cclpaperid={13},keywords={chirp},}
Poster: DataLab: Transactional Data Parallel Computing on an Active Storage Cloud
Brandon Rich and Douglas Thain
In IEEE/ACM High Performance Distributed Computing, 2008
@inproceedings{datalab-hpdc08,author={Rich, Brandon and Thain, Douglas},title={{Poster: DataLab: Transactional Data Parallel Computing on an Active Storage Cloud}},booktitle={{IEEE/ACM High Performance Distributed Computing}},pages={233-234},year={2008},note={{isbn: 10.1145/1383422.1383461}},cclpaperid={27},keywords={chirp, hecura},}
Making the Best of a Bad Situation: Prioritized Storage Management in GEMS
Justin Wozniak, Paul Brenner, Douglas Thain, Aaron Striegel, and Jesus Izaguirre
@article{priority-fgcs,author={Wozniak, Justin and Brenner, Paul and Thain, Douglas and Striegel, Aaron and Izaguirre, Jesus},title={{Making the Best of a Bad Situation: Prioritized Storage Management in GEMS}},journal={{Future Generation Computing Systems}},volume={24},number={1},pages={10-16},year={2008},note={{doi: 10.1016/j.future.2007.04.003}},cclpaperid={15},keywords={chirp},}
Efficient Access to Many Small Files in a Filesystem for Grid Computing
@inproceedings{small-grid07,author={Thain, Douglas and Moretti, Christopher},title={{Efficient Access to Many Small Files in a Filesystem for Grid Computing}},booktitle={{IEEE Grid Computing}},pages={243-250},year={2007},note={{doi: 10.1109/GRID.2007.4354139}},cclpaperid={31},keywords={parrot, chirp, filesystems, career, gridfs},}
Flexible Object Based Filesystems for Scientific Computing
@thesis{moretti-ms-thesis,author={Moretti, Christopher},title={{Flexible Object Based Filesystems for Scientific Computing}},editor={Thesis, M.S.},booktitle={{University of Notre Dame}},year={2007},cclpaperid={65},keywords={chirp, filesystems, career, gridfs},}
Operating System Support for Space Allocation in Grid Storage Systems
@inproceedings{alloc-grid06,author={Thain, Douglas},title={{Operating System Support for Space Allocation in Grid Storage Systems}},booktitle={{IEEE Grid Computing}},pages={104-111},year={2006},note={{doi: 10.1109/ICGRID.2006.311004}},cclpaperid={41},keywords={chirp, allocfs, filesystems, career, gridfs},}
Cacheable Decentralized Groups for Grid Resource Access Control
@inproceedings{caches-hyperio06,author={Vazhkudai, Sudharshan and Thain, Douglas and Ma, Xiaosong and Freeh, Vincent},title={{Positioning Dynamic Storage Caches for Transient Data}},booktitle={{Workshop on High-Performance I/O Techniques and Deployment of Very Large Scale I/O Systems at IEEE Cluster Computing}},pages={1-9},year={2006},note={{doi: 10.1109/CLUSTR.2006.311900}},cclpaperid={43},keywords={chirp},}
The Consequences of Decentralized Security in a Cooperative Storage System
Douglas Thain, Christopher Moretti, Paul Madrid, Phil Snowberger, and Jeff Hemmes
In Workshop on Security in Storage at IEEE FAST, 2005
@inproceedings{cons-sisw05,author={Thain, Douglas and Moretti, Christopher and Madrid, Paul and Snowberger, Phil and Hemmes, Jeff},title={{The Consequences of Decentralized Security in a Cooperative Storage System}},booktitle={{Workshop on Security in Storage at IEEE FAST}},pages={82-94},year={2005},note={{doi: 10.1109/SISW.2005.11}},cclpaperid={51},keywords={parrot, chirp, filesystems, career, gridfs},}
Separating Abstractions from Resources in a Tactical Storage System
Douglas Thain, Sander Klous, Justin Wozniak, Paul Brenner, Aaron Striegel, and Jesus Izaguirre
@inproceedings{tactical-sc05,author={Thain, Douglas and Klous, Sander and Wozniak, Justin and Brenner, Paul and Striegel, Aaron and Izaguirre, Jesus},title={{Separating Abstractions from Resources in a Tactical Storage System}},booktitle={{IEEE/ACM Supercomputing}},pages={55-67},year={2005},note={{doi: 10.1109/SC.2005.64}},cclpaperid={52},keywords={parrot, chirp, allocfs, filesystems, career, hecura, gridfs},}
Generosity and Gluttony in GEMS: Grid Enabled Molecular Simulations
Justin Wozniak, Paul Brenner, Douglas Thain, Aaron Striegel, and Jesus Izaguirre
In IEEE Symposium on High Performance Distributed Computing, 2005
@inproceedings{gems-hpdc05,author={Wozniak, Justin and Brenner, Paul and Thain, Douglas and Striegel, Aaron and Izaguirre, Jesus},title={{Generosity and Gluttony in GEMS: Grid Enabled Molecular Simulations}},booktitle={{IEEE Symposium on High Performance Distributed Computing}},pages={191-200},year={2005},note={{doi: 10.1109/HPDC.2005.1520959}},cclpaperid={56},keywords={chirp},}
