The Parrot Virtual File System is a tool for attaching existing programs to remote I/O systems through the filesystem interface. For example, here’s how to use Parrot with vi in order to edit a file on a remote file server:
% parrot_run vi /chirp/server.nd.edu/mydata
Parrot “speaks” a variety of remote I/O services include HTTP, FTP, GridFTP, iRODS, CVMFS, and Chirp on behalf of ordinary programs. It works by trapping a program’s system calls through the ptrace debugging interface, and replacing them with remote I/O operations as desired. Parrot can be installed and operated by any user without special privileges or kernel changes and can be applied to almost any program without re-writing, re-linking, or re-installing.
Parrot is particularly useful for running batch jobs in large scale distributed systems. It is most widely used to provide access to high energy physics (HEP) software stacks via the global CVMFS filesystem developed at CERN. Because Parrot views every operation on the filesystem, it can make arbitrary changes to an application’s namespace. This allows it to function as a sandbox environment, a dependency tracing tool, virtual machine, and more.
(Note that Parrot is very tightly coupled with the OS kernel. and so it is only available on Linux based operating systems.)
Related Publications
Conducting Reproducible Research with Umbrella: Tracking, Creating, and Preserving Execution Environments
Haiyan Meng, Douglas Thain, Alexander Vyushkov, Matthias Wolf, and Anna Woodard
@inproceedings{umbrella-escience-2016,author={Meng, Haiyan and Thain, Douglas and Vyushkov, Alexander and Wolf, Matthias and Woodard, Anna},title={{Conducting Reproducible Research with Umbrella: Tracking, Creating, and Preserving Execution Environments}},booktitle={{IEEE Conference on e-Science}},year={2016},note={{doi: 10.1109/eScience.2016.7870889}},cclpaperid={931},keywords={parrot, umbrella, daspos},}
A Case Study in Preserving a High Energy Physics Application with Parrot
Haiyan Meng, Matthias Wolf, Peter Ivie, Anna Woodard, Michael Hildreth, and Douglas Thain
In Journal of Physics: Conference Series (CHEP 2015), 2015
@inproceedings{tauroast-chep-2015,author={Meng, Haiyan and Wolf, Matthias and Ivie, Peter and Woodard, Anna and Hildreth, Michael and Thain, Douglas},title={{A Case Study in Preserving a High Energy Physics Application with Parrot}},booktitle={{Journal of Physics: Conference Series (CHEP 2015)}},year={2015},note={{doi: 10.1088/1742-6596/664/3/032022}},cclpaperid={925},keywords={parrot, daspos},}
The Evolution of Global Scale Filesystems for Scientific Software Distribution
Jakob Blomer, Predrag Buncic, Rene Meusel, Gerardo Ganis, Igor Sfiligoi, and Douglas Thain
IEEE/AIP Computing in Science and Engineering, 2015
@article{globalfs-cise-2015,author={Blomer, Jakob and Buncic, Predrag and Meusel, Rene and Ganis, Gerardo and Sfiligoi, Igor and Thain, Douglas},title={{The Evolution of Global Scale Filesystems for Scientific Software Distribution}},journal={{IEEE/AIP Computing in Science and Engineering}},volume={17},number={6},pages={61-71},year={2015},note={{doi: 10.1109/MCSE.2015.111}},cclpaperid={926},keywords={parrot, filesystems, career, gridfs},}
Techniques for Preserving Scientific Software Executions: Preserve the Mess or Encourage Cleanliness?
Douglas Thain, Peter Ivie, and Haiyan Meng
In 12th International Conference on Digital Preservation (iPres), 2015
@inproceedings{techniques-ipres-2015,author={Thain, Douglas and Ivie, Peter and Meng, Haiyan},title={{Techniques for Preserving Scientific Software Executions: Preserve the Mess or Encourage Cleanliness?}},booktitle={{12th International Conference on Digital Preservation (iPres)}},year={2015},note={{doi: 10.7274/R0CZ353M}},cclpaperid={921},keywords={parrot, prune, umbrella, daspos},}
An Invariant Framework for Conducting Reproducible Computational Science
Haiyan Meng, Rupa Kommineni, Quan Pham, Robert Gardner, Tanu Malik, and Douglas Thain
@article{invariant-jocs-2015,author={Meng, Haiyan and Kommineni, Rupa and Pham, Quan and Gardner, Robert and Malik, Tanu and Thain, Douglas},title={{An Invariant Framework for Conducting Reproducible Computational Science}},journal={{Journal of Computational Science}},volume={9},pages={137-142},year={2015},note={{doi: 10.1016/j.jocs.2015.04.012}},cclpaperid={914},keywords={parrot},}
Umbrella: A Portable Environment Creator for Reproducible Computing on Clusters, Clouds, and Grids
Haiyan Meng and Douglas Thain
In Workshop on Virtualization Technologies in Distributed Computing (VTDC) at HPDC, 2015
@inproceedings{umbrella-vtdc15,author={Meng, Haiyan and Thain, Douglas},title={{Umbrella: A Portable Environment Creator for Reproducible Computing on Clusters, Clouds, and Grids}},booktitle={{Workshop on Virtualization Technologies in Distributed Computing (VTDC) at HPDC}},year={2015},note={{doi: 10.1145/2755979.2755982}},cclpaperid={912},keywords={parrot, umbrella},}
Opportunistic High Energy Physics Computing in User Space with Parrot
Dillon Skeehan, Paul Brenner, Ben Tovar, Douglas Thain, Nil Valls, Anna Woodard, Matthew Wolf, Tessa Pearson, Sean Lynch, and Kevin Lannon
In IEEE Symposium on Cluster, Cloud, and Grid Computing (CCGrid), 2014
@inproceedings{oppo-ccgrid14,author={Skeehan, Dillon and Brenner, Paul and Tovar, Ben and Thain, Douglas and Valls, Nil and Woodard, Anna and Wolf, Matthew and Pearson, Tessa and Lynch, Sean and Lannon, Kevin},title={{Opportunistic High Energy Physics Computing in User Space with Parrot}},booktitle={{IEEE Symposium on Cluster, Cloud, and Grid Computing (CCGrid)}},pages={170-175},year={2014},note={{doi: 10.1109/CCGrid.2014.34}},cclpaperid={902},keywords={parrot},}
Fine-Grained Access Control in the Chirp Distributed File System
Patrick Donnelly and Douglas Thain
In IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, 2012
@inproceedings{chirp-tickets-ccgrid12,author={Donnelly, Patrick and Thain, Douglas},title={{Fine-Grained Access Control in the Chirp Distributed File System}},booktitle={{IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing}},year={2012},note={{doi: 10.1109/CCGrid.2012.128}},cclpaperid={101},keywords={parrot, chirp, filesystems, career, gridfs},}
Attaching Cloud Storage to a Campus Grid Using Parrot, Chirp, and Hadoop
Patrick Donnelly, Peter Bui, and Douglas Thain
In IEEE International Conference on Cloud Computing Technology and Science, 2010
@inproceedings{chirp+parrot+hdfs,author={Donnelly, Patrick and Bui, Peter and Thain, Douglas},title={{Attaching Cloud Storage to a Campus Grid Using Parrot, Chirp, and Hadoop}},booktitle={{IEEE International Conference on Cloud Computing Technology and Science}},pages={488-495},year={2010},note={{doi: 10.1109/CloudCom.2010.74}},cclpaperid={90},keywords={parrot, chirp, filesystems, career, gridfs},}
CDF Software Distribution on the Grid using Parrot
Gabrielle Compostella, Simone Pagan Griso, Donatella Lucchesi, Igor Sfiligoi, and Douglas Thain
@inproceedings{parrot-chep09,author={Compostella, Gabrielle and Griso, Simone Pagan and Lucchesi, Donatella and Sfiligoi, Igor and Thain, Douglas},title={{CDF Software Distribution on the Grid using Parrot}},booktitle={{Computing in High Energy Physics}},year={2009},note={{doi: 10.1088/1742-6596/219/6/062009}},cclpaperid={22},keywords={parrot, filesystems, career, gridfs},}
Chirp: A Practical Global Filesystem for Cluster and Grid Computing
Douglas Thain, Christopher Moretti, and Jeffrey Hemmes
@article{chirp-jgc,author={Thain, Douglas and Moretti, Christopher and Hemmes, Jeffrey},title={{Chirp: A Practical Global Filesystem for Cluster and Grid Computing}},journal={{Journal of Grid Computing}},volume={7},number={1},pages={51-72},year={2009},note={{doi: 10.1007/s10723-008-9100-5}},cclpaperid={14},keywords={parrot, chirp, filesystems, career, gridfs},}
Efficient Access to Many Small Files in a Filesystem for Grid Computing
@inproceedings{small-grid07,author={Thain, Douglas and Moretti, Christopher},title={{Efficient Access to Many Small Files in a Filesystem for Grid Computing}},booktitle={{IEEE Grid Computing}},pages={243-250},year={2007},note={{doi: 10.1109/GRID.2007.4354139}},cclpaperid={31},keywords={parrot, chirp, filesystems, career, gridfs},}
Grid Deployment of Legacy Bioinformatics Applications with Transparent Data Access
Christophe Blanchet, Remi Mollon, Douglas Thain, and Gilbert Deleage
@inproceedings{bio-grid06,author={Blanchet, Christophe and Mollon, Remi and Thain, Douglas and Deleage, Gilbert},title={{Grid Deployment of Legacy Bioinformatics Applications with Transparent Data Access}},booktitle={{IEEE Grid Computing}},pages={120-127},year={2006},note={{doi: 10.1109/ICGRID.2006.311006}},cclpaperid={37},keywords={parrot, filesystems, career, gridfs},}
Transparent Access to Grid Resources for User Software
Sander Klous, Jamie Frey, Se-Chang Son, Douglas Thain, Alain Roy, Miron Livny, and Jo Brand
Concurrency and Computation: Practice and Experience, 2006
@article{transparent-ccpe,author={Klous, Sander and Frey, Jamie and Son, Se-Chang and Thain, Douglas and Roy, Alain and Livny, Miron and van den Brand, Jo},title={{Transparent Access to Grid Resources for User Software}},journal={{Concurrency and Computation: Practice and Experience}},volume={18},number={7},pages={787-801},year={2006},note={{doi: 10.1002/cpe.961 }},cclpaperid={17},keywords={parrot, filesystems, career, gridfs},}
Using Condor Glide-Ins and Parrot to Move from Dedicated Resources to the Grid
Stefano Belforte, Matthew Normal, Subir Sarkar, Ifor Sfiligoi, Douglas Thain, and Frank Wuerthwein
@article{parrot-lni06,author={Belforte, Stefano and Normal, Matthew and Sarkar, Subir and Sfiligoi, Ifor and Thain, Douglas and Wuerthwein, Frank},title={{Using Condor Glide-Ins and Parrot to Move from Dedicated Resources to the Grid}},journal={{Lecture Notes in Informatics}},volume={81},pages={285-292},year={2006},cclpaperid={49},keywords={parrot, filesystems, career, gridfs},}
Transparently Distributing CDF Software with Parrot
Douglas Thain, Christopher Moretti, and Igor Sfiligoi
@inproceedings{parrot-chep06,author={Thain, Douglas and Moretti, Christopher and Sfiligoi, Igor},title={{Transparently Distributing CDF Software with Parrot}},booktitle={{Computing in High Energy Physics}},pages={1-4},year={2006},cclpaperid={50},keywords={parrot, filesystems, career, gridfs},}
The Consequences of Decentralized Security in a Cooperative Storage System
Douglas Thain, Christopher Moretti, Paul Madrid, Phil Snowberger, and Jeff Hemmes
In Workshop on Security in Storage at IEEE FAST, 2005
@inproceedings{cons-sisw05,author={Thain, Douglas and Moretti, Christopher and Madrid, Paul and Snowberger, Phil and Hemmes, Jeff},title={{The Consequences of Decentralized Security in a Cooperative Storage System}},booktitle={{Workshop on Security in Storage at IEEE FAST}},pages={82-94},year={2005},note={{doi: 10.1109/SISW.2005.11}},cclpaperid={51},keywords={parrot, chirp, filesystems, career, gridfs},}
Separating Abstractions from Resources in a Tactical Storage System
Douglas Thain, Sander Klous, Justin Wozniak, Paul Brenner, Aaron Striegel, and Jesus Izaguirre
@inproceedings{tactical-sc05,author={Thain, Douglas and Klous, Sander and Wozniak, Justin and Brenner, Paul and Striegel, Aaron and Izaguirre, Jesus},title={{Separating Abstractions from Resources in a Tactical Storage System}},booktitle={{IEEE/ACM Supercomputing}},pages={55-67},year={2005},note={{doi: 10.1109/SC.2005.64}},cclpaperid={52},keywords={parrot, chirp, allocfs, filesystems, career, hecura, gridfs},}
Identity Boxing: A New Technique for Consistent Global Identity
@inproceedings{ibox-sc05,author={Thain, Douglas},title={{Identity Boxing: A New Technique for Consistent Global Identity}},booktitle={{IEEE/ACM Supercomputing}},pages={51-61},year={2005},note={{doi: 10.1109/SC.2005.34}},cclpaperid={53},keywords={parrot},}
Parrot: An Application Environment for Data-Intensive Computing
@article{parrot-scpe05,author={Thain, Douglas and Livny, Miron},title={{Parrot: An Application Environment for Data-Intensive Computing}},journal={{Scalable Computing: Practice and Experience}},volume={6},number={3},pages={9-18},year={2005},cclpaperid={18},keywords={parrot, filesystems, career, gridfs},}
Poster: Identity Boxing: Secure User-Level Containment for the Grid
@article{ibox-hpdc05,author={Thain, Douglas},title={{Poster: Identity Boxing: Secure User-Level Containment for the Grid}},booktitle={{IEEE Symposium on High Performance Distributed Computing}},pages={299-300},year={2005},note={{isbn: 10.1109/HPDC.2005.1520984}},cclpaperid={55},keywords={parrot},}
Parrot: Transparent User-Level Middleware for Data Intensive Computing
Douglas Thain and Miron Livny
In Workshop on Adaptive Grid Middleware at PACT, 2003
@inproceedings{parrot-agm2003,author={Thain, Douglas and Livny, Miron},title={{Parrot: Transparent User-Level Middleware for Data Intensive Computing}},booktitle={{Workshop on Adaptive Grid Middleware at PACT}},year={2003},cclpaperid={67},keywords={parrot, filesystems, career, gridfs},}
