Flexibility and Performance of Parallel File Systems


David Kotz and Nils Nieuwejaar. Flexibility and Performance of Parallel File Systems. ACM Operating Systems Review, volume 30, number 2, pages 63–73. ACM, April 1996. doi:10.1145/232302.232314. ©Copyright the authors. Later revised as kotz:flexibility2.


Many scientific applications for high-performance multiprocessors have tremendous I/O requirements. As a result, the I/O system is often the limiting factor of application performance. Several new parallel file systems have been developed in recent years, each promising better performance for some class of parallel applications. As we gain experience with parallel computing, and parallel file systems in particular, it becomes increasingly clear that a single solution does not suit all applications. For example, it appears to be impossible to find a single appropriate interface, caching policy, file structure, or disk management strategy. Furthermore, the proliferation of file-system interfaces and abstractions make application portability a significant problem.

We propose that the traditional functionality of parallel file systems be separated into two components: a fixed core that is standard on all platforms, encapsulating only primitive abstractions and interfaces, and a set of high-level libraries to provide a variety of abstractions and application-programmer interfaces (APIs). We think of this approach as the “RISC” of parallel file-system design.

We present our current and next-generation file systems as examples of this structure. Their features, such as a three-dimensional file structure, strided read and write interfaces, and I/O-node programs, are specifically designed with the flexibility and performance necessary to support a wide range of applications.

Citable with [BibTeX]

Projects: [galley]

Keywords: [pario]

Available from the publisher: [DOI]

Available from the author: [bib] [pdf] [ps.gz]
This pdf was produced by the authors, is identical in content to the publisher copy (though with cleaner format), and is posted with permission.

[Kotz research]