Progressive File Layouts
The Lustre Progressive File Layout (PFL) feature intends to simplify the use of Lustre so that users can expect reasonable performance for a variety of normal file IO patterns without the need to explicitly understand their IO model or Lustre usage details in advance. In particular, users do not necessarily need to know the size or concurrency of output files in advance of their creation and explicitly specify an optimal layout for each file in order to achieve good performance for both highly concurrent shared-single-large-file IO or parallel IO to many smaller per-process files.
The PFL feature is implemented in several phases, providing incremental functionality with each phase, including the base functionality of Composite layouts which can be used for several other features that affect the file layout.
Phase 1: Prototype Implementation
- PFL Prototype Scope Statement describes the overall goals and intended outcomes of the prototype
- PFL Prototype Solution Architecture describes how the goals of the PFL project may be implemented, and how to measure the completion and outcomes
Phase 2: Static Layout Implementation
The Static PFL Implementation will provide a functional implementation that allows specifying the full layout using standard user tools and addresses any shortcuts and/or defects in the Prototype implementation. The following functionality was implemented:
- PFL2 Scope Statement describes the overall goals and intended outcomes of the production implementation
- PFL2 Solution Architecture describes how the goals of the PFL project may be implemented, and how to measure the completion and outcomes
- PFL2 High Level Design describes the implementation details for the PFL feature
- Implement improved layout handling APIs
- Address technical debt from prototype phase
- Implement RPCs for modifying composite layouts (need Layout APIs)
- Server composite layout support (need Layout APIs)
Phase 3a: PFL Usability Improvements
- Server LOD support for composite layouts
- LFSCK support for composite layouts
- Default layout inheritance
Phase 3b: Dynamic Layout Implementation
- Composite file templates
- Dynamic layout instantiation
- Improved MDS object allocator