Progressive File Layouts: Difference between revisions

From Lustre Wiki
Jump to navigation Jump to search
mNo edit summary
(update with later implementation and design phases)
Line 1: Line 1:
Introduction
The Lustre Progressive File Layout (PFL) feature intends to simplify the use of Lustre so that users can expect reasonable performance for a variety of normal file IO patterns without the need to explicitly understand their IO model or Lustre usage details in advance. In particular, users do not necessarily need to know the size or concurrency of output files in advance of their creation and explicitly specify an optimal layout for each file in order to achieve good performance for both highly concurrent shared-single-large-file IO or parallel IO to many smaller per-process files.


The Lustre Progressive File Layout (PFL) feature intends to simplify the use of Lustre so that users can expect reasonable performance for a variety of normal file IO patterns without the need to explicitly understand their IO model or Lustre usage details in advance. In particular, users do not necessarily need to know the size or concurrency of output files in advance of their creation and explicitly specify an optimal layout for each file in order to achieve good performance for both highly concurrent shared-single-large-file IO or parallel IO to many smaller per-process files.  The [[PFL Prototype Scope Statement]] describes the overall goals and intended outcomes of this project in more detail, and will not be repeated here. The [[PFL Prototype Solution Architecture]] document describes how the goals of the PFL project may be implemented, and how to measure the completion and outcomes of this project.
The PFL feature is implemented in several phases, providing incremental functionality with each phase, including the base functionality of [[Layout_Enhancement_High_Level_Design#2.1._Composite_Layouts|Composite layouts]] which can be used for several other features that affect the file layout.


The PFL feature is implemented in several phases, providing incremental functionality with each phase, including the base functionality of [[Layout_Enhancement_High_Level_Design#2.1._Composite_Layouts|Composite layouts]] which can be used for several other features that affect the file layout.
== Phase 1: Prototype Implementation ==
* [[PFL Prototype Scope Statement]] describes the overall goals and intended outcomes of the prototype
* [[PFL Prototype Solution Architecture]] describes how the goals of the PFL project may be implemented, and how to measure the completion and outcomes
== Phase 2: Static Layout Implementation ==
The Static PFL Implementation will provide a functional implementation that allows specifying the full layout using standard user tools and addresses any shortcuts and/or defects in the Prototype implementation. The following functionality was implemented:
* [[PFL2 Scope Statement]] describes the overall goals and intended outcomes of the production implementation
* [[PFL2 Solution Architecture]] describes how the goals of the PFL project may be implemented, and how to measure the completion and outcomes
* [[PFL2 High Level Design]] describes the implementation details for the PFL feature
* Implement improved layout handling APIs
* Address technical debt from prototype phase
* Implement RPCs for modifying composite layouts (need Layout APIs)
* Server composite layout support (need Layout APIs)
== Phase 3a: PFL Usability Improvements ==
* Server LOD support for composite layouts
* LFSCK support for composite layouts
* Default layout inheritance
== Phase 3b: Dynamic Layout Implementation ==
* Composite file templates
* Dynamic layout instantiation
* Improved MDS object allocator


[[Category:Architecture]]
[[Category:Architecture]]
[[Category:Design]]
[[Category:Design]]
[[Category:PFL]]
[[Category:PFL]]

Revision as of 15:50, 11 January 2017

The Lustre Progressive File Layout (PFL) feature intends to simplify the use of Lustre so that users can expect reasonable performance for a variety of normal file IO patterns without the need to explicitly understand their IO model or Lustre usage details in advance. In particular, users do not necessarily need to know the size or concurrency of output files in advance of their creation and explicitly specify an optimal layout for each file in order to achieve good performance for both highly concurrent shared-single-large-file IO or parallel IO to many smaller per-process files.

The PFL feature is implemented in several phases, providing incremental functionality with each phase, including the base functionality of Composite layouts which can be used for several other features that affect the file layout.

Phase 1: Prototype Implementation

Phase 2: Static Layout Implementation

The Static PFL Implementation will provide a functional implementation that allows specifying the full layout using standard user tools and addresses any shortcuts and/or defects in the Prototype implementation. The following functionality was implemented:

  • PFL2 Scope Statement describes the overall goals and intended outcomes of the production implementation
  • PFL2 Solution Architecture describes how the goals of the PFL project may be implemented, and how to measure the completion and outcomes
  • PFL2 High Level Design describes the implementation details for the PFL feature
  • Implement improved layout handling APIs
  • Address technical debt from prototype phase
  • Implement RPCs for modifying composite layouts (need Layout APIs)
  • Server composite layout support (need Layout APIs)

Phase 3a: PFL Usability Improvements

  • Server LOD support for composite layouts
  • LFSCK support for composite layouts
  • Default layout inheritance

Phase 3b: Dynamic Layout Implementation

  • Composite file templates
  • Dynamic layout instantiation
  • Improved MDS object allocator
Retrieved from "http://wiki.lustre.org/index.php?title=Progressive_File_Layouts&oldid=2143"