Progressive File Layouts

Introduction

The Lustre Progressive File Layout (PFL) feature intends to simplify the use of Lustre so that users can expect reasonable performance for a variety of normal file IO patterns without the need to explicitly understand their IO model or Lustre usage details in advance. In particular, users do not necessarily need to know the size or concurrency of output files in advance of their creation and explicitly specify an optimal layout for each file in order to achieve good performance for both highly concurrent shared-single-large-file IO or parallel IO to many smaller per-process files. The PFL Prototype Scope Statement describes the overall goals and intended outcomes of this project in more detail, and will not be repeated here. The PFL Prototype Solution Architecture document describes how the goals of the PFL project may be implemented, and how to measure the completion and outcomes of this project.

The PFL feature is implemented in several phases, providing incremental functionality with each phase, including the base functionality of Composite layouts which can be used for several other features that affect the file layout.