Lustre Internals Documentation Update

Goals

 * Incremental updating of Lustre internals documentation.
 * The available time for those with Lustre internals knowledge is limited, maximizing the efficiency of their efforts is a priority.

General

 * Updates are moving to Category:Internals and pages with the Internals category in order to better facilitate linking and incremental updating.
 * Updating Lustre internals documentation is discussed regularly on the Lustre Working Group calls, please join us there.

High

 * Migrate ULFI LNET chapter from shared document to wiki
 * Migrate ULFI PtlRPC chapter from shared document to wiki
 * Migrate ULFI LDLM chapter from shared document to wiki
 * Update Subsystem wiki page for current subsystems
 * Add Layered Object wiki page, reference clio.txt and LID/AGI resources
 * Update wiki PtlRPC page to reference PtlRPC documentation work

Medium

 * Generate Doxygen comments from current source tree and put on web
 * Review Doxygen comments in code base and find methods to reference them from wiki
 * Keep Lustre_Internals wiki page?

Low

 * Automate generation/uploading of Doxygen comments to the web
 * Investigate lustre.org mirror of Lustre git source tree and resources available to facilitate web presentation of the source (e.g. http://elixir.free-electrons.com/linux/latest/source )

Feedback
Feedback from discussions with community members on updating Lustre internals documentation in general:


 * Top-down versus bottom-up approach
 * Differing viewpoints on this. Possible to find way to address both?


 * Separate documents (e.g. Understanding Lustre Filesystem Internals document) versus documentation in the code (e.g. Doxygen)
 * Possible to incrementally expand and update both, balance what's in them, and have them mutually reference?

Lustre Protocol Documentation

 * http://wiki.opensfs.org/Contract_SFS-DEV-005
 * https://jira.hpdd.intel.com/browse/LUDOC-280
 * https://build.hpdd.intel.com/job/lustre-protocol-reviews/lastSuccessfulBuild/artifact/protocol.html

Presentations
Sequoia and the ZFS OSD (LUG2013)
 * http://cdn.opensfs.org/wp-content/uploads/2013/04/Morrone_Sequoia_LUG2013.pdf

Intel® Lustre* File Level Replication (LUG2014)
 * http://wiki.lustre.org/images/a/ad/LUG2014-File_Level_Replication-Xiong.pdf

Distributed Name Space Phase I (LUG2013)
 * http://cdn.opensfs.org/wp-content/uploads/2013/04/LUG-2013_DNE.pdf

Lustre & Kerberos: in theory and in practice (LUG2015)
 * http://wiki.lustre.org/images/e/ec/Lustre-and-Kerberos_Buisson.pdf

Doxygen Code Documentation

 * Old: http://wiki.old.lustre.org/lid/doxygen.api/modules.html
 * TODO: Generate new version from codebase and make available on web.

Overview of the Lustre Client I/O (CLIO) subsystem
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/clio.txt

LFSCK
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lfsck.txt


 * LFSCK master slave design
 * Object traversal design reference

Lock Ordering
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lock-ordering

/* dot input file for lock-ordering diagram */

Overview of the Lustre Object Storage Device API
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/osd-api.txt

Overview of Dynamic LNet Configuration
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/dlc.txt

Lustre versioning
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/versioning.txt

Old Site Lustre Internals Documentation Area
http://wiki.old.lustre.org/lid/

Glossary
http://wiki.old.lustre.org/lid/glossary/glossary.html

Brief descriptions of Lustre concepts, objects and major components indexed in various ways.

Lustre Internals: A Gentle Introduction
http://wiki.old.lustre.org/lid/agi/agi.html

Subsystem Map
TODO: Generate new version of
 * http://wiki.old.lustre.org/lid/subsystem-map-old/subsystem-map.html

Original ULFI Table Of Contents

 * Component View on Architecture
 * Lustre Client
 * OSS
 * MDS
 * Lustre Lite
 * Connection
 * Dentry Object
 * Lustre Superblock
 * Lustre inode
 * Path Lookup
 * Path
 * Asynchronous I/O
 * Group I/O (or Synchronous I/O)
 * Direct I/O
 * Interface with VFS
 * Read-Ahead
 * LOV and OSC
 * Device Operations
 * Page Management
 * From OSC Client To OST
 * Grant
 * LDLM: Lock Manager
 * Namespace
 * Resource
 * Lock Type and Mode
 * Callbacks
 * Intent
 * Lock Manager
 * Requesting a Lock
 * Canceling a Lock
 * Policy Function
 * Cases
 * MDS: One Client Read
 * MDS: Two Clients
 * OST: Two Clients Read and Write
 * OST and Obdfilter
 * as OST
 * Initial Setup
 * Dispatching
 * Directory Layout
 * Group Number
 * Object Id
 * obdfilter
 * File Deletion
 * File Creation
 * MDC and Lustre Metadata
 * Overview
 * Striping EA
 * Striping API
 * Infrastructure Support
 * Lustre Client Registration
 * Superblock and Inode Registration
 * Device
 * Import and Export
 * Portal RPC
 * Client Side Interface
 * Server Side Interface
 * Bulk Transfer
 * NRS Optimization
 * Error Recovery: A Client Perspective
 * LNET: Lustre Networking
 * Core Concepts
 * LNET Process Id
 * ME: Matching Entry
 * MD: Memory Descriptor
 * Example Use of Offset
 * MD Options
 * Event Queue
 * Portal RPC: A Client of LNET
 * Get and Put Confusion
 * Router In the Middle
 * Round 1: Client Server Interactions
 * Round 2: More details
 * LNET API
 * Naming Conventions
 * Initialization and Teardown
 * Memory-Oriented Communication Semantics
 * Match Entry Management
 * LNET/LND Semantics and API
 * API Summary
 * LNET Startup and Data Transfer
 * Startup
 * LNET Send
 * LNET Receive
 * The Case for RDMA
 * LNET Routing
 * General Concepts
 * Asymmetric Routing Failure
 * Routing Buffer Management and Flow Control
 * Fine Grain Routing
 * Lustre Generic Filesystem Wraper Layer: fsfilt
 * Overview
 * fsfilt for ext3
 * fsfilt Use Case Examples
 * DIRECT_IO in Lustre
 * Replaying Last Transactions After a Server Crash
 * Client Connect/Disconnect
 * Why ls Is Expensive on Lustre
 * Lustre Disk Filesystem: ldiskfs
 * Kernel Patches
 * Patches: ext3 to ldiskfs
 * Future Work