Lustre Internals Documentation Update

Goals

 * Incremental updating of Lustre internals documentation
 * Near term focus is material in support of upcoming Introduction To Lustre Development webinar
 * The available time for those with Lustre internals knowledge is limited, maximizing the efficiency of their efforts is a priority.

General

 * Updates are moving to Category:Internals and pages with the Internals category in order to better facilitate linking and incremental updating.
 * For the near future full documents (ULFI, LIAGI) are being moved over to a single wiki page. Adding subpage support to MediaWiki is a topic for the next LOWG call and may provide some flexibility there moving forward.
 * Lustre internals is a regular topic on the Lustre Working Group calls, and the Lustre Internals Documentation Update project is discussed regularly on the Lustre.org Working Group calls. Please join us.

Acronyms

 * LIAGI (Lustre Internals: A Gentle Introduction)
 * ULFI (Understanding Lustre Filesystem Internals)

High

 * Migrate ULFI from shared document to wiki to facilitate targeted linking and updating. (in-progress)
 * Migrate LIAGI from web to wiki
 * Update Subsystem Map for current subsystems
 * Add Layered Object wiki page, reference source-tree clio.txt and LIAGI resources
 * Create OSD internals page and reference source-tree osd-api.txt

Medium

 * Work with developers to identify areas that need more work.
 * Update wiki PtlRPC page and reference PtlRPC ULFI chapter as well as Lustre Protocol Document
 * Generate Doxygen comments from current source tree and put on web
 * Review Doxygen comments in code base and find methods to reference them from wiki
 * Keep Lustre_Internals wiki page?
 * Migrate Lustre Protocol Document to wiki?

Low

 * Automate generation/uploading of Doxygen comments to the web
 * Investigate lustre.org mirror of Lustre git source tree and resources available to facilitate web presentation of the source (e.g. http://elixir.free-electrons.com/linux/latest/source )

Feedback
Feedback from discussions with community members on updating Lustre internals documentation in general:


 * Top-down versus bottom-up approach
 * Differing viewpoints on this. Possible to find way to address both?


 * Separate documents (e.g. Understanding Lustre Filesystem Internals document) versus documentation in the code (e.g. Doxygen)
 * Possible to incrementally expand and update both, balance what's in them, and have them mutually reference?

Lustre Protocol Documentation

 * http://wiki.opensfs.org/Contract_SFS-DEV-005
 * https://jira.hpdd.intel.com/browse/LUDOC-280
 * https://build.hpdd.intel.com/job/lustre-protocol-reviews/lastSuccessfulBuild/artifact/protocol.html

Presentations
Sequoia and the ZFS OSD (LUG2013)
 * http://cdn.opensfs.org/wp-content/uploads/2013/04/Morrone_Sequoia_LUG2013.pdf

Intel® Lustre* File Level Replication (LUG2014)
 * http://wiki.lustre.org/images/a/ad/LUG2014-File_Level_Replication-Xiong.pdf

Distributed Name Space Phase I (LUG2013)
 * http://cdn.opensfs.org/wp-content/uploads/2013/04/LUG-2013_DNE.pdf

Lustre & Kerberos: in theory and in practice (LUG2015)
 * http://wiki.lustre.org/images/e/ec/Lustre-and-Kerberos_Buisson.pdf

Doxygen Code Documentation

 * Old: http://wiki.old.lustre.org/lid/doxygen.api/modules.html
 * TODO: Generate new version from codebase and make available on web.

Overview of the Lustre Client I/O (CLIO) subsystem
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/clio.txt

LFSCK
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lfsck.txt


 * LFSCK master slave design
 * Object traversal design reference

Lock Ordering
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lock-ordering

/* dot input file for lock-ordering diagram */

Overview of the Lustre Object Storage Device API
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/osd-api.txt

Overview of Dynamic LNet Configuration
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/dlc.txt

Lustre versioning
http://git.hpdd.intel.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/versioning.txt

Old Site Lustre Internals Documentation Area
http://wiki.old.lustre.org/lid/

Glossary
http://wiki.old.lustre.org/lid/glossary/glossary.html

Brief descriptions of Lustre concepts, objects and major components indexed in various ways.

Lustre Internals: A Gentle Introduction
http://wiki.old.lustre.org/lid/agi/agi.html

Subsystem Map
TODO: Generate new version of
 * http://wiki.old.lustre.org/lid/subsystem-map-old/subsystem-map.html