Lustre Internals Documentation Update

From Lustre Wiki
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Organization

Goals

  • Incremental updating of Lustre internals documentation
  • Near term focus is material in support of upcoming Introduction To Lustre Development webinar
  • The available time for those with Lustre internals knowledge is limited, maximizing the efficiency of their efforts is a priority.

General

  • Updates are moving to Category:Internals and pages with the Internals category in order to better facilitate linking and incremental updating.
  • For the near future full documents (ULFI, LIAGI) are being moved over to a single wiki page. Adding subpage support to MediaWiki is a topic for the next LOWG call and may provide some flexibility there moving forward.
  • Lustre internals is a regular topic on the Lustre Working Group calls, and the Lustre Internals Documentation Update project is discussed regularly on the Lustre.org Working Group calls. Please join us.

Acronyms

  • LIAGI (Lustre Internals: A Gentle Introduction)
  • ULFI (Understanding Lustre Filesystem Internals)

Todo

High

  • Migrate ULFI from shared document to wiki to facilitate targeted linking and updating. (in-progress)
  • Migrate LIAGI from web to wiki
  • Update Subsystem Map for current subsystems
  • Add Layered Object wiki page, reference source-tree clio.txt and LIAGI resources
  • Create OSD internals page and reference source-tree osd-api.txt

Medium

  • Work with developers to identify areas that need more work.
  • Update wiki PtlRPC page and reference PtlRPC ULFI chapter as well as Lustre Protocol Document
  • Generate Doxygen comments from current source tree and put on web
  • Review Doxygen comments in code base and find methods to reference them from wiki
  • Keep Lustre_Internals wiki page?
  • Migrate Lustre Protocol Document to wiki?

Low

Feedback

Feedback from discussions with community members on updating Lustre internals documentation in general:

  • Top-down versus bottom-up approach
    • Differing viewpoints on this. Possible to find way to address both?
  • Separate documents (e.g. Understanding Lustre Filesystem Internals document) versus documentation in the code (e.g. Doxygen)
    • Possible to incrementally expand and update both, balance what's in them, and have them mutually reference?

Resources

Lustre Protocol Documentation

Presentations

Sequoia and the ZFS OSD (LUG2013)

Intel® Lustre* File Level Replication (LUG2014)

Distributed Name Space Phase I (LUG2013)

Lustre & Kerberos: in theory and in practice (LUG2015)

Documentation in Lustre Tree

Doxygen Code Documentation

Overview of the Lustre Client I/O (CLIO) subsystem

http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/clio.txt

LFSCK

http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lfsck.txt

  • LFSCK master slave design
  • Object traversal design reference

Lock Ordering

http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lock-ordering

/* dot input file for lock-ordering diagram */

Overview of the Lustre Object Storage Device API

http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/osd-api.txt

Overview of Dynamic LNet Configuration

http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/dlc.txt

Lustre versioning

http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/versioning.txt

Old Wiki Architectural Documents

Page Comments
Architecture - Backup
Architecture - CROW
Architecture - CTDB with Lustre
Architecture - Caching OSS
Architecture - Changelogs
Architecture - Changelogs 1.6
Architecture - Client Cleanup
Architecture - Clustered Metadata
Architecture - Commit on Share
Architecture - Cuts
Architecture - DMU OSD
Architecture - DMU Zerocopy
Architecture - End-to-end Checksumming
Architecture - Epochs
Architecture - External File Locking
Architecture - FIDs on OST
Architecture - Feature FS Replication
Architecture - Fileset
Architecture - Flash Cache
Architecture - Free Space Management
Architecture - GNS
Architecture - HSM No useful information
Architecture - HSM Migration
Architecture - HSM and Cache No useful information
Architecture - IO system
Architecture - Interoperability 1.6 1.8 2.0
Architecture - Interoperability fids zfs
Architecture - LRE Images
Architecture - Libcfs
Architecture - Llog over OSD
Architecture - Lustre DLDs
Architecture - Lustre HLDs
Architecture - Lustre Logging API
Architecture - MDS-on-DMU
Architecture - MDS striping format
Architecture - MPI IO and NetCDF
Architecture - MPI LND
Architecture - Metadata API
Architecture - Migration (1)
Architecture - Migration (2)
Architecture - Multiple Interfaces For LNET
Architecture - Network Request Scheduler
Architecture - New Metadata API
Architecture - OSS-on-DMU
Architecture - Open by fid
Architecture - PAG
Architecture - Pools of targets
Architecture - Profiling Tools for IO
Architecture - Proxy Cache
Architecture - Punch and Extent Migration
Architecture - Punch and Extent Migration Requirements
Architecture - Recovery Failures
Architecture - Request Redirection
Architecture - Scalable Pinger
Architecture - Security
Architecture - Server Network Striping
Architecture - Simple Space Balance Migration
Architecture - Simplified Interoperation
Architecture - Space Manager
Architecture - Sub Tree Locks
Architecture - User Level Access
Architecture - User Level OSS
Architecture - Userspace Servers
Architecture - Version Based Recovery
Architecture - Wide Striping
Architecture - Wire Level Protocol
Architecture - Write Back Cache
Architecture - Writing Architecture Documents
Architecture - ZFS TinyZAP
Architecture - ZFS for Lustre
Architecture - ZFS large dnodes
Architecture Descriptions

Old Site Lustre Internals Documentation Area

http://wiki.old.lustre.org/lid/

Glossary

http://wiki.old.lustre.org/lid/glossary/glossary.html

Brief descriptions of Lustre concepts, objects and major components indexed in various ways.

Lustre Internals: A Gentle Introduction

http://wiki.old.lustre.org/lid/agi/agi.html

Subsystem Map

TODO: Generate new version of

Retrieved from "http://wiki.lustre.org/index.php?title=Lustre_Internals_Documentation_Update&oldid=3424"