Lustre Internals Documentation Update: Difference between revisions
Jump to navigation
Jump to search
KenRawlings (talk | contribs) (→Medium) |
KenRawlings (talk | contribs) m (Updating links) |
||
| (8 intermediate revisions by the same user not shown) | |||
| Line 4: | Line 4: | ||
* Incremental updating of Lustre internals documentation | * Incremental updating of Lustre internals documentation | ||
* Near term focus is | * Near term focus is material in support of upcoming Introduction To Lustre Development webinar | ||
* The available time for those with Lustre internals knowledge is limited, maximizing the efficiency of their efforts is a priority. | * The available time for those with Lustre internals knowledge is limited, maximizing the efficiency of their efforts is a priority. | ||
| Line 10: | Line 10: | ||
* Updates are moving to [[:Category:Internals]] and pages with the Internals category in order to better facilitate linking and incremental updating. | * Updates are moving to [[:Category:Internals]] and pages with the Internals category in order to better facilitate linking and incremental updating. | ||
* For the near future full documents (ULFI, | * For the near future full documents (ULFI, LIAGI) are being moved over to a single wiki page. Adding subpage support to MediaWiki is a topic for the next LOWG call and may provide some flexibility there moving forward. | ||
* Lustre internals is a regular topic on the [http://wiki.opensfs.org/Lustre_Working_Group Lustre Working Group] calls, and the Lustre Internals Documentation Update project is discussed regularly on the [http://wiki.lustre.org/Lustre.org_Working_Group Lustre.org Working Group] calls. Please join us. | * Lustre internals is a regular topic on the [http://wiki.opensfs.org/Lustre_Working_Group Lustre Working Group] calls, and the Lustre Internals Documentation Update project is discussed regularly on the [http://wiki.lustre.org/Lustre.org_Working_Group Lustre.org Working Group] calls. Please join us. | ||
| Line 18: | Line 18: | ||
* ULFI (Understanding Lustre Filesystem Internals) | * ULFI (Understanding Lustre Filesystem Internals) | ||
== Todo == | |||
=== High === | |||
* Migrate ULFI from shared document to wiki to facilitate targeted linking and updating. (in-progress) | * Migrate ULFI from shared document to wiki to facilitate targeted linking and updating. (in-progress) | ||
| Line 28: | Line 28: | ||
* Create OSD internals page and reference source-tree osd-api.txt | * Create OSD internals page and reference source-tree osd-api.txt | ||
=== Medium === | |||
* Update wiki PtlRPC page and reference PtlRPC ULFI chapter as well as Protocol Document | * Work with developers to identify areas that need more work. | ||
* Update wiki PtlRPC page and reference PtlRPC ULFI chapter as well as Lustre Protocol Document | |||
* Generate Doxygen comments from current source tree and put on web | * Generate Doxygen comments from current source tree and put on web | ||
* Review Doxygen comments in code base and find methods to reference them from wiki | * Review Doxygen comments in code base and find methods to reference them from wiki | ||
* Keep [[Lustre_Internals]] wiki page? | * Keep [[Lustre_Internals]] wiki page? | ||
* Migrate Protocol Document to wiki? | * Migrate Lustre Protocol Document to wiki? | ||
=== Low === | |||
* Automate generation/uploading of Doxygen comments to the web | * Automate generation/uploading of Doxygen comments to the web | ||
| Line 56: | Line 57: | ||
* http://wiki.opensfs.org/Contract_SFS-DEV-005 | * http://wiki.opensfs.org/Contract_SFS-DEV-005 | ||
* https://jira. | * https://jira.whamcloud.com/browse/LUDOC-280 | ||
* https://build. | * https://build.whamcloud.com/job/lustre-protocol-reviews/lastSuccessfulBuild/artifact/protocol.html | ||
=== Presentations === | === Presentations === | ||
| Line 82: | Line 83: | ||
==== Overview of the Lustre Client I/O (CLIO) subsystem ==== | ==== Overview of the Lustre Client I/O (CLIO) subsystem ==== | ||
http://git. | http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/clio.txt | ||
==== LFSCK ==== | ==== LFSCK ==== | ||
http://git. | http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lfsck.txt | ||
* LFSCK master slave design | * LFSCK master slave design | ||
| Line 93: | Line 94: | ||
==== Lock Ordering ==== | ==== Lock Ordering ==== | ||
http://git. | http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lock-ordering | ||
/* dot input file for lock-ordering diagram */ | /* dot input file for lock-ordering diagram */ | ||
| Line 99: | Line 100: | ||
==== Overview of the Lustre Object Storage Device API ==== | ==== Overview of the Lustre Object Storage Device API ==== | ||
http://git. | http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/osd-api.txt | ||
==== Overview of Dynamic LNet Configuration ==== | ==== Overview of Dynamic LNet Configuration ==== | ||
http://git. | http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/dlc.txt | ||
==== Lustre versioning ==== | ==== Lustre versioning ==== | ||
http://git. | http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/versioning.txt | ||
=== Old Wiki Architectural Documents === | === Old Wiki Architectural Documents === | ||
| Line 279: | Line 280: | ||
TODO: Generate new version of | TODO: Generate new version of | ||
* http://wiki.old.lustre.org/lid/subsystem-map-old/subsystem-map.html | * http://wiki.old.lustre.org/lid/subsystem-map-old/subsystem-map.html | ||
Latest revision as of 09:00, 18 October 2018
Organization
Goals
- Incremental updating of Lustre internals documentation
- Near term focus is material in support of upcoming Introduction To Lustre Development webinar
- The available time for those with Lustre internals knowledge is limited, maximizing the efficiency of their efforts is a priority.
General
- Updates are moving to Category:Internals and pages with the Internals category in order to better facilitate linking and incremental updating.
- For the near future full documents (ULFI, LIAGI) are being moved over to a single wiki page. Adding subpage support to MediaWiki is a topic for the next LOWG call and may provide some flexibility there moving forward.
- Lustre internals is a regular topic on the Lustre Working Group calls, and the Lustre Internals Documentation Update project is discussed regularly on the Lustre.org Working Group calls. Please join us.
Acronyms
- LIAGI (Lustre Internals: A Gentle Introduction)
- ULFI (Understanding Lustre Filesystem Internals)
Todo
High
- Migrate ULFI from shared document to wiki to facilitate targeted linking and updating. (in-progress)
- Migrate LIAGI from web to wiki
- Update Subsystem Map for current subsystems
- Add Layered Object wiki page, reference source-tree clio.txt and LIAGI resources
- Create OSD internals page and reference source-tree osd-api.txt
Medium
- Work with developers to identify areas that need more work.
- Update wiki PtlRPC page and reference PtlRPC ULFI chapter as well as Lustre Protocol Document
- Generate Doxygen comments from current source tree and put on web
- Review Doxygen comments in code base and find methods to reference them from wiki
- Keep Lustre_Internals wiki page?
- Migrate Lustre Protocol Document to wiki?
Low
- Automate generation/uploading of Doxygen comments to the web
- Investigate lustre.org mirror of Lustre git source tree and resources available to facilitate web presentation of the source (e.g. http://elixir.free-electrons.com/linux/latest/source )
Feedback
Feedback from discussions with community members on updating Lustre internals documentation in general:
- Top-down versus bottom-up approach
- Differing viewpoints on this. Possible to find way to address both?
- Separate documents (e.g. Understanding Lustre Filesystem Internals document) versus documentation in the code (e.g. Doxygen)
- Possible to incrementally expand and update both, balance what's in them, and have them mutually reference?
Resources
Lustre Protocol Documentation
- http://wiki.opensfs.org/Contract_SFS-DEV-005
- https://jira.whamcloud.com/browse/LUDOC-280
- https://build.whamcloud.com/job/lustre-protocol-reviews/lastSuccessfulBuild/artifact/protocol.html
Presentations
Sequoia and the ZFS OSD (LUG2013)
Intel® Lustre* File Level Replication (LUG2014)
Distributed Name Space Phase I (LUG2013)
Lustre & Kerberos: in theory and in practice (LUG2015)
Documentation in Lustre Tree
Doxygen Code Documentation
- Old: http://wiki.old.lustre.org/lid/doxygen.api/modules.html
- TODO: Generate new version from codebase and make available on web.
Overview of the Lustre Client I/O (CLIO) subsystem
http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/clio.txt
LFSCK
http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lfsck.txt
- LFSCK master slave design
- Object traversal design reference
Lock Ordering
http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/lock-ordering
/* dot input file for lock-ordering diagram */
Overview of the Lustre Object Storage Device API
http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/osd-api.txt
Overview of Dynamic LNet Configuration
http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/dlc.txt
Lustre versioning
http://git.whamcloud.com/fs/lustre-release.git/blob_plain/HEAD:/Documentation/versioning.txt
Old Wiki Architectural Documents
Old Site Lustre Internals Documentation Area
http://wiki.old.lustre.org/lid/
Glossary
http://wiki.old.lustre.org/lid/glossary/glossary.html
Brief descriptions of Lustre concepts, objects and major components indexed in various ways.
Lustre Internals: A Gentle Introduction
http://wiki.old.lustre.org/lid/agi/agi.html
Subsystem Map
TODO: Generate new version of