Lustre Manual Changes

UNDER CONSTRUCTION 06/20/2018

Lustre includes an extensive manual. This is developed much like the source code: anyone can submit patches, review changes and suggest improvements. The manual workflow includes Gerrit and "continuous integration" practice. This enables changes to be automatically verified and conveniently reviewed using Gerrit.

Requesting a Change or Improvement
Manual corrections and enhancements are tracked in the same manor as code changes. A Jira project, LUDOC, where the changes created and tracked. LUDOC tickets can be created, updated or commented by anyone with an account on. Once you have an account, go to JIRA and create your LUDOC ticket:

https://jira.whamcloud.com/browse/LUDOC/

Submitting a Change to the Manual Source
Anyone can create changes that modify the current Lustre manual. The manual is written in Docbook XML version 5.

Quick Summary
The following list is a short summary of the steps required, with full details in the following section. This information is useful for those who are familiar with Docbook and git setup procedures for the manual.
 * Create a LUDOC bug for change tracking in Jira. For changes related to a new feature with an associated LU-nnnn ticket, the LU ticket may also be used.

Minor changes
Minor changes and fixes can be committed against the existing LUDOC-11 issue. The intention is to minimize the overhead for quick fixes. For such a change, your commit message may be:  LUDOC-11 quota: spelling corrections Brief description of the change if it is more extensive. Signed-off-by: Random User 


 * Clone the manual from the "doc/manual" repository: git clone git://git.whamcloud.com/doc/manual lustre-manual
 * Edit the .xml files as needed (vim has syntax highlighting, emacs too. Limited success has been reported with WYSIWYG XML editors, see below.)
 * Run make check to validate the XML.
 * Run make pdf, make xhtml to generate a PDF or xhtml file and visually inspect your change.
 * If using a WYSIWYG editor, run "git diff" to ensure it didn't reformat the whole .xml file, only the text that was modified
 * Commit the patch to your local repository (git commit -a -v -s), with a proper Commit Comment. If your change is documenting a feature that has a related LU-nnnn ticket, include the LU ticket number in the commit message.
 * Run make push to submit the patch to Gerrit for build/inspection (git push ssh://review.whamcloud.com:29418/doc/manual HEAD:refs/for/master).
 * Add two inspectors to the manual: someone with a technical background related to the changes and someone for style/English.

Notes:
 * Building the manual only takes a minute or so on the build servers. If you want to avoid installing the whole tool chain locally, you can just edit the .xml file(s) and post it to Gerrit for validation and build.
 * The manual is kept in Docbook format with multiple .xml files. Each chapter is in a separate file.
 * Ensure you are familiar with Guidance for Manual Reviewers in this document so that you understand the writing criteria reviewers will apply to qualify your change.
 * Reference material for Docbook 5 is available at . This is a good place to lookup the meaning of the various elements.
 * Text should be wrapped at 80 columns so that it is easier to comment on changes during review. Otherwise, it is difficult to make specific comments on an issue after a whole long paragraph of text that is a single line in the .xml document.

Development environment for Docbook 5
Docbook 5 development environment has been tested with Ubuntu 18.04, RHEL7 Linux, as well as Mac OS X. The auto build process occurs on RHEL7. The original development work for the manual was completed using Ubuntu.

Ubuntu 12.04
Install the following packages for ability to build all targets:

Ubuntu 18.04
Install the following packages (last validated 2018-06-20):

RHEL 7
Install the following packages:

Install from EPEL, CentOS, source or Whamcloud toolkit:

fop and xmlgraphics
 * A suitably recent version of fop (>= 1.0) is available for EL7 from Whamcloud toolkit in the 'x86_64' folder.
 * A suitably recent version xmlgraphics-commons to provide dependencies on fop >= 1.0 is a available for EL7 from Whamcloud toolkit in the 'noarch' folder.

Fedora 18
Install the following packages:

Max OS X
It is possible to use either the MacPorts or Homebrew (Brew) package managers on Mac OS X to install the required 3rd-party DocBook tools, instead of compiling them yourself. Only one of these two package managers should be used at one time.

If using the MacPorts package, install the following packages:

You may experience problems if you don't use MacPorts. These problems typically can be resolved by editing the Makefile to give the correct path to 'docbookxi.rng'. If using Brew install following packages:

Workflow: Changing the Manual
The manual is controlled with Jira with git and Gerrit (details below). The general workflow is:  git clone git://git.whamcloud.com/doc/manual lustre-manual  git checkout -b b_new_change; vi XXXX.xml  make check && make pdf  open lustre_manual.pdf  git commit -a -v -s  make push This is similar to the process for changing Lustre source. Verifying the document and building a html/pdf file can be performed with the following commands (tested on Mac OS X, Ubuntu 18.04 and RHEL7).
 * git clone the manual:
 * Make change in a local branch:
 * Verify the document is still valid:
 * Check the document looks good:
 * Commit the change locally:
 * Push the change for review:

Verify the Docbook
Docbook 5 is verfied using a relaxNG schema with schematron rules. The Manual is build using separate xml files for each chapter which uses XInclude elements. Two complementary tools support relaxNG schemas with XInclude. The tools can be used together to identify problems, and they are xmllint and jing. The easiest way to verify the changes are valid XML is to use the make check target:  make check
 * This will internally verify with xmllint the ./index.xml of doc/manual.
 * OPTIONAL additional verification with jing the ./index.xml of doc/manual:

Fixing Problems
If make check fails, an error will be reported. Finding the source of the problem can be tricky if incorrectly positioned tags are the problem, i.e. a misplaced. In this case, experience suggests that the problem may be below the indicated problem in the returned error.

Building The Manual
Docbook manuals can be transformed into a number of formats automatically. Two formats of interest are HTML and PDF. HTML transforms can be made directly from the Docbook xml (with xsltproc). PDF transforms require transforming first into an intermediatory fop xml format, and then into a PDF with a tool called fop. The easiest way to generate the output files is via the make targets:  make html  make pdf Building a xhtml and epub version of the manual is also supported by the makefile.
 * To build a HTML version of the manual:
 * Similarly, to build a PDF version of the manual:

Debugging
There are two types of errors that are common:
 * 1) well formed xml
 * 'make check' will do a good job of discovering and precisely identifying the location of these types of errors.
 * 1) valid xml
 * 'make check' will validate the xml but does a poor job of identifying errors when they occur. The first complication is that xinclude is used to include separate xml files. As a result, xmllint is unable to really spot where the problem is.

Example
 $ make check xmllint --noout --xinclude --noent --relaxng /usr/share/xml/docbook5/schema/rng/5.0/docbookxi.rng ./index.xml ./index.xml:6: element itemizedlist: Relax-NG validity error : Expecting an element listitem, got nothing ./index.xml:6: element itemizedlist: Relax-NG validity error : Expecting element example, got itemizedlist ./index.xml:63: element section: Relax-NG validity error : Element chapter has extra content: section ./index.xml:1: element chapter: Relax-NG validity error : Did not expect element chapter there ./index.xml:1: element chapter: Relax-NG validity error : Expecting element include, got chapter ./index.xml fails to validate

This complication can be avoided by first flattening the manual into a single xml file:  $ xmllint --xinclude  index.xml  >  /tmp/resolved.xml

Now, you can get a slightly better idea of where the problem is using xmllint – but it's still not that great:  $ xmllint --noout --xinclude --noent --relaxng /usr/share/xml/docbook5/schema/rng/5.0/docbookxi.rng /tmp/resolved.xml /tmp/resolved.xml:8757: element itemizedlist: Relax-NG validity error : Expecting an element listitem, got nothing /tmp/resolved.xml:8757: element itemizedlist: Relax-NG validity error : Expecting element example, got itemizedlist /tmp/resolved.xml:8812: element section: Relax-NG validity error : Element chapter has extra content: section /tmp/resolved.xml:8752: element chapter: Relax-NG validity error : Did not expect element chapter there /tmp/resolved.xml:8752: element chapter: Relax-NG validity error : Expecting element include, got chapter /tmp/resolved.xml fails to validate

However, if you have Jing installed, you can get an even better idea of what the problem is - as well as where it is:  $ jing /usr/share/xml/docbook5/schema/rng/5.0/docbookxi.rng /tmp/resolved.xml /tmp/resolved.xml:9127:29: error: element "para" not allowed here; expected the element end-tag, text or element "abbrev", "accel", "acronym", "address", "alt", "anchor", "annotation", "application", "author", "bibliolist", "biblioref", "blockquote", "bridgehead", "calloutlist", "caution", "citation", "citebiblioid", "citerefentry", "citetitle", "classname", "classsynopsis", "cmdsynopsis", "code", "command", "computeroutput", "constant", "constraintdef", "constructorsynopsis", "coref", "database", "date", "destructorsynopsis", "editor", "email", "emphasis", "envar", "epigraph", "equation", "errorcode", "errorname", "errortext", "errortype", "example", "exceptionname", "fieldsynopsis", "figure", "filename", "firstterm", "footnote", "footnoteref", "foreignphrase", "funcsynopsis", "function", "glosslist", "glossterm", "guibutton", "guiicon", "guilabel", "guimenu", "guimenuitem", "guisubmenu", "hardware", "important", "indexterm", "informalequation", "informalexample", "informalfigure", "informaltable", "initializer", "inlineequation", "inlinemediaobject", "interfacename", "itemizedlist", "jobtitle", "keycap", "keycode", "keycombo", "keysym", "link", "literal", "literallayout", "markup", "mediaobject", "menuchoice", "methodname", "methodsynopsis", "modifier", "mousebutton", "msgset", "nonterminal", "note", "olink", "ooclass", "ooexception", "oointerface", "option", "optional", "orderedlist", "org", "orgname", "package", "parameter", "person", "personname", "phrase", "procedure", "productionset", "productname", "productnumber", "programlisting", "programlistingco", "prompt", "property", "qandaset", "quote", "remark", "replaceable", "returnvalue", "revhistory", "screen", "screenco", "screenshot", "segmentedlist", "shortcut", "sidebar", "simplelist", "subscript", "superscript", "symbol", "synopsis", "systemitem", "table", "tag", "task", "termdef", "tip", "token", "trademark", "type", "uri", "userinput", "variablelist", "varname", "warning", "wordasword", "xi:include" or "xref"

Other Tools
Jira has a project specifically for Manual changes: LUDOC.  git clone git://git.whamcloud.com/doc/manual lustre-manual <pre style="overflow-x:auto;"> make push Internally this does something like: <pre style="overflow-x:auto;"> git push ssh://USERNAME@review.whamcloud.com:29418/doc/manual HEAD:refs/for/master Using Gerrit describes how to submit changes to the manual. Criteria for manual changes to land a change on master is: +2 (+1 from two human reviewers) and +1 verified from Jenkins.
 * Jira
 * Git
 * Clone the manual as follows:
 * Push changes for review:
 * Gerrit

= Reviewing a Change = Like code, Manual changes are submitted to a review process. All the manual patches that are in Gerrit for review can be seen. Logged-in users can review previous versions of the Manual. Updates to the Manual are automatically built from the change and available as a review Manual build, links to these builds are in the comments of a change. Reviewers are advised to read the guidance for Manual reviewers in this section.