Lustre Manual Changes
UNDER CONSTRUCTION 06/20/2018
Lustre includes an extensive manual. This is developed much like the source code: anyone can submit patches, review changes, and suggest improvements. The manual workflow includes Gerrit and continuous integration practice. This enables changes to be automatically verified and conveniently reviewed using Gerrit.
- 1 Requesting a Change or Improvement
- 2 Submitting a Change to the Manual Source
- 3 Tool Reference
- 4 Change Review Guidelines
Requesting a Change or Improvement
Manual corrections and enhancements are tracked in the same manor as code changes. A Jira project exists, LUDOC, where changes are created and tracked. LUDOC tickets can be created, updated or commented by anyone with an account on JIRA. Once you have an account, go to JIRA and create your LUDOC ticket:
To browse all existing issues, please visit: https://jira.whamcloud.com/browse/LUDOC/
Minor changes and fixes can be committed against the existing LUDOC-11 issue. The intention is to minimize the overhead for quick fixes. For such a change, your commit message may be:
LUDOC-11 quota: spelling corrections Brief description of the change if it is more extensive. Signed-off-by: Random User <[email protected]>
Submitting a Change to the Manual Source
Anyone can create changes that modify the current Lustre manual. The manual is written in Docbook XML version 5, with each chapter residing in a separate xml file. Reference material is readily available on the web for Docbook XML syntax.
Development Environment Setup
The Docbook 5 development environment has been tested with Ubuntu 18.04 (previous Ubuntu versions back to 10.0 are known to work), RHEL7 Linux, and Mac OS X. The auto build process occurs on RHEL7. The original development work for the manual was completed using Ubuntu.
Install the following packages (last validated 2018-06-20):
apt install make git libxml2-utils xsltproc fop docbook5-xml docbook-xsl-ns
Install the following packages:
yum install fop libxml2 libxslt docbook5-style-xsl docbook5-schema docbook5-schemas deja*
Install from EPEL, CentOS, source or Whamcloud toolkit:
yum localinstall fop xmlgraphics-commons-1.4-3
fop and xmlgraphics
- A suitably recent version of fop (>= 1.0) is available for EL7 from Whamcloud toolkit in the 'x86_64' folder.
- A suitably recent version xmlgraphics-commons to provide dependencies on fop >= 1.0 is a available for EL7 from Whamcloud toolkit in the 'noarch' folder.
Install the following packages:
yum install libxml2 libxslt fop docbook5-style-xsl docbook5-schemas docbook5-schemas
Max OS X
It is possible to use either the MacPorts or Homebrew (Brew) package managers on Mac OS X to install the required 3rd-party DocBook tools, instead of compiling them yourself. Only one of these two package managers should be used at one time.
If using the MacPorts package, install the following packages:
sudo port install libxml2 libxslt fop docbook-xml-5.0 docbook-xsl
You may experience problems if you don't use MacPorts. These problems typically can be resolved by editing the Makefile to give the correct path to 'docbookxi.rng'.
If using Brew, install following packages:
brew install libxml2 libxslt fop docbook docbook-xsl
Changing the Manual
The manual is controlled using, Jira, git, and Gerrit (details below). The general workflow is:
- Clone the manual from the "doc/manual" repository
git clone git://git.whamcloud.com/doc/manual lustre-manual
- Make change(s) in a local branch, editing .xml files as needed
git checkout -b b_new_change; vi XXXX.xml
A few notes regarding edits:
- Vim has syntax highlighting, as does emacs. Limited success has been reported with WYSIWYG XML editors.
- If using a WYSIWYG editor, run "git diff" to ensure it didn't reformat the whole .xml file and that only the intended text was modified.
- Text should be wrapped at 80 columns so that it is easier to comment on changes during review. Otherwise, it is difficult to make specific comments on an issue after a whole long paragraph of text that is a single line in the .xml document.
- Ensure you are familiar with Change Review Guidelines specified in this document so that you understand the writing criteria reviewers will apply to qualify your change.
- Check that the document still validates
See Verify the Docbook for more information on validation.
- Build a target (i.e. html or pdf) and visually check that the document looks good
make pdf open lustre_manual.pdf
- Commit the change locally
git commit -a -v -s
- Push the change for review
Note: 'make push' simply calls
git push ssh://review.whamcloud.com:29418/doc/manual HEAD:refs/for/master
- Add two inspectors to the manual
One inspector should be someone with a technical background related to the changes and another person should review for style/English.
Verify the Docbook
Docbook 5 is verfied using a relaxNG schema with schematron rules. The manual is built using separate xml files for each chapter, which use XInclude elements. Two complementary tools support relaxNG schemas with XInclude, xmllint and jing. The tools can be used together to identify problems as shown here. The easiest way to verify the changes are valid XML is to use the make check target:
- This will internally verify with xmllint the ./index.xml of doc/manual.
- OPTIONAL additional verification with jing the ./index.xml of doc/manual:
If make check fails, an error will be reported. Finding the source of the problem can be tricky if incorrectly positioned tags are the underlying issue, i.e. a misplaced <section>. In this case, experience suggests that the problem may be below the indicated problem in the returned error.
Building The Manual
Docbook manuals can be transformed into a number of formats automatically. Two formats of interest for testing are HTML and PDF. HTML transforms are made directly from the Docbook xml (with xsltproc). PDF transforms require transforming into an intermediatory fop xml format first, and then into a PDF with using fop. The easiest way to generate the output files is via the make targets:
- To build a HTML version of the manual
- Similarly, to build a PDF version of the manual
Building an xhtml and epub version of the manual is also supported by the makefile.
There are two types of errors that are common:
- well formed xml
- 'make check' will do a good job of discovering and precisely identifying the location of these types of errors.
- valid xml
- 'make check' will validate the xml but does a poor job of identifying errors when they occur. The first complication is that xinclude is used to include separate xml files. As a result, xmllint is unable to really spot where the problem is.
$ make check xmllint --noout --xinclude --noent --relaxng /usr/share/xml/docbook5/schema/rng/5.0/docbookxi.rng ./index.xml ./index.xml:6: element itemizedlist: Relax-NG validity error : Expecting an element listitem, got nothing ./index.xml:6: element itemizedlist: Relax-NG validity error : Expecting element example, got itemizedlist ./index.xml:63: element section: Relax-NG validity error : Element chapter has extra content: section ./index.xml:1: element chapter: Relax-NG validity error : Did not expect element chapter there ./index.xml:1: element chapter: Relax-NG validity error : Expecting element include, got chapter ./index.xml fails to validate
This complication can be avoided by first flattening the manual into a single xml file:
$ xmllint --xinclude index.xml > /tmp/resolved.xml
Now, you can get a slightly better idea of where the problem is using xmllint – but it's still not that great:
$ xmllint --noout --xinclude --noent --relaxng /usr/share/xml/docbook5/schema/rng/5.0/docbookxi.rng /tmp/resolved.xml /tmp/resolved.xml:8757: element itemizedlist: Relax-NG validity error : Expecting an element listitem, got nothing /tmp/resolved.xml:8757: element itemizedlist: Relax-NG validity error : Expecting element example, got itemizedlist /tmp/resolved.xml:8812: element section: Relax-NG validity error : Element chapter has extra content: section /tmp/resolved.xml:8752: element chapter: Relax-NG validity error : Did not expect element chapter there /tmp/resolved.xml:8752: element chapter: Relax-NG validity error : Expecting element include, got chapter /tmp/resolved.xml fails to validate
However, if you have Jing installed, you can get an even better idea of what the problem is - as well as where it is:
$ jing /usr/share/xml/docbook5/schema/rng/5.0/docbookxi.rng /tmp/resolved.xml /tmp/resolved.xml:9127:29: error: element "para" not allowed here; expected the element end-tag, text or element "abbrev", "accel", "acronym", "address", "alt", "anchor", "annotation", "application", "author", "bibliolist", "biblioref", "blockquote", "bridgehead", "calloutlist", "caution", "citation", "citebiblioid", "citerefentry", "citetitle", "classname", "classsynopsis", "cmdsynopsis", "code", "command", "computeroutput", "constant", "constraintdef", "constructorsynopsis", "coref", "database", "date", "destructorsynopsis", "editor", "email", "emphasis", "envar", "epigraph", "equation", "errorcode", "errorname", "errortext", "errortype", "example", "exceptionname", "fieldsynopsis", "figure", "filename", "firstterm", "footnote", "footnoteref", "foreignphrase", "funcsynopsis", "function", "glosslist", "glossterm", "guibutton", "guiicon", "guilabel", "guimenu", "guimenuitem", "guisubmenu", "hardware", "important", "indexterm", "informalequation", "informalexample", "informalfigure", "informaltable", "initializer", "inlineequation", "inlinemediaobject", "interfacename", "itemizedlist", "jobtitle", "keycap", "keycode", "keycombo", "keysym", "link", "literal", "literallayout", "markup", "mediaobject", "menuchoice", "methodname", "methodsynopsis", "modifier", "mousebutton", "msgset", "nonterminal", "note", "olink", "ooclass", "ooexception", "oointerface", "option", "optional", "orderedlist", "org", "orgname", "package", "parameter", "person", "personname", "phrase", "procedure", "productionset", "productname", "productnumber", "programlisting", "programlistingco", "prompt", "property", "qandaset", "quote", "remark", "replaceable", "returnvalue", "revhistory", "screen", "screenco", "screenshot", "segmentedlist", "shortcut", "sidebar", "simplelist", "subscript", "superscript", "symbol", "synopsis", "systemitem", "table", "tag", "task", "termdef", "tip", "token", "trademark", "type", "uri", "userinput", "variablelist", "varname", "warning", "wordasword", "xi:include" or "xref"
Jira has a project specifically for Manual changes: LUDOC.
See Changing the Manual for command line reference.
Using Gerrit describes how to submit changes to the lustre-release repository. The same git configuration will work for the manual.
- Editing Tools
Docbook is xml and as such can be edited with a xml, or text editor. xsd aware editors provide additional functionality that users may find advantageous. The following observations are recorded for the tools below:
- oXygen XML editor
oXygen make a number of xml editing products that support docbook5. See: [http://www.oxygenxml.com] for more information.
Serna supports XSD and has been found to work well for editing blocks of existing text. However, it can be tricky to add significant new content. It is kind of WYSIWYG but allows an unsuspecting user to shift elements into invalid positions. The version tested was the free version on Ubuntu 11.10. Serna doesn't have Docbook 5 support. Fortunately, you can tell Serna to use the Docbook 5 XSD (on Ubuntu):
- open 'index.xml' from the manual.
- 'choose template'
- choose 'Docbook V4.5'
- 'Details...' 'Schema:' /usr/share/xml/docbook/schema/xsd/5.0/docbook.xsd
Problem: Serna doesn't seem to like xinclude. xlinclue is used to break the manual into multiple separate xml files. The result of this is that Serna complains that the document is not valid and it cannot resolve xrefs. The screenshot below illustrates this:
Work-around: Use the
make check. command line for validation.
Vim works well, and is a good choice for developers that are already making changes to the Lustre code, since it is configured to provide XML syntax highlighting by default. Any other text editor would also be usable. You might find the following commands useful for turning on xml sensitive spell checking:
:syn spell toplevel :redraw :setlocal spell
Change Review Guidelines
Like code, Manual changes are submitted to a review process.
- All the manual patches that are in Gerrit for review are publicly accessible for viewing. Logged-in users can review previous versions of the Manual.
- Updates to the Manual are automatically built from the change and available as a review Manual build. The links to these builds are in the comments of a change.
Rules on Proper Use of Other Party Trademarks
- Designate other party trademarks with a superscript asterisk (*)
- In the first appearance in a headline, and/or the first prominent use.
- In the first appearance in the body of text.
- The asterisk (*) may go at the end of the other party's trademarked name or at the end of the product name.
- Lustre* software
- HP Pavilion system*
- Adobe Acrobat* software
- Use an appropriate noun after the trademark.
- On a Lustre* file system
- Use simple sentence structure. Break up long sentences.
- Describe a series of steps as a numbered list.
- Use active voice. You can often avoid passive voice by using second person ("you") or imperative voice ("Do this...").
- Wrong example (passive voice):
- The module files should be created before they are referenced by another program.
- Correct example (active voice):
- Make sure that your program creates the module files before another program references them.
- Wrong example (passive voice):
- Use present tense whenever possible.
- Wrong example:
- When you press <CR>, the program will use the default value of zero
- Correct example:
- When you press <CR>, the program uses the default value of zero
- Wrong example:
- Use parallel constructs for heading titles, table entries, numbered lists, and bulleted lists.
- Correct example (all bullets start with a verb):
*The relay uses the system services interface to perform these functions: **Allocate, modify, and free a system buffer. **Send and receive data messages with modules. **Manage timers and scheduling.
- Correct example (all bullets start with a gerund):
*The relay uses the system services interface for the following functions: **Allocating, modifying, and freeing a system buffer **Sending and receiving data messages with modules **Managing timers and scheduling
- Use second person ("you") when referring to the target reader.
- Use third person entities for other than the targeted reader ("users", "programmers").
- Do not use first person references ("we").
- Gender neutrality should be maintained; do not use these terms: "he", "she", "his", "her", or "him". Rewrite to avoid these terms if possible. If necessary, use third person ("their" rather than "his/her").
- Lustre version specific hints
- [LUDOC-104] introduces a set of conditional element values that declares a section, para, etc... as only relevant for a particular Lustre release. The currently supported values for the 'condition' attribute are: l21, l22, l23, l24, l25, l26, l27, l28, l29, l2A, l2B.. These values correspond to 'Introduced in Lustre 2.3' up through 'Introduced in Lustre 2.11', respectively. An example of the decoration created by including a conditional value of 'l2B' is available in the Manual if you search for 'Lustre 2.11'.
- Always visit the HTML or PDF version of the document.
- In addition to using the 'diff all side-by-side' to inspect changes within Gerrit, it is vital to check the document rendering well.
- The number of lines that a typical change touches can be numerous. Viewing these changes in Gerrit is useful to ensure readability, grammar and style. Inspecting the change after a HTML or PDF build ensures that formatting is correct. Checking formatting within Gerrit is time-consuming and tricky, it is much easier once formatting is rendered.
- Ensure that xml readability and format is not degraded.
- Changes may be submitted from GUI xml editors. This is fine - as long as the XML is well formatted for humans to read.