Lustre Manual Changes: Difference between revisions

From Lustre Wiki
Jump to navigation Jump to search
mNo edit summary
mNo edit summary
Line 3: Line 3:
'''UNDER CONSTRUCTION 06/20/2018'''
'''UNDER CONSTRUCTION 06/20/2018'''


Lustre includes an [https://mediawiki.org extensive manual].  This is developed much like the source code: anyone can submit patches, review changes and suggest improvements. The manual workflow includes Gerrit and [http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/ "continuous integration"] practice.  This enables changes to be automatically verified and conveniently reviewed using Gerrit.
Lustre includes an [https://mediawiki.org extensive manual].  This is developed much like the source code: anyone can submit patches, review changes, and suggest improvements. The manual workflow includes Gerrit and [http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/ continuous integration] practice.  This enables changes to be automatically verified and conveniently reviewed using Gerrit.


__TOC__
__TOC__


== Requesting a Change or Improvement ==
== Requesting a Change or Improvement ==
Manual corrections and enhancements are tracked in the same manor as code changes. A Jira project exists, [http://jira.whamcloud.com/browse/LUDOC LUDOC], where the changes created and tracked. LUDOC tickets can be created, updated or commented by anyone with an account on [http://jira.whamcloud.com/].  Once you have an account, go to JIRA and create your LUDOC ticket:
Manual corrections and enhancements are tracked in the same manor as code changes. A Jira project exists, [http://jira.whamcloud.com/browse/LUDOC LUDOC], where changes are created and tracked. LUDOC tickets can be created, updated or commented by anyone with an account on [http://jira.whamcloud.com/ JIRA].  Once you have an account, go to [http://jira.whamcloud.com/ JIRA] and create your [http://jira.whamcloud.com/browse/LUDOC LUDOC] ticket:


https://jira.whamcloud.com/browse/LUDOC/
To browse all existing issues, please visit:  https://jira.whamcloud.com/browse/LUDOC/


===Minor changes===
===Minor changes===
Line 21: Line 21:


== Submitting a Change to the Manual Source ==
== Submitting a Change to the Manual Source ==
Anyone can create changes that modify the current Lustre manual. The manual is written in Docbook XML version 5, which each chapter residing in a separate xml file.  [https://tdg.docbook.org/tdg/5.0/docbook.html Reference material] is readily available on the web for Docbook XML syntax.
Anyone can create changes that modify the current Lustre manual. The manual is written in Docbook XML version 5, with each chapter residing in a separate xml file.  [https://tdg.docbook.org/tdg/5.0/docbook.html Reference material] is readily available on the web for Docbook XML syntax.


===Development Environment Setup===
===Development Environment Setup===
Docbook 5 development environment has been tested with Ubuntu 18.04, RHEL7 Linux, as well as Mac OS X. The auto build process occurs on RHEL7. The original development work for the manual was completed using Ubuntu.
The Docbook 5 development environment has been tested with Ubuntu 18.04 (previous Ubuntu versions back to 10.0 are known to work), RHEL7 Linux, and Mac OS X. The auto build process occurs on RHEL7. The original development work for the manual was completed using Ubuntu.
==== Ubuntu 18.04 ====
==== Ubuntu 18.04 ====
Install the following packages (last validated 2018-06-20):
Install the following packages (last validated 2018-06-20):
Line 62: Line 62:
You may experience problems if you don't use MacPorts. These problems typically can be resolved by editing the Makefile to give the correct path to 'docbookxi.rng'.
You may experience problems if you don't use MacPorts. These problems typically can be resolved by editing the Makefile to give the correct path to 'docbookxi.rng'.
   
   
If using Brew install following packages:
If using Brew, install following packages:
<code>
<code>
brew install libxml2 libxslt fop docbook docbook-xsl
brew install libxml2 libxslt fop docbook docbook-xsl
Line 68: Line 68:


===Changing the Manual ===
===Changing the Manual ===
The manual is controlled with Jira with git and Gerrit (details below).  The general workflow is:
The manual is controlled using, Jira, git, and Gerrit (details below).  The general workflow is:
;Clone the manual from the "doc/manual" repository:
;Clone the manual from the "doc/manual" repository:
<pre style="overflow-x:auto;">
<pre style="overflow-x:auto;">
git clone git://git.whamcloud.com/doc/manual lustre-manual
git clone git://git.whamcloud.com/doc/manual lustre-manual
</pre>
</pre>
;Make change in a local branch, editing .xml files as needed:
;Make change(s) in a local branch, editing .xml files as needed:
<pre style="overflow-x:auto;">
<pre style="overflow-x:auto;">
git checkout -b b_new_change; vi XXXX.xml
git checkout -b b_new_change; vi XXXX.xml
Line 87: Line 87:
make check
make check
</pre>
</pre>
See [[#Verify the Docbook|Verify the Docbook]] for more information on validation.
;Build a target (i.e. html or pdf) and visually check that the document looks good:
;Build a target (i.e. html or pdf) and visually check that the document looks good:
<pre style="overflow-x:auto;">
<pre style="overflow-x:auto;">
Line 101: Line 103:
</pre>
</pre>
Note:  'make push' simply calls <code>git push ssh://review.whamcloud.com:29418/doc/manual HEAD:refs/for/master</code>
Note:  'make push' simply calls <code>git push ssh://review.whamcloud.com:29418/doc/manual HEAD:refs/for/master</code>
;Add two inspectors to the manualsomeone with a technical background related to the changes and someone for style/English.
;Add two inspectors to the manual
One inspector should be someone with a technical background related to the changes and another person should review for style/English.


=== Verify the Docbook ===
=== Verify the Docbook ===
Docbook 5 is verfied using a relaxNG schema with schematron rules. The Manual is build using separate xml files for each chapter which uses XInclude elements. Two complementary tools support relaxNG schemas with XInclude. The tools can be used together to identify problems, and they are xmllint and jing.
Docbook 5 is verfied using a relaxNG schema with schematron rules. The manual is built using separate xml files for each chapter, which use XInclude elements. Two complementary tools support relaxNG schemas with XInclude, xmllint and jing. The tools can be used together to identify problems as shown here.
The easiest way to verify the changes are valid XML is to use the '''make check''' target:
The easiest way to verify the changes are valid XML is to use the '''make check''' target:
<pre style="overflow-x:auto;">
<pre style="overflow-x:auto;">
Line 112: Line 115:
*OPTIONAL additional verification with jing the ./index.xml of doc/manual:
*OPTIONAL additional verification with jing the ./index.xml of doc/manual:
====Fixing Problems====
====Fixing Problems====
If make check fails, an error will be reported. Finding the source of the problem can be tricky if incorrectly positioned tags are the problem, i.e. a misplaced <section>. In this case, experience suggests that the problem may be below the indicated problem in the returned error.
If make check fails, an error will be reported. Finding the source of the problem can be tricky if incorrectly positioned tags are the underlying issue, i.e. a misplaced <section>. In this case, experience suggests that the problem may be below the indicated problem in the returned error.


=== Building The Manual ===
=== Building The Manual ===
Docbook manuals can be transformed into a number of formats automatically. Two formats of interest are HTML and PDF. HTML transforms can be made directly from the Docbook xml (with '''xsltproc'''). PDF transforms require transforming first into an intermediatory fop xml format, and then into a PDF with a tool called '''fop'''. The easiest way to generate the output files is via the make targets:
Docbook manuals can be transformed into a number of formats automatically. Two formats of interest for testing are HTML and PDF. HTML transforms are made directly from the Docbook xml (with '''xsltproc'''). PDF transforms require transforming into an intermediatory '''fop''' xml format first, and then into a PDF with using '''fop'''. The easiest way to generate the output files is via the make targets:
;To build a HTML version of the manual:
;To build a HTML version of the manual:
<pre style="overflow-x:auto;">
<pre style="overflow-x:auto;">
Line 124: Line 127:
make pdf
make pdf
</pre>
</pre>
Building a xhtml and epub version of the manual is also supported by the makefile.
Building an xhtml and epub version of the manual is also supported by the makefile.


===Debugging===
===Debugging===

Revision as of 09:14, 25 June 2018


UNDER CONSTRUCTION 06/20/2018

Lustre includes an extensive manual. This is developed much like the source code: anyone can submit patches, review changes, and suggest improvements. The manual workflow includes Gerrit and continuous integration practice. This enables changes to be automatically verified and conveniently reviewed using Gerrit.

Requesting a Change or Improvement

Manual corrections and enhancements are tracked in the same manor as code changes. A Jira project exists, LUDOC, where changes are created and tracked. LUDOC tickets can be created, updated or commented by anyone with an account on JIRA. Once you have an account, go to JIRA and create your LUDOC ticket:

To browse all existing issues, please visit: https://jira.whamcloud.com/browse/LUDOC/

Minor changes

Minor changes and fixes can be committed against the existing LUDOC-11 issue. The intention is to minimize the overhead for quick fixes. For such a change, your commit message may be:

LUDOC-11 quota: spelling corrections
Brief description of the change if it is more extensive.
Signed-off-by: Random User <[email protected]>

Submitting a Change to the Manual Source

Anyone can create changes that modify the current Lustre manual. The manual is written in Docbook XML version 5, with each chapter residing in a separate xml file. Reference material is readily available on the web for Docbook XML syntax.

Development Environment Setup

The Docbook 5 development environment has been tested with Ubuntu 18.04 (previous Ubuntu versions back to 10.0 are known to work), RHEL7 Linux, and Mac OS X. The auto build process occurs on RHEL7. The original development work for the manual was completed using Ubuntu.

Ubuntu 18.04

Install the following packages (last validated 2018-06-20): apt install make git libxml2-utils xsltproc fop docbook5-xml docbook-xsl-ns

RHEL 7

Install the following packages: yum install fop libxml2 libxslt docbook5-style-xsl docbook5-schema docbook5-schemas deja*

Install from EPEL, CentOS, source or Whamcloud toolkit: yum localinstall fop xmlgraphics-commons-1.4-3

fop and xmlgraphics

  • A suitably recent version of fop (>= 1.0) is available for EL7 from Whamcloud toolkit in the 'x86_64' folder.
  • A suitably recent version xmlgraphics-commons to provide dependencies on fop >= 1.0 is a available for EL7 from Whamcloud toolkit in the 'noarch' folder.

Fedora 18

Install the following packages: yum install libxml2 libxslt fop docbook5-style-xsl docbook5-schemas docbook5-schemas

Max OS X

It is possible to use either the MacPorts or Homebrew (Brew) package managers on Mac OS X to install the required 3rd-party DocBook tools, instead of compiling them yourself. Only one of these two package managers should be used at one time.

If using the MacPorts package, install the following packages: sudo port install libxml2 libxslt fop docbook-xml-5.0 docbook-xsl

You may experience problems if you don't use MacPorts. These problems typically can be resolved by editing the Makefile to give the correct path to 'docbookxi.rng'.

If using Brew, install following packages: brew install libxml2 libxslt fop docbook docbook-xsl

Changing the Manual

The manual is controlled using, Jira, git, and Gerrit (details below). The general workflow is:

Clone the manual from the "doc/manual" repository
git clone git://git.whamcloud.com/doc/manual lustre-manual
Make change(s) in a local branch, editing .xml files as needed
git checkout -b b_new_change; vi XXXX.xml

A few notes regarding edits:

  • Vim has syntax highlighting, as does emacs. Limited success has been reported with WYSIWYG XML editors.
  • If using a WYSIWYG editor, run "git diff" to ensure it didn't reformat the whole .xml file and that only the intended text was modified.
  • Text should be wrapped at 80 columns so that it is easier to comment on changes during review. Otherwise, it is difficult to make specific comments on an issue after a whole long paragraph of text that is a single line in the .xml document.
  • Ensure you are familiar with Change Review Guidelines specified in this document so that you understand the writing criteria reviewers will apply to qualify your change.
Check that the document still validates
make check

See Verify the Docbook for more information on validation.

Build a target (i.e. html or pdf) and visually check that the document looks good
make pdf
open lustre_manual.pdf
Commit the change locally
git commit -a -v -s
Push the change for review
make push

Note: 'make push' simply calls git push ssh://review.whamcloud.com:29418/doc/manual HEAD:refs/for/master

Add two inspectors to the manual

One inspector should be someone with a technical background related to the changes and another person should review for style/English.

Verify the Docbook

Docbook 5 is verfied using a relaxNG schema with schematron rules. The manual is built using separate xml files for each chapter, which use XInclude elements. Two complementary tools support relaxNG schemas with XInclude, xmllint and jing. The tools can be used together to identify problems as shown here. The easiest way to verify the changes are valid XML is to use the make check target:

make check
  • This will internally verify with xmllint the ./index.xml of doc/manual.
  • OPTIONAL additional verification with jing the ./index.xml of doc/manual:

Fixing Problems

If make check fails, an error will be reported. Finding the source of the problem can be tricky if incorrectly positioned tags are the underlying issue, i.e. a misplaced <section>. In this case, experience suggests that the problem may be below the indicated problem in the returned error.

Building The Manual

Docbook manuals can be transformed into a number of formats automatically. Two formats of interest for testing are HTML and PDF. HTML transforms are made directly from the Docbook xml (with xsltproc). PDF transforms require transforming into an intermediatory fop xml format first, and then into a PDF with using fop. The easiest way to generate the output files is via the make targets:

To build a HTML version of the manual
make html
Similarly, to build a PDF version of the manual
make pdf

Building an xhtml and epub version of the manual is also supported by the makefile.

Debugging

There are two types of errors that are common:

  1. well formed xml
    'make check' will do a good job of discovering and precisely identifying the location of these types of errors.
  2. valid xml
    'make check' will validate the xml but does a poor job of identifying errors when they occur. The first complication is that xinclude is used to include separate xml files. As a result, xmllint is unable to really spot where the problem is.
Example
$ make check
xmllint --noout --xinclude --noent --relaxng /usr/share/xml/docbook5/schema/rng/5.0/docbookxi.rng ./index.xml
./index.xml:6: element itemizedlist: Relax-NG validity error : Expecting an element listitem, got nothing
./index.xml:6: element itemizedlist: Relax-NG validity error : Expecting element example, got itemizedlist
./index.xml:63: element section: Relax-NG validity error : Element chapter has extra content: section
./index.xml:1: element chapter: Relax-NG validity error : Did not expect element chapter there
./index.xml:1: element chapter: Relax-NG validity error : Expecting element include, got chapter
./index.xml fails to validate

This complication can be avoided by first flattening the manual into a single xml file:

$ xmllint  --xinclude  index.xml  >  /tmp/resolved.xml

Now, you can get a slightly better idea of where the problem is using xmllint – but it's still not that great:

$ xmllint --noout --xinclude --noent --relaxng /usr/share/xml/docbook5/schema/rng/5.0/docbookxi.rng /tmp/resolved.xml
/tmp/resolved.xml:8757: element itemizedlist: Relax-NG validity error : Expecting an element listitem, got nothing
/tmp/resolved.xml:8757: element itemizedlist: Relax-NG validity error : Expecting element example, got itemizedlist
/tmp/resolved.xml:8812: element section: Relax-NG validity error : Element chapter has extra content: section
/tmp/resolved.xml:8752: element chapter: Relax-NG validity error : Did not expect element chapter there
/tmp/resolved.xml:8752: element chapter: Relax-NG validity error : Expecting element include, got chapter
/tmp/resolved.xml fails to validate

However, if you have Jing installed, you can get an even better idea of what the problem is - as well as where it is:

$ jing /usr/share/xml/docbook5/schema/rng/5.0/docbookxi.rng /tmp/resolved.xml 
/tmp/resolved.xml:9127:29: error: element "para" not allowed here; expected the element end-tag, text or element "abbrev", "accel", "acronym", "address", "alt", "anchor", "annotation", "application", "author", "bibliolist", "biblioref", "blockquote", "bridgehead", "calloutlist", "caution", "citation", "citebiblioid", "citerefentry", "citetitle", "classname", "classsynopsis", "cmdsynopsis", "code", "command", "computeroutput", "constant", "constraintdef", "constructorsynopsis", "coref", "database", "date", "destructorsynopsis", "editor", "email", "emphasis", "envar", "epigraph", "equation", "errorcode", "errorname", "errortext", "errortype", "example", "exceptionname", "fieldsynopsis", "figure", "filename", "firstterm", "footnote", "footnoteref", "foreignphrase", "funcsynopsis", "function", "glosslist", "glossterm", "guibutton", "guiicon", "guilabel", "guimenu", "guimenuitem", "guisubmenu", "hardware", "important", "indexterm", "informalequation", "informalexample", "informalfigure", "informaltable", "initializer", "inlineequation", "inlinemediaobject", "interfacename", "itemizedlist", "jobtitle", "keycap", "keycode", "keycombo", "keysym", "link", "literal", "literallayout", "markup", "mediaobject", "menuchoice", "methodname", "methodsynopsis", "modifier", "mousebutton", "msgset", "nonterminal", "note", "olink", "ooclass", "ooexception", "oointerface", "option", "optional", "orderedlist", "org", "orgname", "package", "parameter", "person", "personname", "phrase", "procedure", "productionset", "productname", "productnumber", "programlisting", "programlistingco", "prompt", "property", "qandaset", "quote", "remark", "replaceable", "returnvalue", "revhistory", "screen", "screenco", "screenshot", "segmentedlist", "shortcut", "sidebar", "simplelist", "subscript", "superscript", "symbol", "synopsis", "systemitem", "table", "tag", "task", "termdef", "tip", "token", "trademark", "type", "uri", "userinput", "variablelist", "varname", "warning", "wordasword", "xi:include" or "xref"

Other Tools

Jira

Jira has a project specifically for Manual changes: LUDOC.

Git
  • Clone the manual as follows:
git clone git://git.whamcloud.com/doc/manual lustre-manual
  • Push changes for review:
make push

Internally this does something like:

git push ssh://[email protected]:29418/doc/manual HEAD:refs/for/master
Gerrit

Using Gerrit describes how to submit changes to the manual. Criteria for manual changes to land a change on master is: +2 (+1 from two human reviewers) and +1 verified from Jenkins.

Change Review Guidelines

Like code, Manual changes are submitted to a review process. All the manual patches that are in Gerrit for review can be seen. Logged-in users can review previous versions of the Manual. Updates to the Manual are automatically built from the change and available as a review Manual build, links to these builds are in the comments of a change. Reviewers are advised to read the guidance for Manual reviewers in this section.