Simple Gerrit Builder Howto

Simple build testing framework for gerit patches
To ensure Lustre is properly building and working on other architectures, we need to run some tests on every commit to get early warnings for patches that break those architectures/distributions before the patches get committed to the tree.

This page explains how to set up a simple build bot for the desired environment. Extending it to perform additional verifications is left as an exercise for the reader. Step-by-step guide

{    "review.whamcloud.com": { "gerrit/http": { "username": "USERNAME", "password": "PASSWORD" }    } }
 * 1) Download gerrit_buildpatch.py and run_build.sh
 * 2) Install python and python-requests module
 * 3) choose a working directory for the framework, this is the directory where the framework will hold its data
 * 4) Go to the HPDD gerrit and either create a new account or log into an existing one you'd like to use.
 * 5) Go to the gerrit account settings and switch to "HTTP Password" tab, select a username and generate a new password.
 * 6) In the gerrit working directory create file named GERRIT_AUTH with this content (replacing USERNAME and PASSWORD with username and password selected:
 * 1) Check out gerrit Lustre tree off HPDD git
 * 2) Create kernel against which you want to build Lustre
 * 3) apply necessary patches to the kernel, configure and build it
 * 4) Try to build lustre against the just built kernel to ensure that it actually works and all necessary -devel packages are installed
 * 5) Edit run_build.sh to update path to your built kernel tree and to where you checked out Lustre source.
 * 6) Edit gerrit_buildpatch.py and update at least CHECKPATCH_PATHS to point at run_build.sh and BUILDER_TYPE to explain what it is your builder is testing
 * 7) in the gerrit working directory create REVIEW_HISTORY file with echo "0 - - 0" >REVIEW_HISTORY (This will start at the last 500 patches or so, replace leadign 0 with unix time of a change you want to start with, use current unix time in seconds to start from that point - 1 hour, e.g. 1437884668)
 * 8) Now run gerrit_buildpatch.py to make sure all is working as intended. Watch /tmp/builder_out.txt for build progress and gerrit_buildpatch.py output for overall progress.
 * 9) Stop gerrit_buildpatch.py
 * 10) Edit gerrit_buildpatch.py and change self.post_enabled setting from False to True - this will enable posting of the build results
 * 11) Update REVIEW_HISTORY - add a line referencing more or less current time as the last line to avoid building stale changes for no good reason.
 * 12) Add gerrit_buildpatch.py to your startup scripts or otherwise make it run automatically (mind the current working dir that always should be where the auth and history files are)
 * 13) Once you are really-really sure results are working good and you want to make your testing results "binding" you might also update USE_CODE_REVIEW_SCORE in gerrit_buildpatch.py to True and then build failure would set a real -1 checkmark.

Now some explanations of the constraints.
run_build.sh depends on the prebuilt kernel source tree (on my old PowerMac kernel build takes over 1 hours, so it's impractical to build the kernel for every patch). The outputs are error code and stdout. If errro code is 0, build is considered a success and output is ignored. If the code is not 0, then output would be appended to the gerrit commet by the framework. Currently output last 20 lines preceding the failure, but in fact I guess you can copy the whole build output to some web server and print out a link to that or do something similar. Currently only very minimal Lustre build is performed but you can extend it as much as you want including a full rpm build if desired.

It's possible to extend run_build.sh to also kick some basic testing in a VM or otherwise (e.g. nfs-root hosted VM that would run a subset of sanity.sh with one or several nodes right out of the build tree) and include the results either into the same run (simple) or have some sort of a dispatcher implemented that would run those tests in parallel and would post test results as they complete (more involved, you'll need to write a new posting module for this too).

Finally run_build.sh has some extra locking in place that would allow you to pause builds while you perform some maintenance without stopping the gerrit_buildpatch script. A simple maintenance script is provided as tree_update.sh that you would run from cron once a day to update git repo. You might want to extend it to e.g. update the kernel you are using with fresh patches every day, update kernel and other system components versions (zfs?) and so on.