KVM Quick Start Guide

= Introduction = Lustre combines a custom in-kernel network stack called Lustre Networking (LNET) along with its own object block devices, which are made up of local file systems, such as ldiskfs or ZFS. Together, these components make up the storage framework used to create a clustered file system.

Like any file on any other file system, a single file is made up of several component parts. The file and directory structure that indexes the file (metadata), and data blocks (objects) that comprise the data within the file. The file system has to keep track of where those objects are located, who and what can access those objects and with what permissions (read, write, and execute). This extra information that describes a user's file is referred to as metadata.

Most conventional file systems combine file data blocks and their metadata on a single, continuous block storage device. The performance and scalability of Lustre comes from a design that aggregates many storage servers through dedicated metadata and object servers to form a single file system that is a superset of its component storage nodes. Clients communicate through LNET protocol to metadata servers and establish direct connections to file data stored on one or more object storage servers. This enables, not only the ability to parallelize a single clients access to file system data across multiple servers, but also to service many clients concurrently across a virtualized file system that goes beyond conventional boundaries of size, capacity, and throughput.

Lustre scales for capacity, redundancy, and concurrency by arranging and adding storage components. The metadata servers keep track of object locations, while facilitating client-to-storage connections and arbitration of access to file system data.

= What are the parts of a Lustre Cluster? = In the descriptions that follow, there is often a symmetry in which each particular component of a Lustre cluster is broken up into a server and one or more targets. Server components communicate to other Lustre components via LNET, while the associated targets function as storage for the service or server. Targets can present themselves as wide range of block storage devices from directly attached disk drives and RAID arrays to other network technologies.

It is important to remember that the server provides access to, and abstracts, the underlying target that contains the data.

Management Server (MGS)
The MGS is the top level entry and access point through which client-visible Lustre operations take place. Because Lustre delegates specific file system storage and management to task-specific networked nodes, the MGS serves as the primary configuration point for all nodes on the file system.

The MGS also services as a facilitator for the imperative recovery (IR) feature.

Metadata Server (MDS)
An MDS communicates the state and location of all stored entities in a Lustre file system. It answers client requests and tracks locations and state of data stored throughout the cluster.

An MDS responds to client requests for metadata information, and tracks and records that information. A Lustre file system can have multiple metadata servers. The number of clients, service requirements of the internal cluster, and performance requirements of the deployment determine the number of MDSs you need to install.

Metadata Target (MDT)
The MDT is the primary block storage device used to store all file system metadata. One or more metadata targets can exist on a Lustre file system. When multiple MDTs are present, they are aggregated by the MDS to support the storage needs to track the metadata for the entire cluster. The bigger the Lustre file system, the larger the total storage space of the combined MDTs will need to be.

Object Storage Server (OSS)
The OSS handles the actual file data I/O request. It is the primary interface into the actual backing storage on each of the storage device nodes that make up a Lustre cluster.

Object Storage Target (OST)
One or more object storage targets make up the physical data storage served up by the OSSs. An OSS may use one or more OSTs to store data objects that make up a single Lustre file in a striped or non-striped configuration across one or multiple OSTs.

Lustre Client
A Lustre client is the end user node of a Lustre file system and a Lustre cluster can support thousands of clients at the same time.

The client uses its LNET connections to the MGS, MDS, and OSSs to enable mounting the composite storage of the cluster as a single POSIX compatible file system on a local mount point.

Internally, the client uses the MGS as the authoritative source of global configuration data including connection information to the MDS and OSSs. The MDS provides clients with file system metadata and object storage locations to and within the OSSs. Each OSS hosts one or more OSTs which in turn, store the objects that make up file data of interest to the end user. =Lustre Requirements = Lustre is only supported on Linux. Because Lustre implementation involves modifications to the Linux kernel, prebuilt packages are only available for a handful of kernels, which are compatible with Red Hat and Ubuntu. Nevertheless, several Linux distributions build their own Lustre packages. Because most feature development takes place on Red Hat compatible distributions, this document focuses on working with a RHEL compatible distribution.

You can group a number of Lustre file system components together on a single compute node, depending on performance, capacity, and redundancy requirements. You can consolidate nodes by combining the MGS and MDS onto a single node. You can also configure an OSS to host one or more of its own OSTs. It is important, however, that you never combine OSS and MDS on a single node because of concurrency issues.

= Virtualization and Operating System Selection = This guide uses KVM virtualization to create a very simple Lustre cluster. It also assumes that you have a working installation of qemu-kvm-1.5, virt-install, and qemu-kvm-tools-1.5 or later running on a host machine running RHEL-compatible Linux. Running a version of Linux that is compatible with RHEL 6 or RHEL 7 provides the best development platform for the purposes of this document. This document uses CentOS 6, a Red Hat Enterprise compatible distribution, to install Lustre.

For the purpose of this document, we will be using CentOS 6, a Red Hat Enterprise compatible distribution to install Lustre on.

The repository for installation is available here:
 * http://isoredirect.centos.org/centos/6/isos/x86_64/

The repository for installation is available here:
 * http://mirror.centos.org/centos/6/os/x86_64/

This document discusses and demonstrates the steps involved in installing Lustre, and provides bash-compatible scripts to use as reference for a semi-automated setup.

= Creating The CentOS 6 KVM Virtual Machines = Installing an operating system involves a bootstrapping process where first an operating system is started from installation media that runs entirely in RAM. It is then used to install the "real" operating system on the target machine. Though there are LOTS of ways to do this, this guide will be doing everything over simple HTTP (web) protocols with no physical media except a floppy disk image we create that will hold the installation instructions.

You need root access on the machine used to host the Lustre virtual machines.


 * HINT: To avoid running as root or constantly providing passwords, add yourself to the group "wheel" and modify /etc/sudoers file to enable passwordless sudo.

sudo usermod YOUR_USER_NAME -G wheel sudo visudo
 * uncomment (remove prefixed hash mark)

# %wheel ALL=(ALL)       NOPASSWD: ALL
 * which becomes:

%wheel ALL=(ALL)       NOPASSWD: ALL

Before proceeding, create a directory to host the VM files. mkdir -p ~/lustre_vms cd ~/lustre_vms

Creating the floppy disk image
This procedure uses a floppy disk image connected to the virtual machine to provide the setup and configuration instructions needed for installation.

Navigate to the directory where you want your virtual machine files to be stored, and then enter the following commands:

mkdir -p floppy.dir chmod 777 floppy.dir qemu-img create -f raw floppy.img 1440k mkfs.msdos -s 1 floppy.img sudo mount -o loop, floppy.img floppy.dir

If everything completes without error, you should have a file, "floppy.img" whose contents are accessible in the directory "floppy.dir".


 * HINT: If qemu-img is not installed, then attempting to create the floppy image generates a error. To install qemu-img, enter the following command:

yum install -y qemu-img

Creating a Kickstart file
The Kickstart file is used by the Anaconda system built into Red Hat Enterprise compatible Linux distributions to install and configure the installation of Lustre's Linux operating system. You can find the full text of the kickstart file in the Scripts and Configuration Files section at the end of this document. Copy the configuration code into a file named ks-base-centos6.cfg in the floppy.dir directory.

The only thing that you must change is the hostname field in the line: *network --hostname=HOSTNAME*

When executed by anaconda, the kickstart does the following:
 * Sets the hostname (Set a unique hostname for each virtual machine you create.)
 * Configures the virtual machine to bridge to the host eth0 network interface.
 * If your primary network connection on the host is not through eth0, then you must modify this line with the correct interface.
 * Configures the virtual machine’s network interface to use DHCP.
 * If you do not have a DHCP server, this line should be modified to use a static IP configuration.
 * Sets the time zone to Denver Colorado
 * You can find other valid locations in /usr/share/zoneinfo.
 * Boot straps the install from the CentOS mirror repository for version 6.6
 * Disables SELINUX (Secure Linux Features).
 * Lustre on SELINUX is not entirely straight forward to configure.
 * Creates a user named '"dev"', makes the user part of the wheel group.
 * Modifies the /etc/sudoers file to allow passwordless sudo to root.
 * Sets the password for both the 'dev' and 'root' users to "seagate"
 * Performs a minimal install of CentOS 6.6 while adding the packages needed to develop and debug Lustre.
 * Creates a new repository entry for YUM that takes precedence over all others, from which to install Lustre.
 * Creates a several scripts in the root user's home directory to install and configure Lustre server and client nodes.

Installing CentOS 6 for Lustre under KVM
After completing the previous step, a directory called floppy.dir exists and contains the ks-base-centos6.cfg configuration file. The floppy.dir directory is also the mount point for the floppy.img disk image.

The following procedure creates three virtual machines:
 * A client
 * A combination MGS, MDS, and MDT
 * An OSS with a single OST

Before configuring the virtualization settings and initiating the kickstart-based installer, you must change both the name of the VM in the virt-install command and the hostname in the kickstart file for each virtual machine. There are three named VMs, each named as follows:
 * lustre_mgs_mdt
 * lustre_oss1
 * lustre_client

It is highly recommended that. for each named VM, you do the following:
 * 1) Edit the kickstart file (ks-base-centos6.cfg) in the floppy.dir directory to set the hostname line.
 * 1) Export the virtual machine to the shell using the following command:

export VM_NAME=(name of VM)

- edit the kickstart file (ks-base-centos6.cfg) in floppy.dir/ and set the hostname line: * network --hostname=(_name of VM_) - do the virt-install command after exporting VM_NAME=(_name of VM_)

export VM_NAME=(_name of VM_)

Immediately after the previous (long) command, enter the following to observe and monitor the boot and installation:

Troubleshooting: * If you get an error about virt-install not being a valid command: - sudo yum install -y virt-install * Likewise, if "virsh" is not found: - sudo yum install -y libvirt libvirt-client libvirt-daemon libvirt-docs

With a bit of luck, you should now have a running virtual machine that is installing itself completely from the web. At the end of the install, it will reboot, but the reboot might fail to restart. You can manually start the virtual machine and re-attach the serial console with the following command:

Installing Lustre Server Components via helper scripts (the easy way)
If you've followed the guide up to this point, you should have virtual machine based on CentOS 6 configured and setup via the kickstart file above. The kickstart file should have created a number of files in /root: - install_lustre_client_from_wham_cloud - install_lustre_server_from_whamcloud - configure_lustre_client - configure_lustre_mgs_mdt_mds - configure_lustre_oss Connect to each of the running vm's and run the install appropriate install scripts: - sudo virsh console lustre_client1 (log on as root/seagate) * ./install_lustre_client_from_whamcloud * reboot - sudo virsh console lustre_mgs_mds (log on as root/seagate) * ./install_lustre_server_from_whamcloud * reboot - sudo virsh console lustre_oss1 (log on as root/seagate) * ./install_lustre_server_from_whamcloud * reboot Connect to each of the running vm's and configure their roles. The order is important! Configure the mgs/mds first, the OSS next, and finally a client: - sudo virsh console lustre_mgs_mds (log on as root/seagate) * ./configure_lustre_mgs_mdt_mds - sudo virsh console lustre_oss1 (log on as root/seagate) * ./configure_lustre_oss - sudo virsh console lustre_client1 * ./configure_lustre_client

Installing Lustre Server Components without helper scripts (the hard way)
The following details how to manually install Lustre without the script or having used the kickstart file.

First, create the yum repo file in:

Populate it with the following information:

Make sure yum-plugin-priorities is installed:

Manually install the Lustre components:

Configure lnet and enable Lustre:

Reboot the machine

You should now have a virtual machine that is ready to be configured as a Lustre server component (MGS, MDT, OSS).

= Configuring virtual Lustre Cluster Machines = You need a minimum of three virtual machines. This means you'll have to repeat the "Creating a CentOS 6 KVM virtual machine from scratch" section once for each machine. Make sure each machine is given a unique hostname!

It is highly recommended that after creating the initial floppy disk image and kickstart files, that the virsh-install command be run once for each VM to create, modifying the hostname in the kickstart file each time before running virt install.

Configure as MGS, MDT, and MDS
We'll be combining the all three of the first three Lustre server functions into the first virtual machine node. Start the vitual machine up and attach to either its serial console

To start a VM with serial console attached:

Alternatively, ssh into the VM (kickstart sets all passwords to 'seagate')

Configure mount options
= Scripts and Config Files =

Create CentOS 6 VM script
This script, when run side-by-side with the ks-base-centos.cfg kickstart file from above, can create a generic virtual machine ready to be configured either as a Lustre Client or Server node. For each kind of VM, run once with the name of the VM as the first parameter # ./create.centos6 lustre_client1 # ./create.centos6 lustre_mgs_mds # ./create.centos6 lustre_oss1 Once each VM is created, configure them for their roles.