KVM Quick Start Guide: Difference between revisions

From Lustre Wiki
Jump to navigation Jump to search
(Created page with "%title Basic Installation and Demonstration of a Virtual Lustre Cluster <br> %toc Basic Installation and Demonstration of a Virtual Lustre Cluster <br> = Introduction = Lust...")
 
(Basic Installation and Demonstration of a Virtual Lustre Cluster)
Line 1: Line 1:
%title Basic Installation and Demonstration of a Virtual Lustre Cluster
<br>
%toc Basic Installation and Demonstration of a Virtual Lustre Cluster
<br>
= Introduction =
= Introduction =
Lustre combines a custom in-kernel network stack called Lustre Networking (LNET) along with its own object block devices, which are made up of local file systems, such as ldiskfs or ZFS. Together, these components make up the storage framework used to create a clustered file system.
Lustre combines a custom in-kernel network stack called Lustre Networking (LNET) along with its own object block devices, which are made up of local file systems, such as ldiskfs or ZFS. Together, these components make up the storage framework used to create a clustered file system.

Revision as of 11:34, 23 June 2015

Introduction

Lustre combines a custom in-kernel network stack called Lustre Networking (LNET) along with its own object block devices, which are made up of local file systems, such as ldiskfs or ZFS. Together, these components make up the storage framework used to create a clustered file system.

Like any file on any other file system, a single file is made up of several component parts. The file and directory structure that indexes the file (metadata), and data blocks (objects) that comprise the data within the file. The file system has to keep track of where those objects are located, who and what can access those objects and with what permissions (read, write, and execute). This extra information that describes a user's file is referred to as metadata.

Most conventional file systems combine file data blocks and their metadata on a single, continuous block storage device. The performance and scalability of Lustre comes from a design that aggregates many storage servers through dedicated metadata and object servers to form a single file system that is a superset of its component storage nodes. Clients communicate through LNET protocol to metadata servers and establish direct connections to file data stored on one or more object storage servers. This enables, not only the ability to parallelize a single clients access to file system data across multiple servers, but also to service many clients concurrently across a virtualized file system that goes beyond conventional boundaries of size, capacity, and throughput.

Lustre scales for capacity, redundancy, and concurrency by arranging and adding storage components. The metadata servers keep track of object locations, while facilitating client-to-storage connections and arbitration of access to file system data.

What are the parts of a Lustre Cluster?

In the descriptions that follow, there is often a symmetry in which each particular component of a Lustre cluster is broken up into a server and one or more targets. Server components communicate to other Lustre components via LNET, while the associated targets function as storage for the service or server. Targets can present themselves as wide range of block storage devices from directly attached disk drives and RAID arrays to other network technologies.

It is important to remember that the server provides access to, and abstracts, the underlying target that contains the data.

Management Server (MGS)

The MGS is the top level entry and access point through which client-visible Lustre operations take place. Because Lustre delegates specific file system storage and management to task-specific networked nodes, the MGS serves as the primary configuration point for all nodes on the file system.

The MGS also services as a facilitator for the imperative recovery (IR) feature.

Metadata Server (MDS)

An MDS communicates the state and location of all stored entities in a Lustre file system. It answers client requests and tracks locations and state of data stored throughout the cluster.
An MDS responds to client requests for metadata information, and tracks and records that information. A Lustre file system can have multiple metadata servers. The number of clients, service requirements of the internal cluster, and performance requirements of the deployment determine the number of MDSs you need to install.

Metadata Target (MDT)

The MDT is the primary block storage device used to store all file system metadata. One or more metadata targets can exist on a Lustre file system. When multiple MDTs are present, they are aggregated by the MDS to support the storage needs to track the metadata for the entire cluster. The bigger the Lustre file system, the larger the total storage space of the combined MDTs will need to be.

Object Storage Server (OSS)

The OSS handles the actual file data I/O request. It is the primary interface into the actual backing storage on each of the storage device nodes that make up a Lustre cluster.

Object Storage Target (OST)

One or more object storage targets make up the physical data storage served up by the OSSs. An OSS may use one or more OSTs to store data objects that make up a single Lustre file in a striped or non-striped configuration across one or multiple OSTs.

Lustre Client

A Lustre client is the end user node of a Lustre file system and a Lustre cluster can support thousands of clients at the same time.

The client uses its LNET connections to the MGS, MDS, and OSSs to enable mounting the composite storage of the cluster as a single POSIX compatible file system on a local mount point.

Internally, the client uses the MGS as the authoritative source of global configuration data including connection information to the MDS and OSSs. The MDS provides clients with file system metadata and object storage locations to and within the OSSs. Each OSS hosts one or more OSTs which in turn, store the objects that make up file data of interest to the end user.

Lustre Requirements

Lustre is only supported on Linux. Because Lustre implementation involves modifications to the Linux kernel, prebuilt packages are only available for a handful of kernels, which are compatible with Red Hat and Ubuntu. Nevertheless, several Linux distributions build their own Lustre packages. Because most feature development takes place on Red Hat compatible distributions, this document focuses on working with a RHEL compatible distribution.

You can group a number of Lustre file system components together on a single compute node, depending on performance, capacity, and redundancy requirements. You can consolidate nodes by combining the MGS and MDS onto a single node. You can also configure an OSS to host one or more of its own OSTs. It is important, however, that you never combine OSS and MDS on a single node because of concurrency issues.

Virtualization and Operating System Selection

This guide uses KVM virtualization to create a very simple Lustre cluster. It also assumes that you have a working installation of qemu-kvm-1.5, virt-install, and qemu-kvm-tools-1.5 or later running on a host machine running RHEL-compatible Linux. Running a version of Linux that is compatible with RHEL 6 or RHEL 7 provides the best development platform for the purposes of this document. This document uses CentOS 6, a Red Hat Enterprise compatible distribution, to install Lustre.

For the purpose of this document, we will be using CentOS 6, a Red Hat Enterprise compatible distribution to install Lustre on.

The repository for installation is available here: {{{ http://isoredirect.centos.org/centos/6/isos/x86_64/ }}} The repository for installation is available here: {{{ http://mirror.centos.org/centos/6/os/x86_64/ }}} This document discusses and demonstrates the steps involved in installing Lustre, and provides bash-compatible scripts to use as reference for a semi-automated setup.

Creating The CentOS 6 KVM Virtual Machines

Installing an operating system involves a bootstrapping process where first an operating system is started from installation media that runs entirely in RAM. It is then used to install the "real" operating system on the target machine. Though there are LOTS of ways to do this, this guide will be doing everything over simple HTTP (web) protocols with no physical media except a floppy disk image we create that will hold the installation instructions.

You will need "root" access on the machine used to host the Lustre virtual machines.

_HINT: To avoid running as root or constantly providing passwords, add yourself to the group "wheel" and modify /etc/sudoers file to enable passwordless sudo._ {{{ sudo usermod YOUR_USER_NAME -G wheel sudo visudo uncomment (remove prefixed hash mark) # %wheel ALL=(ALL) NOPASSWD: ALL becomes %wheel ALL=(ALL) NOPASSWD: ALL }}}
Before proceeding, you'll probably want to create a directory where all the VM files will created and hosted.


{{{ mkdir -p ~/lustre_vms cd ~/lustre_vms }}}

Creating the floppy disk image

We'll be using a floppy disk image connected to the virtual machine to provide the setup and configuration instructions needed for installation.

Go to the directory where your virtual machine files will be stored. Enter the following commands:

{{{ mkdir -p floppy.dir chmod 777 floppy.dir qemu-img create -f raw floppy.img 1440k mkfs.msdos -s 1 floppy.img sudo mount -o loop, floppy.img floppy.dir }}}

If everything completes without error, you should have a file, "floppy.img" whose contents are accessible in the directory "floppy.dir".

HINT: If you get an error running qemu-img, you might not have it installed yet. run: {{{ yum install -y qemu-img }}}

Creating a Kickstart file

The Kickstart file is used by the Anaconda system built into Red Hat Enterprise compatible Linux distributions to install and configure the installation of Lustre's Linux operating system. You can find the full text of the kickstart file in the Scripts and Configuration Files section at the end of this document. Copy the configuration code into a file named _ks-base-centos6.cfg_ in the floppy.dir directory.

The only thing that you *should* change is the hostname field in the line:

*network --hostname=HOSTNAME*

When executed, the kickstart does the following: * Sets the hostname (Set a unique hostname for each virtual machine you create.) * Configures the virtual machine to bridge to the host eth0 network interface. - If you're primary network connection on the host is *not* through eth0, please modify this line with the correct network interface name. * Configures the virtual machine’s network interface to use DHCP. -If you do not have a DHCP server, this line should be modified. * Sets the time zone to Denver Colorado. - You can find other valid locations in /usr/share/zoneinfo. * Boot straps the install from the CentOS mirror repo for version 6.6 * Disables SELINUX (Secure Linux Features). - Lustre on SELINUX is difficult. * Creates a user named "dev", makes the user part of the wheel group. * Modifies the /etc/sudoers file to allow passwordless sudo to root. * *Sets the password for both the "dev" and "root" users to "seagate"* * Does a minimal install of CentOS 6.6 while adding the packages needed to develop and debug Lustre. * Creates a new repository entry for YUM that takes precedence over all other to install Lustre from. * Creates a script in the root user's home directory "install_lustre_from_whamcloud" that will install the latest release packages needed to create Lustre server nodes"

Installing CentOS 6 for Lustre under KVM

At this point we should have a file called ks-base-centos6.cfg which is the full text of the kickstart file from the previous section in the directory floppy.dir which itself a loop back mount of the file floppy.img.

We will be creating 3 virtual machines - a client, a mgs/mds/mdt, and an OSS with a single OST. The command below will take care of all the virtualization settings and initiate the kickstart based install but you *MUST* change both the name for the VM in the virt-install command AND the hostname in the kickstart file for each virtual machine. The three named VM's will be: - lustre_mgs_mds - lustre_oss1 - lustre_client1

It is highly recommended that for each named VM, you - edit the kickstart file (ks-base-centos6.cfg) in floppy.dir/ and set the hostname line: * network --hostname=(_name of VM_) - do the virt-install command after exporting VM_NAME=(_name of VM_)

export VM_NAME=(_name of VM_) {{{ sudo virt-install \ --connect qemu:///system \ --name "$VM_NAME" \ --virt-type=kvm \ --memory 2048 \ --vcpus=1 \ --disk device=floppy,path="$PWD/floppy.img" \ --disk device=disk,path="$PWD/$VM_NAME-sys.raw",size=15,format=raw \ --disk device=disk,path="$PWD/$VM_NAME-data.raw",size=1,format=raw \ --os-variant rhel6.6 \ --location http://mirror.centos.org/centos/6.6/os/x86_64/ \ --noautoconsole \ --graphics vnc,listen=0.0.0.0 \ --accelerate \ --network=bridge:br0 \ --extra-args=="console=tty0 console=ttyS0,115200 ksdevice=eth0 ks=hd:fd0:/ks.cfg" \ --hvm }}} Immediately after the previous (long) command, enter the following to observe and monitor the boot and installation: {{{ sudo virsh console (name of VM) }}}

Troubleshooting: * If you get an error about virt-install not being a valid command: - sudo yum install -y virt-install * Likewise, if "virsh" is not found: - sudo yum install -y libvirt libvirt-client libvirt-daemon libvirt-docs

With a bit of luck, you should now have a running virtual machine that is installing itself completely from the web. At the end of the install, it will reboot, but the reboot might fail to restart. You can manually start the virtual machine and re-attach the serial console with the following command: {{{ sudo virt start (name of VM) --console }}}

Installing Lustre Server Components via helper scripts (the easy way)

If you've followed the guide up to this point, you should have virtual machine based on CentOS 6 configured and setup via the kickstart file above. The kickstart file should have created a number of files in /root: - install_lustre_client_from_wham_cloud - install_lustre_server_from_whamcloud - configure_lustre_client - configure_lustre_mgs_mdt_mds - configure_lustre_oss Connect to each of the running vm's and run the install appropriate install scripts: - sudo virsh console lustre_client1 (log on as root/seagate) * ./install_lustre_client_from_whamcloud * reboot - sudo virsh console lustre_mgs_mds (log on as root/seagate) * ./install_lustre_server_from_whamcloud * reboot - sudo virsh console lustre_oss1 (log on as root/seagate) * ./install_lustre_server_from_whamcloud * reboot Connect to each of the running vm's and configure their roles. The order is important! Configure the mgs/mds first, the OSS next, and finally a client: - sudo virsh console lustre_mgs_mds (log on as root/seagate) * ./configure_lustre_mgs_mdt_mds - sudo virsh console lustre_oss1 (log on as root/seagate) * ./configure_lustre_oss - sudo virsh console lustre_client1 * ./configure_lustre_client

Installing Lustre Server Components without helper scripts (the hard way)

The following details how to manually install Lustre without the script or having used the kickstart file.

First, create the yum repo file in: {{{ /etc/yum.repos.d/whamcloud.repo }}} Populate it with the following information: {{{ [lustre_latest_el6_client] name=whamcloud_lustre_client baseurl=https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/client/ enabled=1 priority=5 gpgcheck=0

[lustre_latest_el6_server] name=whamcloud_lustre_server baseurl=https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/ enabled=1 priority=5 gpgcheck=0

[e2fsprogs_latest] name=whamcloud_e2fsprogs baseurl=https://downloads.hpdd.intel.com/public/e2fsprogs/latest/el6/RPMS/ enabled=1 priority=5 gpgcheck=0 }}} Make sure yum-plugin-priorities is installed: {{{ yum install yum-plugin-priorities }}} Manually install the Lustre components: {{{ rpm -e --nodeps kernel-firmware yum install -y \ kernel-*lustre \ kernel-firmware*lustre \ lustre-modules \ libss \ libcom_err \ e2fsprogs \ e2fsprogs-libs \ lustre-osd-ldiskfs \ lustre-osd-ldiskfs-mount \ lustre }}} Configure lnet and enable Lustre: {{{ echo "options lnet networks=tcp0(eth0)">/etc/modprobe.d/lnet.conf chkconfig lnet --add chkconfig lnet on chkconfig lustre --add chkconfig lustre on }}} Reboot the machine {{{ shutdown -r now }}} You should now have a virtual machine that is ready to be configured as a Lustre server component (MGS, MDT, OSS).

Configuring virtual Lustre Cluster Machines

You need a minimum of three virtual machines. This means you'll have to repeat the "Creating a CentOS 6 KVM virtual machine from scratch" section once for each machine. Make sure each machine is given a unique hostname!

It is highly recommended that after creating the initial floppy disk image and kickstart files, that the virsh-install command be run once for each VM to create, modifying the hostname in the kickstart file each time before running virt install.

Configure as MGS, MDT, and MDS

We'll be combining the all three of the first three Lustre server functions into the first virtual machine node. Start the vitual machine up and attach to either its serial console

To start a VM with serial console attached: {{{ sudo virsh start VM_NAME --console }}} Alternatively, ssh into the VM (kickstart sets all passwords to 'seagate') {{{ ssh root@VM_NAME }}}

Configure storage

{{{ mkfs.lustre --fsname=lustre1 --mgs --mdt --index=0 /dev/vdb }}}

Configure OSS

Configure Storage

{{{ mkfs.lustre --mgsnode=lustre_mgs_mdt --fsname=lustre1 --ost --index=0 /dev/vdb }}}

Configure mount options

{{{ /dev/vdb /ossost lustre defaults,_netdev 0 0 }}}

Scripts and Config Files

ks-base-centos6.cfg

Create CentOS 6 VM script

This script, when run side-by-side with the ks-base-centos.cfg kickstart file from above, can create a generic virtual machine ready to be configured either as a Lustre Client or Server node.
For each kind of VM, run once with the name of the VM as the first parameter # ./create.centos6 lustre_client1 # ./create.centos6 lustre_mgs_mds # ./create.centos6 lustre_oss1
Once each VM is created, configure them for their roles.

grep "$1" >/dev/null

[ $? -eq 0 ] && echo 1 && return 1 echo 0 && return 0 }

  1. Remove any existing floppy image artifacts.

clean_up_floppy_image() { if [ -d "$FLOPPY_DISK_IMAGE_DIR" ] then if [ $(is_mounted "$FLOPPY_DISK_IMAGE_DIR") -eq 1 ] then echo Unmounting $FLOPPY_DISK_IMAGE_DIR sudo umount "$FLOPPY_DISK_IMAGE_DIR"