Lustre with ZFS Install: Difference between revisions
Jump to navigation
Jump to search
m (→Build Lustre with ZFS (state 2017-04-12): use standard date forma) |
Kylehutson (talk | contribs) (Updated Lustre git repository) |
||
Line 61: | Line 61: | ||
<ol> | <ol> | ||
<li>Get Lustre source code </li> | <li>Get Lustre source code </li> | ||
<pre>git clone git://git. | <pre>git clone git://git.whamcloud.com/fs/lustre-release.git</pre> | ||
<li>Configure (<tt>--disable-ldiskfs</tt> for ZFS backend, <tt>--disable-server</tt> for client only) </li> | <li>Configure (<tt>--disable-ldiskfs</tt> for ZFS backend, <tt>--disable-server</tt> for client only) </li> | ||
<pre>cd lustre-release/ | <pre>cd lustre-release/ |
Latest revision as of 09:12, 20 September 2018
Introduction
This page is an attempt to provide some information on how to install Lustre with a ZFS backend. You are encouraged to add your own version, either as a separate section or by editing this page into a general guide.
Build Lustre with ZFS (state as of 2017-12-04)
The following guides are valid for CentOS 7.
Build Lustre
HowTo build a one-node Lustre file system with ZFS backend:
- Prepare System
- Disable SELinux for older clients
- Install the kernel development tools
- Install additional dependencies
- Prepare ZFS backend (follow the guide for packaged ZFS or go to this section for custom ZFS build)
- EPEL release
- For the newest Lustre releases change /etc/yum.repos.d/zfs.repo to switch from dkms to kmod (more info [1] and [2])
- Install ZFS and its associated SPL packages
- kmod packages for newer releases
- dkms packages for dkms support
- Build Lustre
- Get Lustre source code
- Configure (--disable-ldiskfs for ZFS backend, --disable-server for client only)
- Make and install rpms
- You may need to reboot and to explicitly load the ZFS and Lustre modules
- Format targets (change /tmp in this example to real devices or partitions and you may not need the device size then)
- Change /etc/ldev.conf
- Run Lustre
- Reconfigure the firewall to allow incoming connections on TCP port 988 (for socklnd only), or temporarily disable it and fix up later
- Start servers
- Mount client
sed -i '/^SELINUX=/s/.*/SELINUX=disabled/' /etc/selinux/config
yum -y groupinstall 'Development Tools' yum -y install epel-release
yum -y install xmlto asciidoc elfutils-libelf-devel zlib-devel kernel-devel libyaml-devel yum -y install binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed yum -y install bison elfutils-devel audit-libs-devel python-docutils sg3_utils expect yum -y install attr lsof quilt libselinux-devel
URL='http://download.zfsonlinux.org' yum -y install --nogpgcheck $URL/epel/zfs-release.el7.noarch.rpm
URL='http://download.zfsonlinux.org' [zfs] name=ZFS on Linux for EL 7 - dkms baseurl=http://download.zfsonlinux.org/epel/7/$basearch/ -enabled=1 +enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux @@ -9,7 +9,7 @@ [zfs-kmod] name=ZFS on Linux for EL 7 - kmod baseurl=http://download.zfsonlinux.org/epel/7/kmod/$basearch/ -enabled=0 +enabled=1 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
yum install -y zfs libzfs2-devel kmod-spl-devel kmod-zfs-devel
yum install -y zfs libzfs2-devel zfs-dkms
git clone git://git.whamcloud.com/fs/lustre-release.git
cd lustre-release/ sh ./autogen.sh ./configure --disable-ldiskfs
make rpms yum -y install *.$(arch).rpm
reboot modprobe zfs modprobe lustre
mkfs.lustre --mgs --backfstype=zfs --fsname=lustre --device-size=1048576 lustre-mgs/mgs /tmp/lustre-mgs mkfs.lustre --mdt --backfstype=zfs --fsname=lustre --index=0 --mgsnode=$(hostname)@tcp --device-size=1048576 lustre-mdt0/mdt0 /tmp/lustre-mdt0 mkfs.lustre --ost --backfstype=zfs --fsname=lustre --index=0 --mgsnode=$(hostname)@tcp --device-size=1048576 lustre-ost0/ost0 /tmp/lustre-ost0
hostname - mgs zfs:lustre-mgs/mgs hostname - mdt0 zfs:lustre-mdt0/mdt0 hostname - ost0 zfs:lustre-ost0/ost0
systemctl stop firewalld systemctl disable firewalld
systemctl start lustre
mkdir /mnt/lustre/client mount -t lustre $(hostname):/lustre /mnt/lustre/client
Build ZFS
HowTo build a custom ZFS:
- Prepare System
- Disable SELinux
- Install the kernel development tools
- Install additional dependencies
- Clone Git-repositories
- Perform all the following steps for both directories (complete spl first)
- Configure for specific system
- Build RPMs in both directories
- kmod
- dkms
- Install RPMs
sed -i '/^SELINUX=/s/.*/SELINUX=disabled/' /etc/selinux/config
yum -y groupinstall 'Development Tools' yum -y install epel-release
yum -y install parted lsscsi wget ksh yum -y install kernel-devel zlib-devel libattr-devel yum -y install libuuid-devel libblkid-devel libselinux-devel libudev-devel yum -y install device-mapper-devel openssl-devel
git clone https://github.com/zfsonlinux/spl.git git clone https://github.com/zfsonlinux/zfs.git
cd <spl|zfs> ./autogen.sh ./configure --with-spec=redhat
make pkg-utils pkg-kmod
make pkg-utils rpm-dkms
yum localinstall *.$(arch).rpm
SSEC Example
Helpful links
- http://zfsonlinux.org/lustre-configure-single.html
- https://github.com/chaos/lustre/commit/04a38ba7 - ZFS and HA
This version applies to systems with JBODs where ZFS manages the disk directly without a Dell Raid Controller in between. This guide is very specific for a single installation at UW SSEC: versions have changed, and we use puppet to provide various software packages and configurations. However, it is included as some information may be useful to others.
- Lustre Server Prep Work
- OS Installation (RHEL6)
- You must use the RHEL/Centos 6.4 Kernel 2.6.32-358
- Use the "lustre" kickstart option which installs a 6.4 kernel
- Define the host in puppet so that it is not a default host - NOTE: We Use Puppet at SSEC to distribute various required packages, other environments will vary!
- Lustre 2.4 installation
- Puppet Modules Needed
- OS Installation (RHEL6)
- zfs-repo
- lustre-healthcheck
- ib-mellanox
- check_mk_agent-ssec
- puppetConfigFile
- lustre-shutdown
- nagios_plugins
- lustre24-server-zfs
- selinux-disable
- Configure Metadata Controller
- Map metadata drives to enclosures (with scripts to help)
- For our example mds system we made aliases for 'ssd0' ssd1 ssd2 and ssd3
- put these in /etc/zfs/vdev_id.conf - for example:
- alias arch03e07s6 /dev/disk/by-path/pci-0000:04:00.0-sas-0x5000c50056b69199-lun-0
- run udevadm trigger to load drive aliases
- On metadata controller, run mkfs.lustre to create metadata partition. On our example system:
- Use separate MGS for multiple filesystems on same metadata server.
- Separate MGS: mkfs.lustre --mgs --backfstype=zfs lustre-meta/mgs mirror d2 d3 mirror d4 d5
- Separate MDT: mkfs.lustre --fsname=arcdata1 --mdt --mgsnode=172.16.23.14@o2ib --backfstype=zfs lustre-meta/arcdata1-meta
- Create /etc/ldev.conf and add the metadata partition. On example system, we added:
- geoarc-2-15 - MGS zfs:lustre-meta/mgs geoarc-2-15 - arcdata-MDT0000 zfs:lustre-meta/arcdata-meta
- Create /etc/modprobe.d/lustre.conf
- options lnet networks="o2ib" routes="tcp metadataip@o2ib0 172.16.24.[220-229]@o2ib0"
- NOTE: if you do not want routing, or if you are having trouble with setup, the simple options lnet networks="o2ib" is fine
- Start Lustre. If you have multiple metadata mounts, you can just run service lustre start.
- Add lnet service to chkconfig and ensure on startup. We may want to leave lustre off on startup for metadata controllers.
- Configure OSTs
- Map drives to enclosures (with scripts to help!)
- Run udevadm trigger to load drive aliases.
- mkfs.lustre on MD1200s.
- Example RAIDZ2 on one MD1200: mkfs.lustre --fsname=cove --ost --backfstype=zfs --index=0 --mgsnode=172.16.24.12@o2ib lustre-ost0/ost0 raidz2 e17s0 e17s1 e17s2 e17s3 e17s4 e17s5 e17s6 e17s7 e17s8 e17s9 e17s10 e17s11
- Example RAIDZ2 with 2 disks from each enclosure, 5 enclosures (our cove test example): mkfs.lustre --fsname=cove --ost --backfstype=zfs --index=0 --mgsnode=172.16.24.12@o2ib lustre-ost0/ost0 raidz2 e13s0 e13s1 e15s0 e15s1 e17s0 e17s1 e19s0 e19s1 e21s0 e21s1
- Repeat as necessary for additional enclosures.
- Create /etc/ldev.conf
- Example on lustre2-8-11:
- lustre2-8-11 - cove-OST0000 zfs:lustre-ost0/ost0 lustre2-8-11 - cove-OST0001 zfs:lustre-ost1/ost1 lustre2-8-11 - cove-OST0002 zfs:lustre-ost2/ost2
- Start OSTs. Example: service lustre start. Repeat as necessary for additional enclosures.
- Add services to chkconfig and setup.
- Configure backup metadata controller (future)
- Mount the Lustre file system on clients
- Add entry to /etc/fstab. With our example system, our fstab entry is:
- 172.16.24.12@o2ib:/cove /cove lustre defaults,_netdev,user_xattr 0 0
- Create empty folder for mountpoint, and mount file system (e.g., mkdir /cove; mount /cove).