MDT Mirroring with ZFS and SRP - Revision history

Adilger: link to local copy of presentation

2016-12-15T00:41:21Z

link to local copy of presentation

← Older revision		Revision as of 17:41, 14 December 2016
Line 1:		Line 1:
	Original information provided by Jesse Stroik, January 2014.		Original information provided by Jesse Stroik, January 2014.

	At SSEC we tested mirrored meta data targets for our Lustre file systems. The idea is to use ZFS to mirror storage targets in 2 different MDS - so the data is always available on both servers without using iscsi or other technologies. The basis of this idea comes from Charles Taylor's LUG 2012 presentation "[~~http~~:~~//cdn.opensfs.org/wp-content/uploads/2011/11/~~LUG-2012.~~pptx~~ High Availability Lustre Using SRP-Mirrored LUNs]" ([http://www.youtube.com/watch?v=dgRH-zheXzg video]).		At SSEC we tested mirrored meta data targets for our Lustre file systems. The idea is to use ZFS to mirror storage targets in 2 different MDS - so the data is always available on both servers without using iscsi or other technologies. The basis of this idea comes from Charles Taylor's LUG 2012 presentation "[[Media:LUG-2012-SRP_Mirroring-UFL.pdf\|High Availability Lustre Using SRP-Mirrored LUNs]]" ([http://www.youtube.com/watch?v=dgRH-zheXzg video]).

	Instead of LVM, we will use ZFS to provide the mirror. SCST for infiniband RDMA providing the targets and ZFS mirrors performed well in our testing. We did not have a chance to test more thoroughly for production.		Instead of LVM, we will use ZFS to provide the mirror. SCST for infiniband RDMA providing the targets and ZFS mirrors performed well in our testing. We did not have a chance to test more thoroughly for production.

Adilger: /* Setting up the Devices */ minor formatting

2015-08-18T15:12:57Z

Setting up the Devices: minor formatting

← Older revision		Revision as of 08:12, 18 August 2015
Line 79:		Line 79:
	=== Setting up the Devices ===		=== Setting up the Devices ===

	NOTE: we ended up using HW RAID on the bottom, exported that via SRP, and ZFS mirrored at the top level. This very first step can be skipped		NOTE: we ended up using HW RAID on the bottom, exported that via SRP, and ZFS mirrored at the top level. This very first step can be skipped.

	If you are using ZFS, you need to create a logical volume. For example, let's say we have the pool shps-meta which is comprised of some disks.		If you are using ZFS, you need to create a logical volume. For example, let's say we have the pool shps-meta which is comprised of some disks:

	zfs create -V 300G shps-meta/MDT zfs set canmount=off shps-meta		zfs create -V 300G shps-meta/MDT
			zfs set canmount=off shps-meta

	Now you have the device /dev/zvol/shps-meta/MDT		Now you have the device '''/dev/zvol/shps-meta/MDT'''.

	On each system, once you have your LUN prepared (with the RAID controller or zpool) it's time to register that device:		On each system, once you have your LUN prepared (with the RAID controller or zpool) it's time to register that device:
Line 93:		Line 94:
	Then list the device and target:		Then list the device and target:

	scstadmin -list_device scstadmin -list_target ls -l /sys/kernel/scst_tgt/devices		scstadmin -list_device
			scstadmin -list_target
			ls -l /sys/kernel/scst_tgt/devices

	You should get some info from each: the MDT1 dev you just created, and also ib_srpt_target0. If you don't get that, reload the ib_srpt module.		You should get some info from each: the MDT1 dev you just created, and also '''ib_srpt_target0'''. If you don't get that, reload the '''ib_srpt''' module.

	Define a security group (the hosts that can write):		Define a security group (the hosts that can write):

	scstadmin --add_group MDS -driver ib_srpt -target ib_srpt_target_0 scstadmin -list_group		scstadmin --add_group MDS -driver ib_srpt -target ib_srpt_target_0
			scstadmin -list_group

	Add initiators to the group:		Add initiators to the group:
Line 105:		Line 109:
	(for testing, leave this open)		(for testing, leave this open)

	Assign the LUNs to the target		Assign the LUNs to the target:

	scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group MDS -device MDT1		scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group MDS -device MDT1
Line 114:		Line 118:

	And enable the driver:		And enable the driver:

	scstadmin -set_drv_attr ib_srpt -attributes enabled=1		scstadmin -set_drv_attr ib_srpt -attributes enabled=1

	Modprobe modifications to pass the driver. This example is access over one-target-per-HCA-port		Modprobe modifications to pass the driver. This example is access over '''one-target-per-HCA-port''':

	cat /etc/modprobe.d/ib_srpt.conf options ib_srpt one_target_per_port=1		cat /etc/modprobe.d/ib_srpt.conf
			options ib_srpt one_target_per_port=1

	Set up permissions for the LUN (necessary)		Set up permissions for the LUN (necessary):

	scstadmin -add_init '*' -driver ib_srpt -target ib_srpt_target_0 -group MDS		scstadmin -add_init '*' -driver ib_srpt -target ib_srpt_target_0 -group MDS

Adilger: add URLs for presentation/video

2015-07-07T20:08:46Z

add URLs for presentation/video

← Older revision		Revision as of 13:08, 7 July 2015
Line 1:		Line 1:
	Original information provided by Jesse Stroik, January 2014.		Original information provided by Jesse Stroik, January 2014.

	At SSEC we tested mirrored meta data targets for our Lustre file systems. The idea is to use ZFS to mirror storage targets in 2 different MDS - so the data is always available on both servers without using iscsi or other technologies. The basis of this idea comes from Charles Taylor's LUG 2012 presentation "High Availability Lustre Using SRP-Mirrored LUNs"		At SSEC we tested mirrored meta data targets for our Lustre file systems. The idea is to use ZFS to mirror storage targets in 2 different MDS - so the data is always available on both servers without using iscsi or other technologies. The basis of this idea comes from Charles Taylor's LUG 2012 presentation "[http://cdn.opensfs.org/wp-content/uploads/2011/11/LUG-2012.pptx High Availability Lustre Using SRP-Mirrored LUNs]" ([http://www.youtube.com/watch?v=dgRH-zheXzg video]).

	Instead of LVM, we will use ZFS to provide the mirror. SCST for infiniband RDMA providing the targets and ZFS mirrors performed well in our testing. We did not have a chance to test more thoroughly for production.		Instead of LVM, we will use ZFS to provide the mirror. SCST for infiniband RDMA providing the targets and ZFS mirrors performed well in our testing. We did not have a chance to test more thoroughly for production.

ChrisMorrone: /* Setting up the Devices */

2015-06-25T18:12:21Z

Setting up the Devices

ChrisMorrone: /* Installing the SCST */

2015-06-25T18:11:00Z

Installing the SCST

← Older revision		Revision as of 11:11, 25 June 2015
Line 107:		Line 107:

	Assign the LUNs to the target		Assign the LUNs to the target
	scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group MDS -device MDT1
			scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group MDS -device MDT1

	Now enable the target:		Now enable the target:
Line 116:		Line 117:
	scstadmin -set_drv_attr ib_srpt -attributes enabled=1		scstadmin -set_drv_attr ib_srpt -attributes enabled=1

	Modprobe modifications to pass the driver. This example is access over one-target-per-HCA-port		Modprobe modifications to pass the driver. This example is access over one-target-per-HCA-port

	cat /etc/modprobe.d/ib_srpt.conf options ib_srpt one_target_per_port=1		cat /etc/modprobe.d/ib_srpt.conf options ib_srpt one_target_per_port=1
Line 132:		Line 133:
	First, load the module ib_srp:		First, load the module ib_srp:

	modprobe ib_srp		modprobe ib_srp

	Note: This module is part of OFED. OFED also includes the ib_srpt (target) module which is used to host the FS.		Note: This module is part of OFED. OFED also includes the ib_srpt (target) module which is used to host the FS.



	Now, search for the available targets:		Now, search for the available targets:

	srp_daemon -oacd/dev/infiniband/umad0		srp_daemon -oacd/dev/infiniband/umad0

	Note: There could be multiple /dev/infiniband/umad devices. (umad bro?)		Note: There could be multiple /dev/infiniband/umad devices. (umad bro?)


	Add the a		Add the a
	#scstadmin -add_target
			#scstadmin -add_target

			find /sys -iname add_target -print echo "id_ext=0002c90300b77f40,ioc_guid=0002c90300b77f40, dgid=fe800000000000000002c90300b77f41,pkey=ffff, \ service_id=0002c90300b77f40" > /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0/infiniband_srp/srp-mlx4_0-1/add_target

	find /sys -iname add_target -print echo "id_ext=0002c90300b77f40,ioc_guid=0002c90300b77f40, dgid=fe800000000000000002c90300b77f41,pkey=ffff, \ service_id=0002c90300b77f40" > /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0/infiniband_srp/srp-mlx4_0-1/add_target

	Note: in the above example, the part echoed is the result of the previous srp_daemon command (there may be multiple devices to add this way), and the redirection is into the result of the find command.		Note: in the above example, the part echoed is the result of the previous srp_daemon command (there may be multiple devices to add this way), and the redirection is into the result of the find command.



	Be sure to write the config:		Be sure to write the config:


	scstadmin -write_config /etc/scst.conf		scstadmin -write_config /etc/scst.conf

	And then ensure the startup script is in chkconfig:		And then ensure the startup script is in chkconfig:

	chkconfig --list scst ckconfig --list srpd chkconfig --list rdma		chkconfig --list scst ckconfig --list srpd chkconfig --list rdma

	/etc/rdma/rdma.conf must contain the line:		/etc/rdma/rdma.conf must contain the line:

	SRP_LOAD=yes		SRP_LOAD=yes

	== Metadata Backup ==		== Metadata Backup ==

ChrisMorrone: /* Setting up the Devices */

2015-06-25T18:09:45Z

Setting up the Devices

← Older revision		Revision as of 11:09, 25 June 2015
Line 109:		Line 109:
	scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group MDS -device MDT1		scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group MDS -device MDT1

	Now enable the target:		Now enable the target:


	scstadmin -enable_target ib_srpt_target_0 -driver ib_srpt		scstadmin -enable_target ib_srpt_target_0 -driver ib_srpt

	And enable the driver:		And enable the driver:
Line 124:		Line 123:

	scstadmin -add_init '*' -driver ib_srpt -target ib_srpt_target_0 -group MDS		scstadmin -add_init '*' -driver ib_srpt -target ib_srpt_target_0 -group MDS

	=== Initiator setup ===		=== Initiator setup ===

ChrisMorrone: /* Installing the SCST */

2015-06-25T18:08:49Z

Installing the SCST

Sknolin at 19:12, 6 May 2015

2015-05-06T19:12:44Z

← Older revision		Revision as of 12:12, 6 May 2015
Line 179:		Line 179:
	=== Snapshots ===		=== Snapshots ===

	[[ZFS Snapshots for MDT ~~Backup~~]]		[[ZFS Snapshots for MDT backup]]

	This is our backup. We can just use ZFS features to keep the metadata reasonably in sync. This won't be perfectly up to date, sadly, but a sync from an hour or two ago means an hour or two of data may have been lost, which is very acceptable in many cases.		This is our backup. We can just use ZFS features to keep the metadata reasonably in sync. This won't be perfectly up to date, sadly, but a sync from an hour or two ago means an hour or two of data may have been lost, which is very acceptable in many cases.

Sknolin at 19:12, 6 May 2015

2015-05-06T19:12:11Z

← Older revision		Revision as of 12:12, 6 May 2015
Line 178:		Line 178:

	=== Snapshots ===		=== Snapshots ===

			[[ZFS Snapshots for MDT Backup]]

	This is our backup. We can just use ZFS features to keep the metadata reasonably in sync. This won't be perfectly up to date, sadly, but a sync from an hour or two ago means an hour or two of data may have been lost, which is very acceptable in many cases.		This is our backup. We can just use ZFS features to keep the metadata reasonably in sync. This won't be perfectly up to date, sadly, but a sync from an hour or two ago means an hour or two of data may have been lost, which is very acceptable in many cases.

Sknolin at 18:55, 6 May 2015

2015-05-06T18:55:38Z

← Older revision		Revision as of 11:55, 6 May 2015
Line 175:		Line 175:
	SRP_LOAD=yes		SRP_LOAD=yes

	== Metadata Backup		== Metadata Backup ==

	=== Snapshots ===		=== Snapshots ===

← Older revision		Revision as of 11:12, 25 June 2015
Line 89:		Line 89:
	On each system, once you have your LUN prepared (with the RAID controller or zpool) it's time to register that device:		On each system, once you have your LUN prepared (with the RAID controller or zpool) it's time to register that device:

	scstadmin -open_dev MDT1 -handler vdisk_blockio -attributes filename=/dev/zvol/shps-meta/MDT		scstadmin -open_dev MDT1 -handler vdisk_blockio -attributes filename=/dev/zvol/shps-meta/MDT

	Then list the device and target:		Then list the device and target:

	scstadmin -list_device scstadmin -list_target ls -l /sys/kernel/scst_tgt/devices		scstadmin -list_device scstadmin -list_target ls -l /sys/kernel/scst_tgt/devices

	You should get some info from each: the MDT1 dev you just created, and also ib_srpt_target0. If you don't get that, reload the ib_srpt module.		You should get some info from each: the MDT1 dev you just created, and also ib_srpt_target0. If you don't get that, reload the ib_srpt module.
Line 99:		Line 99:
	Define a security group (the hosts that can write):		Define a security group (the hosts that can write):

	scstadmin --add_group MDS -driver ib_srpt -target ib_srpt_target_0 scstadmin -list_group		scstadmin --add_group MDS -driver ib_srpt -target ib_srpt_target_0 scstadmin -list_group

	Add initiators to the group:		Add initiators to the group:

	(for testing, leave this open)		(for testing, leave this open)


	Assign the LUNs to the target		Assign the LUNs to the target

	scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group MDS -device MDT1		scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group MDS -device MDT1

	Now enable the target:		Now enable the target:

	scstadmin -enable_target ib_srpt_target_0 -driver ib_srpt		scstadmin -enable_target ib_srpt_target_0 -driver ib_srpt

	And enable the driver:		And enable the driver:
	scstadmin -set_drv_attr ib_srpt -attributes enabled=1		scstadmin -set_drv_attr ib_srpt -attributes enabled=1

	Modprobe modifications to pass the driver. This example is access over one-target-per-HCA-port		Modprobe modifications to pass the driver. This example is access over one-target-per-HCA-port

	cat /etc/modprobe.d/ib_srpt.conf options ib_srpt one_target_per_port=1		cat /etc/modprobe.d/ib_srpt.conf options ib_srpt one_target_per_port=1

	Set up permissions for the LUN (necessary)		Set up permissions for the LUN (necessary)

← Older revision		Revision as of 11:08, 25 June 2015
Line 69:		Line 69:

	# Download and install the SCST package and scstadmin: http://sourceforge.net/projects/scst/files/		# Download and install the SCST package and scstadmin: http://sourceforge.net/projects/scst/files/

	# Extract the SCST package, and verify your Makefile lines if necessary:		# Extract the SCST package, and verify your Makefile lines if necessary:
	## export KDIR=/usr/src/kernels/2.6.32-431.el6.x86_64/ export KVER=2.6.32-431.el6.x86_64		## export KDIR=/usr/src/kernels/2.6.32-431.el6.x86_64/ export KVER=2.6.32-431.el6.x86_64

	# Build and install SRPT and SCST		# Build and install SRPT and SCST

	## make scst && make scst_install make srpt && make srpt_install		## make scst && make scst_install make srpt && make srpt_install
	## Then load the modules into the kernel, and set it up to start on boot.		## Then load the modules into the kernel, and set it up to start on boot.

	## /usr/lib/lsb/install_initd scst chkconfig --add scst		## /usr/lib/lsb/install_initd scst chkconfig --add scst
	## modprobe scst modprobe ib_srpt modprobe scst_vdisk		## modprobe scst modprobe ib_srpt modprobe scst_vdisk
Line 87:		Line 83:
	If you are using ZFS, you need to create a logical volume. For example, let's say we have the pool shps-meta which is comprised of some disks.		If you are using ZFS, you need to create a logical volume. For example, let's say we have the pool shps-meta which is comprised of some disks.

	## zfs create -V 300G shps-meta/MDT zfs set canmount=off shps-meta		zfs create -V 300G shps-meta/MDT zfs set canmount=off shps-meta

	~~###~~ Now you have the device /dev/zvol/shps-meta/MDT		Now you have the device /dev/zvol/shps-meta/MDT



	On each system, once you have your LUN prepared (with the RAID controller or zpool) it's time to register that device:		On each system, once you have your LUN prepared (with the RAID controller or zpool) it's time to register that device: