Lustre Object Storage Service (OSS) - Revision history

Malcolm at 04:39, 9 November 2017

2017-11-09T04:39:34Z

← Older revision		Revision as of 21:39, 8 November 2017
Line 21:		Line 21:

	<div id="Figure 2">		<div id="Figure 2">
	[[File:OSS Scaling lowres v1.png\|thumb\|center\|~~512px~~\|		[[File:OSS Scaling lowres v1.png\|thumb\|center\|576px\|
	Figure 2. Performance and Capacity Scale with Linearly with Increasing Numbers of OSSs and OSTs]]		Figure 2. Performance and Capacity Scale with Linearly with Increasing Numbers of OSSs and OSTs]]
	</div>		</div>

Malcolm: /* Object Storage Server Building Blocks */

2017-11-09T04:38:55Z

Object Storage Server Building Blocks

← Older revision		Revision as of 21:38, 8 November 2017
Line 36:		Line 36:

	<div id="Figure 3a">		<div id="Figure 3a">
	[[File:Object Storage Server HA Cluster small lowres v1.png\|thumb\|center\|~~512px~~\|		[[File:Object Storage Server HA Cluster small lowres v1.png\|thumb\|center\|576px\|
	Figure 3a. Object Storage Server HA building block with 4 OSTs using RAIDZ2 / RAID 6 (9+2)]]		Figure 3a. Object Storage Server HA building block with 4 OSTs using RAIDZ2 / RAID 6 (9+2)]]
	</div>		</div>

	<div id="Figure 3b">		<div id="Figure 3b">
	[[File:Object Storage Server HA Cluster large lowres v1.png\|thumb\|center\|~~512px~~\|		[[File:Object Storage Server HA Cluster large lowres v1.png\|thumb\|center\|576px\|
	Figure 3b. Object Storage Server HA building block with 10 OSTs using RAIDZ2 / RAID 6 (10+2)]]		Figure 3b. Object Storage Server HA building block with 10 OSTs using RAIDZ2 / RAID 6 (10+2)]]
	</div>		</div>

Malcolm: /* Object Storage Server Building Blocks */

2017-11-09T04:37:56Z

Object Storage Server Building Blocks

← Older revision		Revision as of 21:37, 8 November 2017
Line 31:		Line 31:
	As with other Lustre server components, object storage servers can be combined into high availability configurations, and this is the normal practice for Lustre file system implementations. Unlike the metadata and management services, object storage servers are only grouped with other object storage servers when designing for HA.		As with other Lustre server components, object storage servers can be combined into high availability configurations, and this is the normal practice for Lustre file system implementations. Unlike the metadata and management services, object storage servers are only grouped with other object storage servers when designing for HA.

	The physical design of a typical OSS building block is very similar to that of the MDS. The most significant external difference is the storage configuration, which is typically designed to maximize capacity and read/write bandwidth. Storage volumes are commonly formatted for RAID-6, or in the case of ZFS, ~~RAID-Z2~~. This provides a balance between optimal capacity and data integrity. The example in [[#Figure 3a\|Figure 3a]] is a low-density configuration with only 48 disks across two enclosures. It is more common to see multiple high-density storage enclosures with 60 disks, 72 disks, or more attached to object storage servers, an example of which is shown in [[#Figure 3b\|Figure 3b]].		The physical design of a typical OSS building block is very similar to that of the MDS. The most significant external difference is the storage configuration, which is typically designed to maximize capacity and read/write bandwidth. Storage volumes are commonly formatted for RAID 6, or in the case of ZFS, RAIDZ2. This provides a balance between optimal capacity and data integrity. The example in [[#Figure 3a\|Figure 3a]] is a low-density configuration with only 48 disks across two enclosures. It is more common to see multiple high-density storage enclosures with 60 disks, 72 disks, or more attached to object storage servers, an example of which is shown in [[#Figure 3b\|Figure 3b]].

	A typical building block configuration will comprise two OSS hosts connected to a common pool of shared storage. This storage may be a single enclosure or several, depending on the requirements of the implementation: a site may wish to optimize the Lustre installation for throughput performance, choosing lower capacity, high performance storage devices attached to a relatively high number of servers, or it may place an emphasis on capacity over throughput with high density storage connected to relatively fewer servers. Lustre is very versatile and affords system architects flexibility in designing a solution appropriate to the requirements of the site.		A typical building block configuration will comprise two OSS hosts connected to a common pool of shared storage. This storage may be a single enclosure or several, depending on the requirements of the implementation: a site may wish to optimize the Lustre installation for throughput performance, choosing lower capacity, high performance storage devices attached to a relatively high number of servers, or it may place an emphasis on capacity over throughput with high density storage connected to relatively fewer servers. Lustre is very versatile and affords system architects flexibility in designing a solution appropriate to the requirements of the site.
Line 37:		Line 37:
	<div id="Figure 3a">		<div id="Figure 3a">
	[[File:Object Storage Server HA Cluster small lowres v1.png\|thumb\|center\|512px\|		[[File:Object Storage Server HA Cluster small lowres v1.png\|thumb\|center\|512px\|
	Figure 3a. Object Storage Server HA building block with 4 OSTs using RAID 9+2]]		Figure 3a. Object Storage Server HA building block with 4 OSTs using RAIDZ2 / RAID 6 (9+2)]]
	</div>		</div>

	<div id="Figure 3b">		<div id="Figure 3b">
	[[File:Object Storage Server HA Cluster large lowres v1.png\|thumb\|center\|512px\|		[[File:Object Storage Server HA Cluster large lowres v1.png\|thumb\|center\|512px\|
	Figure 3b. Object Storage Server HA building block with 10 OSTs using RAID 10+2]]		Figure 3b. Object Storage Server HA building block with 10 OSTs using RAIDZ2 / RAID 6 (10+2)]]
	</div>		</div>

Adilger: /* Object Storage Server Design Considerations */ fix image name

2017-11-09T00:27:17Z

Object Storage Server Design Considerations: fix image name

← Older revision		Revision as of 17:27, 8 November 2017
Line 62:		Line 62:

	<div id="Figure 4">		<div id="Figure 4">
	[[File:~~ZFS Pool Performance and HDD Efficiency for Different Layouts v1~~.png\|thumb\|center\|512px\|		[[File:ZFS_Pool_Performance_and_HDD_Efficiency_for_Different_RAIDZ2_Layouts_v1.png\|thumb\|center\|512px\|
	Figure 4. Performance and HDD efficiency for different ZFS pool’s layout.		Figure 4. Performance and HDD efficiency for different ZFS pool’s layout.
	<br/><br/>		<br/><br/>

Adilger: /* Object Storage Server Design Considerations */ fix image name

2017-11-09T00:25:20Z

Object Storage Server Design Considerations: fix image name

← Older revision		Revision as of 17:25, 8 November 2017
Line 62:		Line 62:

	<div id="Figure 4">		<div id="Figure 4">
	[[File:ZFS Pool Performance and HDD Efficiency for Different ~~RAID-Z2~~ Layouts v1.png\|thumb\|center\|512px\|		[[File:ZFS Pool Performance and HDD Efficiency for Different Layouts v1.png\|thumb\|center\|512px\|
	Figure 4. Performance and HDD efficiency for different ZFS pool’s layout.		Figure 4. Performance and HDD efficiency for different ZFS pool’s layout.
	<br/><br/>		<br/><br/>

Adilger: minor edits

2017-11-09T00:23:10Z

minor edits

Show changes

Malcolm: /* Object Storage Server Design Considerations */

2017-09-22T00:07:36Z

Object Storage Server Design Considerations

← Older revision		Revision as of 17:07, 21 September 2017
Line 55:		Line 55:
	Whereas a RAID 6 volume or LUN for LDISKFS will be used as an individual object storage target, ZFS OSTs will be created from a single dataset per zpool. The composition of the zpool in terms of the vdevs does require experimentation to find the optimal arrangement balancing capacity utilization and performance.		Whereas a RAID 6 volume or LUN for LDISKFS will be used as an individual object storage target, ZFS OSTs will be created from a single dataset per zpool. The composition of the zpool in terms of the vdevs does require experimentation to find the optimal arrangement balancing capacity utilization and performance.

	Each OST zpool must contain a minimum of one ~~RAIDZ2~~ vdev, but may contain many, ~~creating~~ a ~~stripe of~~ RAIDZ2 vdevs. This arrangement, analogous to a RAID 6+0 layout for non-ZFS storage, can yield good performance for ZFS when compared to a pool containing a single RAIDZ2 vdev. Again, experimentation is essential in order to drive the best performance from the hardware.		Each OST zpool must contain a minimum of one vdev, but may contain many – ZFS will stripe data across all the vdevs in a pool. For example, a pool might contain multiple RAIDZ2 vdevs. This arrangement, analogous to a RAID 6+0 layout for non-ZFS storage, can yield good performance for ZFS when compared to a pool containing a single RAIDZ2 vdev. Again, experimentation is essential in order to drive the best performance from the hardware.

	In theory, when determining the best configuration for performance, the same layout used for LDISKFS RAID6 (namely 10 disks, 8+2) should also apply to a ZFS RAIDZ2 vdev, but this is not always the case. While LDISKFS OSDs with RAID-6 usually require 8+2 (or some other power of 2 layout) for best performance, ZFS + RAID-Z2 is much more flexible and the RAID geometry should be chosen to best match the JBOD enclosure (also taking spare devices into account). For example, a benchmarking study conducted by Intel® Corporation ([[#Figure 4\|Figure 4]]) found that using an 11- or 12-disk RAIDZ2 vdev configuration can yield better overall throughput results than a 10-disk allocation, particularly when working with large I/O block sizes. This data is based on ZFS version 0.6.5, and it is to be expected that as ZFS development continues, performance characteristics will continue to improve for different configurations.		In theory, when determining the best configuration for performance, the same layout used for LDISKFS RAID6 (namely 10 disks, 8+2) should also apply to a ZFS RAIDZ2 vdev, but this is not always the case. While LDISKFS OSDs with RAID-6 usually require 8+2 (or some other power of 2 layout) for best performance, ZFS + RAID-Z2 is much more flexible and the RAID geometry should be chosen to best match the JBOD enclosure (also taking spare devices into account). For example, a benchmarking study conducted by Intel® Corporation ([[#Figure 4\|Figure 4]]) found that using an 11- or 12-disk RAIDZ2 vdev configuration can yield better overall throughput results than a 10-disk allocation, particularly when working with large I/O block sizes. This data is based on ZFS version 0.6.5, and it is to be expected that as ZFS development continues, performance characteristics will continue to improve for different configurations.

Malcolm at 05:00, 8 September 2017

2017-09-08T05:00:18Z

← Older revision		Revision as of 22:00, 7 September 2017
Line 3:		Line 3:
	Files in Lustre are composed of one or more OST objects, in addition to the metadata inode. The allocation of objects to a file is referred to as the file’s layout, and is determined when the file is created. The layout for a file defines the set of OST objects that will be used to hold the file’s data. Each object for a given file is held on a separate OST, and data is written to the objects in a round-robin allocation as a stripe. As an analogy, one can think of the objects of a file as virtual equivalents of disk drives in a RAID-0 storage array. Data is written in fixed-sized chunks in stripes across the objects. The stripe width is also configurable when the file is created, and defaults to 1MiB.		Files in Lustre are composed of one or more OST objects, in addition to the metadata inode. The allocation of objects to a file is referred to as the file’s layout, and is determined when the file is created. The layout for a file defines the set of OST objects that will be used to hold the file’s data. Each object for a given file is held on a separate OST, and data is written to the objects in a round-robin allocation as a stripe. As an analogy, one can think of the objects of a file as virtual equivalents of disk drives in a RAID-0 storage array. Data is written in fixed-sized chunks in stripes across the objects. The stripe width is also configurable when the file is created, and defaults to 1MiB.

	<div id="Figure ~~2-3-~~1">		<div id="Figure 1">
	[[File:Object Storage Simple Stripe File Layout lowres v1.png\|thumb\|center\|512px\|		[[File:Object Storage Simple Stripe File Layout lowres v1.png\|thumb\|center\|512px\|
	Figure ~~2-3-~~1. Files are written to one or more objects stored on OSTs in a stripe pattern]]		Figure 1. Files are written to one or more objects stored on OSTs in a stripe pattern]]
	</div>		</div>

	[[#Figure ~~2-3-~~1\|Figure ~~2-3-~~1]] shows some examples of file with different storage layouts. File A is striped across 3 objects, File B comprises 2 objects and File C is a single object. The number of objects and the size of the stripe is configurable when the file is created.		[[#Figure 1\|Figure 1]] shows some examples of file with different storage layouts. File A is striped across 3 objects, File B comprises 2 objects and File C is a single object. The number of objects and the size of the stripe is configurable when the file is created.

	The object layout specification for a file can be supplied directly by the user or application, otherwise it will be assigned by the MDT based on either the file system default layout, or by a layout policy defined for the directory in which the file has been created. The MDT is responsible for the assignment of objects to a file.		The object layout specification for a file can be supplied directly by the user or application, otherwise it will be assigned by the MDT based on either the file system default layout, or by a layout policy defined for the directory in which the file has been created. The MDT is responsible for the assignment of objects to a file.
Line 20:		Line 20:
	Each OST operates independently of all other OSTs in the file system and there are no dependencies between objects themselves (a file may comprise multiple objects but the objects themselves don’t have a direct relationship). This, along with a flat namespace structure for objects, allows the performance of the file system to scale linearly as more OSTs are added.		Each OST operates independently of all other OSTs in the file system and there are no dependencies between objects themselves (a file may comprise multiple objects but the objects themselves don’t have a direct relationship). This, along with a flat namespace structure for objects, allows the performance of the file system to scale linearly as more OSTs are added.

	<div id="Figure ~~2-3-~~2">		<div id="Figure 2">
	[[File:OSS Scaling lowres v1.png\|thumb\|center\|512px\|		[[File:OSS Scaling lowres v1.png\|thumb\|center\|512px\|
	Figure ~~2-3-~~2. Performance and Capacity Scale with Linearly with Increasing Numbers of OSSs and OSTs]]		Figure 2. Performance and Capacity Scale with Linearly with Increasing Numbers of OSSs and OSTs]]
	</div>		</div>

Line 31:		Line 31:
	As for other Lustre server components, object storage servers can be combined into high availability configurations, and this is the normal practice for Lustre file system implementations. Unlike the metadata and management services, object storage servers are only grouped with other object storage servers when designing for HA.		As for other Lustre server components, object storage servers can be combined into high availability configurations, and this is the normal practice for Lustre file system implementations. Unlike the metadata and management services, object storage servers are only grouped with other object storage servers when designing for HA.

	The physical design of a typical OSS building block is very similar to that of the MDS. The most significant external difference is the storage configuration, which is typically designed to maximize capacity and read/write bandwidth. Storage volumes are commonly formatted for RAID 6, or in the case of ZFS, RAID-Z2. This provides a balance between optimal capacity and data integrity. The example in [[#Figure ~~2-3-~~3a\|Figure ~~2-3-~~3a]] is a low-density configuration with only 48 disks across two enclosures. It is more common to see multiple high-density storage enclosures with 60 disks, 72 disks, or more attached to object storage servers, an example of which is shown in [[#Figure ~~2-3-~~3b\|Figure ~~2-3-~~3b]].		The physical design of a typical OSS building block is very similar to that of the MDS. The most significant external difference is the storage configuration, which is typically designed to maximize capacity and read/write bandwidth. Storage volumes are commonly formatted for RAID 6, or in the case of ZFS, RAID-Z2. This provides a balance between optimal capacity and data integrity. The example in [[#Figure 3a\|Figure 3a]] is a low-density configuration with only 48 disks across two enclosures. It is more common to see multiple high-density storage enclosures with 60 disks, 72 disks, or more attached to object storage servers, an example of which is shown in [[#Figure 3b\|Figure 3b]].

	A typical building block configuration will comprise two OSS hosts connected to a common pool of shared storage. This storage may be a single enclosure or several, depending on the requirements of the implementation: a site may wish to optimize the Lustre installation for throughput performance, choosing lower capacity, high performance storage devices attached to a relatively high number of servers, or it may place an emphasis on capacity over throughput with high density storage connected to relatively fewer servers. Lustre is very versatile and affords system architects flexibility in designing a solution appropriate to the requirements of the site.		A typical building block configuration will comprise two OSS hosts connected to a common pool of shared storage. This storage may be a single enclosure or several, depending on the requirements of the implementation: a site may wish to optimize the Lustre installation for throughput performance, choosing lower capacity, high performance storage devices attached to a relatively high number of servers, or it may place an emphasis on capacity over throughput with high density storage connected to relatively fewer servers. Lustre is very versatile and affords system architects flexibility in designing a solution appropriate to the requirements of the site.

	<div id="Figure ~~2-3-~~3a">		<div id="Figure 3a">
	[[File:Object Storage Server HA Cluster small lowres v1.png\|thumb\|center\|512px\|		[[File:Object Storage Server HA Cluster small lowres v1.png\|thumb\|center\|512px\|
	Figure ~~2-3-~~3a. Object Storage Server HA building block with 4 OSTs]]		Figure 3a. Object Storage Server HA building block with 4 OSTs]]
	</div>		</div>

	<div id="Figure ~~2-3-~~3b">		<div id="Figure 3b">
	[[File:Object Storage Server HA Cluster large lowres v1.png\|thumb\|center\|512px\|		[[File:Object Storage Server HA Cluster large lowres v1.png\|thumb\|center\|512px\|
	Figure ~~2-3-~~3b. Object Storage Server HA building block with 10 OSTs]]		Figure 3b. Object Storage Server HA building block with 10 OSTs]]
	</div>		</div>

Line 57:		Line 57:
	Each OST zpool must contain a minimum of one RAIDZ2 vdev, but may contain many, creating a stripe of RAIDZ2 vdevs. This arrangement, analogous to a RAID 6+0 layout for non-ZFS storage, can yield good performance for ZFS when compared to a pool containing a single RAIDZ2 vdev. Again, experimentation is essential in order to drive the best performance from the hardware.		Each OST zpool must contain a minimum of one RAIDZ2 vdev, but may contain many, creating a stripe of RAIDZ2 vdevs. This arrangement, analogous to a RAID 6+0 layout for non-ZFS storage, can yield good performance for ZFS when compared to a pool containing a single RAIDZ2 vdev. Again, experimentation is essential in order to drive the best performance from the hardware.

	In theory, when determining the best configuration for performance, the same layout used for LDISKFS RAID6 (namely 10 disks, 8+2) should also apply to a ZFS RAIDZ2 vdev, but this is not always the case. While LDISKFS OSDs with RAID-6 usually require 8+2 (or some other power of 2 layout) for best performance, ZFS + RAID-Z2 is much more flexible and the RAID geometry should be chosen to best match the JBOD enclosure (also taking spare devices into account). For example, a benchmarking study conducted by Intel® Corporation ([[#Figure ~~2-3-~~4\|Figure ~~2-3-~~4]]) found that using an 11- or 12-disk RAIDZ2 vdev configuration can yield better overall throughput results than a 10-disk allocation, particularly when working with large I/O block sizes. This data is based on ZFS version 0.6.5, and it is to be expected that as ZFS development continues, performance characteristics will continue to improve for different configurations.		In theory, when determining the best configuration for performance, the same layout used for LDISKFS RAID6 (namely 10 disks, 8+2) should also apply to a ZFS RAIDZ2 vdev, but this is not always the case. While LDISKFS OSDs with RAID-6 usually require 8+2 (or some other power of 2 layout) for best performance, ZFS + RAID-Z2 is much more flexible and the RAID geometry should be chosen to best match the JBOD enclosure (also taking spare devices into account). For example, a benchmarking study conducted by Intel® Corporation ([[#Figure 4\|Figure 4]]) found that using an 11- or 12-disk RAIDZ2 vdev configuration can yield better overall throughput results than a 10-disk allocation, particularly when working with large I/O block sizes. This data is based on ZFS version 0.6.5, and it is to be expected that as ZFS development continues, performance characteristics will continue to improve for different configurations.

	Experimenting with different RAIDZ layouts, such as using 11 or 12 disks for RAIDZ2 as well as the more conventional or traditional 10 disk configuration, is therefore recommended in order to identify the optimum structure for achieving strong performance.		Experimenting with different RAIDZ layouts, such as using 11 or 12 disks for RAIDZ2 as well as the more conventional or traditional 10 disk configuration, is therefore recommended in order to identify the optimum structure for achieving strong performance.

	<div id="Figure ~~2-3-~~4">		<div id="Figure 4">
	[[File:ZFS Pool Performance and HDD Efficiency for Different RAIDZ2 Layouts v1.png\|thumb\|center\|512px\|		[[File:ZFS Pool Performance and HDD Efficiency for Different RAIDZ2 Layouts v1.png\|thumb\|center\|512px\|
	Figure ~~2-3-~~4. Performance and HDD efficiency for different ZFS pool’s layout.		Figure 4. Performance and HDD efficiency for different ZFS pool’s layout.
	<br/><br/>		<br/><br/>
	'''Note:''' <code>obdfilter-survey</code> write benchmark on one single OST using 8 objects and 64 threads. Comparison of 4 different ZFS pool layouts: 8+2, 9+2, 10+2 and 12+2. The primary Y axis measures global performance, the secondary Y axis measures the efficiency of a single disk, using only the active disks (8/9/10/12) for the calculation. The test was performed on a 60-HDD JBOD.]]		'''Note:''' <code>obdfilter-survey</code> write benchmark on one single OST using 8 objects and 64 threads. Comparison of 4 different ZFS pool layouts: 8+2, 9+2, 10+2 and 12+2. The primary Y axis measures global performance, the secondary Y axis measures the efficiency of a single disk, using only the active disks (8/9/10/12) for the calculation. The test was performed on a 60-HDD JBOD.]]

Malcolm at 04:35, 8 September 2017

2017-09-08T04:35:53Z

← Older revision		Revision as of 21:35, 7 September 2017
Line 3:		Line 3:
	Files in Lustre are composed of one or more OST objects, in addition to the metadata inode. The allocation of objects to a file is referred to as the file’s layout, and is determined when the file is created. The layout for a file defines the set of OST objects that will be used to hold the file’s data. Each object for a given file is held on a separate OST, and data is written to the objects in a round-robin allocation as a stripe. As an analogy, one can think of the objects of a file as virtual equivalents of disk drives in a RAID-0 storage array. Data is written in fixed-sized chunks in stripes across the objects. The stripe width is also configurable when the file is created, and defaults to 1MiB.		Files in Lustre are composed of one or more OST objects, in addition to the metadata inode. The allocation of objects to a file is referred to as the file’s layout, and is determined when the file is created. The layout for a file defines the set of OST objects that will be used to hold the file’s data. Each object for a given file is held on a separate OST, and data is written to the objects in a round-robin allocation as a stripe. As an analogy, one can think of the objects of a file as virtual equivalents of disk drives in a RAID-0 storage array. Data is written in fixed-sized chunks in stripes across the objects. The stripe width is also configurable when the file is created, and defaults to 1MiB.

	<div id="Figure 4">		<div id="Figure 2-3-1">
	[[File:Object Storage Simple Stripe File Layout lowres v1.png\|thumb\|center\|~~450px~~\|		[[File:Object Storage Simple Stripe File Layout lowres v1.png\|thumb\|center\|512px\|
	Figure 4. Files are written to one or more objects stored on OSTs in a stripe pattern]]		Figure 2-3-1. Files are written to one or more objects stored on OSTs in a stripe pattern]]
	</div>		</div>

	[[#Figure 4\|Figure 4]] shows some examples of file with different storage layouts. File A is striped across 3 objects, File B comprises 2 objects and File C is a single object. The number of objects and the size of the stripe is configurable when the file is created.		[[#Figure 2-3-1\|Figure 2-3-1]] shows some examples of file with different storage layouts. File A is striped across 3 objects, File B comprises 2 objects and File C is a single object. The number of objects and the size of the stripe is configurable when the file is created.

	The object layout specification for a file can be supplied directly by the user or application, otherwise it will be assigned by the MDT based on either the file system default layout, or by a layout policy defined for the directory in which the file has been created. The MDT is responsible for the assignment of objects to a file.		The object layout specification for a file can be supplied directly by the user or application, otherwise it will be assigned by the MDT based on either the file system default layout, or by a layout policy defined for the directory in which the file has been created. The MDT is responsible for the assignment of objects to a file.
Line 20:		Line 20:
	Each OST operates independently of all other OSTs in the file system and there are no dependencies between objects themselves (a file may comprise multiple objects but the objects themselves don’t have a direct relationship). This, along with a flat namespace structure for objects, allows the performance of the file system to scale linearly as more OSTs are added.		Each OST operates independently of all other OSTs in the file system and there are no dependencies between objects themselves (a file may comprise multiple objects but the objects themselves don’t have a direct relationship). This, along with a flat namespace structure for objects, allows the performance of the file system to scale linearly as more OSTs are added.

	<div id="Figure 5">		<div id="Figure 2-3-2">
	[[File:OSS Scaling lowres v1.png\|thumb\|center\|~~450px~~\|		[[File:OSS Scaling lowres v1.png\|thumb\|center\|512px\|
	Figure 5. Performance and Capacity Scale with Linearly with Increasing Numbers of OSSs and OSTs]]		Figure 2-3-2. Performance and Capacity Scale with Linearly with Increasing Numbers of OSSs and OSTs]]
	</div>		</div>

Line 31:		Line 31:
	As for other Lustre server components, object storage servers can be combined into high availability configurations, and this is the normal practice for Lustre file system implementations. Unlike the metadata and management services, object storage servers are only grouped with other object storage servers when designing for HA.		As for other Lustre server components, object storage servers can be combined into high availability configurations, and this is the normal practice for Lustre file system implementations. Unlike the metadata and management services, object storage servers are only grouped with other object storage servers when designing for HA.

	The physical design of a typical OSS building block is very similar to that of the MDS. The most significant external difference is the storage configuration, which is typically designed to maximize capacity and read/write bandwidth. Storage volumes are commonly formatted for RAID 6, or in the case of ZFS, RAID-Z2. This provides a balance between optimal capacity and data integrity. The example in [[#Figure 6a\|Figure 6a]] is a low-density configuration with only 48 disks across two enclosures. It is more common to see multiple high-density storage enclosures with 60 disks, 72 disks, or more attached to object storage servers, an example of which is shown in [[#Figure 6b\|Figure 6b]].		The physical design of a typical OSS building block is very similar to that of the MDS. The most significant external difference is the storage configuration, which is typically designed to maximize capacity and read/write bandwidth. Storage volumes are commonly formatted for RAID 6, or in the case of ZFS, RAID-Z2. This provides a balance between optimal capacity and data integrity. The example in [[#Figure 2-3-3a\|Figure 2-3-3a]] is a low-density configuration with only 48 disks across two enclosures. It is more common to see multiple high-density storage enclosures with 60 disks, 72 disks, or more attached to object storage servers, an example of which is shown in [[#Figure 2-3-3b\|Figure 2-3-3b]].

	A typical building block configuration will comprise two OSS hosts connected to a common pool of shared storage. This storage may be a single enclosure or several, depending on the requirements of the implementation: a site may wish to optimize the Lustre installation for throughput performance, choosing lower capacity, high performance storage devices attached to a relatively high number of servers, or it may place an emphasis on capacity over throughput with high density storage connected to relatively fewer servers. Lustre is very versatile and affords system architects flexibility in designing a solution appropriate to the requirements of the site.		A typical building block configuration will comprise two OSS hosts connected to a common pool of shared storage. This storage may be a single enclosure or several, depending on the requirements of the implementation: a site may wish to optimize the Lustre installation for throughput performance, choosing lower capacity, high performance storage devices attached to a relatively high number of servers, or it may place an emphasis on capacity over throughput with high density storage connected to relatively fewer servers. Lustre is very versatile and affords system architects flexibility in designing a solution appropriate to the requirements of the site.

	<div id="Figure 6a">		<div id="Figure 2-3-3a">
	[[File:Object Storage Server HA Cluster small lowres v1.png\|thumb\|center\|~~450px~~\|		[[File:Object Storage Server HA Cluster small lowres v1.png\|thumb\|center\|512px\|
	Figure 6a. Object Storage Server HA building block with 4 OSTs]]		Figure 2-3-3a. Object Storage Server HA building block with 4 OSTs]]
	</div>		</div>

	<div id="Figure 6b">		<div id="Figure 2-3-3b">
	[[File:Object Storage Server HA Cluster large lowres v1.png\|thumb\|center\|~~450px~~\|		[[File:Object Storage Server HA Cluster large lowres v1.png\|thumb\|center\|512px\|
	Figure 6b. Object Storage Server HA building block with 10 OSTs]]		Figure 2-3-3b. Object Storage Server HA building block with 10 OSTs]]
	</div>		</div>

Line 57:		Line 57:
	Each OST zpool must contain a minimum of one RAIDZ2 vdev, but may contain many, creating a stripe of RAIDZ2 vdevs. This arrangement, analogous to a RAID 6+0 layout for non-ZFS storage, can yield good performance for ZFS when compared to a pool containing a single RAIDZ2 vdev. Again, experimentation is essential in order to drive the best performance from the hardware.		Each OST zpool must contain a minimum of one RAIDZ2 vdev, but may contain many, creating a stripe of RAIDZ2 vdevs. This arrangement, analogous to a RAID 6+0 layout for non-ZFS storage, can yield good performance for ZFS when compared to a pool containing a single RAIDZ2 vdev. Again, experimentation is essential in order to drive the best performance from the hardware.

	In theory, when determining the best configuration for performance, the same layout used for LDISKFS RAID6 (namely 10 disks, 8+2) should also apply to a ZFS RAIDZ2 vdev, but this is not always the case. While LDISKFS OSDs with RAID-6 usually require 8+2 (or some other power of 2 layout) for best performance, ZFS + RAID-Z2 is much more flexible and the RAID geometry should be chosen to best match the JBOD enclosure (also taking spare devices into account). For example, a benchmarking study conducted by Intel® Corporation ([[#Figure 7\|Figure 7]]) found that using an 11- or 12-disk RAIDZ2 vdev configuration can yield better overall throughput results than a 10-disk allocation, particularly when working with large I/O block sizes. This data is based on ZFS version 0.6.5, and it is to be expected that as ZFS development continues, performance characteristics will continue to improve for different configurations.		In theory, when determining the best configuration for performance, the same layout used for LDISKFS RAID6 (namely 10 disks, 8+2) should also apply to a ZFS RAIDZ2 vdev, but this is not always the case. While LDISKFS OSDs with RAID-6 usually require 8+2 (or some other power of 2 layout) for best performance, ZFS + RAID-Z2 is much more flexible and the RAID geometry should be chosen to best match the JBOD enclosure (also taking spare devices into account). For example, a benchmarking study conducted by Intel® Corporation ([[#Figure 2-3-4\|Figure 2-3-4]]) found that using an 11- or 12-disk RAIDZ2 vdev configuration can yield better overall throughput results than a 10-disk allocation, particularly when working with large I/O block sizes. This data is based on ZFS version 0.6.5, and it is to be expected that as ZFS development continues, performance characteristics will continue to improve for different configurations.

	Experimenting with different RAIDZ layouts, such as using 11 or 12 disks for RAIDZ2 as well as the more conventional or traditional 10 disk configuration, is therefore recommended in order to identify the optimum structure for achieving strong performance.		Experimenting with different RAIDZ layouts, such as using 11 or 12 disks for RAIDZ2 as well as the more conventional or traditional 10 disk configuration, is therefore recommended in order to identify the optimum structure for achieving strong performance.

	<div id="Figure 7">		<div id="Figure 2-3-4">
	[[File:ZFS Pool Performance and HDD Efficiency for Different RAIDZ2 Layouts v1.png\|thumb\|center\|~~450px~~\|		[[File:ZFS Pool Performance and HDD Efficiency for Different RAIDZ2 Layouts v1.png\|thumb\|center\|512px\|
	Figure 7. Performance and HDD efficiency for different ZFS pool’s layout.		Figure 2-3-4. Performance and HDD efficiency for different ZFS pool’s layout.
	<br/><br/>		<br/><br/>
	'''Note:''' <code>obdfilter-survey</code> write benchmark on one single OST using 8 objects and 64 threads. Comparison of 4 different ZFS pool layouts: 8+2, 9+2, 10+2 and 12+2. The primary Y axis measures global performance, the secondary Y axis measures the efficiency of a single disk, using only the active disks (8/9/10/12) for the calculation. The test was performed on a 60-HDD JBOD.]]		'''Note:''' <code>obdfilter-survey</code> write benchmark on one single OST using 8 objects and 64 threads. Comparison of 4 different ZFS pool layouts: 8+2, 9+2, 10+2 and 12+2. The primary Y axis measures global performance, the secondary Y axis measures the efficiency of a single disk, using only the active disks (8/9/10/12) for the calculation. The test was performed on a 60-HDD JBOD.]]

Malcolm at 05:45, 31 August 2017

2017-08-31T05:45:14Z

← Older revision		Revision as of 22:45, 30 August 2017
Line 1:		Line 1:
	~~= Lustre Object Storage Service (OSS) =~~

	The Object Storage Servers (OSS) in a Lustre file system provide the bulk data storage for all file content. Each OSS provides access to a set of storage volumes referred to as Object Storage Targets (OSTs) and each object storage target contains a number of binary objects representing the data for files in Lustre.		The Object Storage Servers (OSS) in a Lustre file system provide the bulk data storage for all file content. Each OSS provides access to a set of storage volumes referred to as Object Storage Targets (OSTs) and each object storage target contains a number of binary objects representing the data for files in Lustre.

Line 71:		Line 69:

	[[Category:Lustre File System Components]]		[[Category:Lustre File System Components]]
			[[Category:Lustre Systems Administration]]