Erasure Coding Read-Only High Level Design - Revision history

Mvef: Adding "Supporting Multiple EC Code Types" section

2026-03-24T13:18:38Z

Adding "Supporting Multiple EC Code Types" section

@@ Line 221: / Line 221: @@
 ==Requirements==
 ===Erasure Code Library===
-We’d choose an existing library supporting erasure code encode and decode. Intel Intelligent Storage Acceleration Library (ISA-L) (https://github.com/01org/isa-l) supports fast block Reed-Solomon type erasure codes for any encode/decode matrix in GF(28), and we can leverage for the parity code generating and data restore.
+We’d choose an existing library supporting erasure code encode and decode. Intel Intelligent Storage Acceleration Library (ISA-L) (https://github.com/01org/isa-l) supports fast block Reed-Solomon type erasure codes for any encode/decode matrix in GF(2^8), and we can leverage for the parity code generating and data restore.
 ===User Space Tools===
@@ Line 237: / Line 237: @@
 ===Write to EC File with Immediate Parity===
 In a later development phase, it may be possible to implement Immediate EC Write for the restricted (but fairly common) use case of a single client writing a new file in linear offset order (which is the only IO method supported for most object stores).  The EC components would be initialized as "in progress" or "STALE" during initial file writes. The client could compute the code for the EC component in a similar manner as is done for Read of Degraded File, and write it asynchronously to the EC stripes.  If the write of the file's data and code components have completed and sync'd without errors (the most common case), then the EC component(s) can be marked uptodate when the file is closed.  This is essentially combining the Delayed Write with the Erasure Code Resynchronization step, and could cover a large fraction of the normal use cases.
 [[Category:Architecture]]
 [[Category:FLR]]
 [[Category:Layout]]

Adilger: /* RAID Sets */ identify where the component boundaries are.

2026-03-11T18:59:03Z

RAID Sets: identify where the component boundaries are.

← Older revision		Revision as of 11:59, 11 March 2026
Line 72:		Line 72:
	For files where the stripe count exceeds the number of stripes in the EC geometry, the concept of a RAID set is introduced. This is necessary when files are widely striped (e.g. 128 data stripes) to allow the EC Geometry to remain reasonable for reconstruction. In such cases, the data stripes are split into sets of <code>k</code> stripes, each of which has its own set of <code>m</code> parity stripes. For example, in a 16-stripe file with <code>--ec 8+2</code> layout there will be 16 data stripes in the data mirror split into two 8+2 RAID sets, with two corresponding 2-stripe EC parity stripes (total 4 stripes) in the EC mirror. This allows the EC parity calculations to remain relatively efficient, while still allowing widely-striped files to have parity.		For files where the stripe count exceeds the number of stripes in the EC geometry, the concept of a RAID set is introduced. This is necessary when files are widely striped (e.g. 128 data stripes) to allow the EC Geometry to remain reasonable for reconstruction. In such cases, the data stripes are split into sets of <code>k</code> stripes, each of which has its own set of <code>m</code> parity stripes. For example, in a 16-stripe file with <code>--ec 8+2</code> layout there will be 16 data stripes in the data mirror split into two 8+2 RAID sets, with two corresponding 2-stripe EC parity stripes (total 4 stripes) in the EC mirror. This allows the EC parity calculations to remain relatively efficient, while still allowing widely-striped files to have parity.

	The following image shows an example RAID-4 layout with 6 ~~data~~ stripes in two <code>k=3 m=1</code> RAID sets (0, 1, 2 and 3, 4, 5) and ~~two~~ parity stripes (p, q), one for each RAID set. The data stripes 0, 1, 2 use parity stripe p, and data stripes 3, 4, 5 use parity stripe q. The 6 data stripes themselves are in a single component with a RAID-0 layout that make up the data mirror, and the two EC parity stripe are in their own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parities (Ap+Aq, Bp+Bq, Cp+Cq, Dp+Dq) in the parity stripe. It should be noted that this is only an example to illustrate the concept of RAID sets in a constrained space, and in a real system it is much more likely that this file would use a single EC 6+2 geometry to allow recovery from two concurrent failures.		The following image shows an example RAID-4 layout with 6 stripes in the data mirror component split into two <code>k=3 m=1</code> RAID sets (0, 1, 2 and 3, 4, 5), and the EC mirror component has 2 parity stripes (p, q), one for each RAID set. The data stripes 0, 1, 2 use parity stripe p, and data stripes 3, 4, 5 use parity stripe q. The 6 data stripes themselves are in a single component with a RAID-0 layout that make up the data mirror, and the two EC parity stripe are in their own component that makes up the EC mirror. Each data and parity stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parities (Ap+Aq, Bp+Bq, Cp+Cq, Dp+Dq) in the parity stripe. It should be noted that this is only an example to illustrate the concept of RAID sets in a constrained space, and in a real system it is much more likely that this file would use a single EC 6+2 geometry to allow recovery from two concurrent failures.

	[[File:RAID_set_2x3+1.png]]		[[File:RAID_set_2x3+1.png]]

Adilger: /* RAID Sets */ add discussion of unequal RAID sets in a file

2026-03-11T18:55:36Z

RAID Sets: add discussion of unequal RAID sets in a file

← Older revision		Revision as of 11:55, 11 March 2026
Line 77:		Line 77:

	RAID sets also represent isolated failure domains within a file, allowing the reconstruction of data independently and in parallel across each RAID set. In addition, for widely-striped files where the number of data stripes are allocated on most or all of the OSTs in the filesystem, it allows the data and parity stripes to be distributed across all of the OSTs in the filesystem since the only requirement to maintain redundancy is that the OSTs do not overlap within a single RAID set.		RAID sets also represent isolated failure domains within a file, allowing the reconstruction of data independently and in parallel across each RAID set. In addition, for widely-striped files where the number of data stripes are allocated on most or all of the OSTs in the filesystem, it allows the data and parity stripes to be distributed across all of the OSTs in the filesystem since the only requirement to maintain redundancy is that the OSTs do not overlap within a single RAID set.

			==== Unequal RAID Sets====
			If the <code>stripe_count</code> of a file is not an integer multiple of <code>k</code>, then the RAID sets in that file will not all have identical EC geometry. This can happen for multiple reasons, such as the total OST count not matching the EC geometry, OSTs being offline or full when files are being created, if a single EC geometry is specified for a PFL file that has component stripe counts that do not match, if there is a default EC geometry for the filesystem that doesn't align with the default stripe count in a directory, or if the user explicitly specifies a stripe count and EC geometry that do not match.

			In such cases, the EC geometry will be adjusted automatically to best align with the file's (component's) data stripe count, '''''while maintaining at least the requested redundancy'''''. That means that the EC geometry may reduce the EC stripe count <code>k</code> to better align with the file's stripe count, so that all of the RAID sets in a component will have an EC stripe count within one of each other. The RAID sets with the higher stripe count will cover the first stripes of the component, and 0 or more RAID sets with one fewer stripe will cover the last stripes of the component.

			For example, a file with <code>stripe_count=20, k=8</code> ''could'' have three RAID sets <code>8+2</code>, <code>8+2</code>, <code>4+2</code> to cover the 20 data stripes, but the last RAID set would have higher EC overhead compared to the first two. Instead, the EC layout would use RAID sets <code>7+2</code>, <code>7+2</code>, <code>6+2</code> so that the EC overhead is more uniform across the file.

			The following image shows an example EC layout with 7 data stripes with one <code>k=4 m=1</code> RAID set and one <code>k=3 m=1</code> RAID set (data stripes 0, 1, 2, 3, and 4, 5, 6 respectively) and two parity stripes (p, q), one for each RAID set. The data stripes 0, 1, 2, 3 use parity stripe p, and data stripes 4, 5, 6 use parity stripe q.

			[[File:RAID_set_uneven.png]]

	====layout header====		====layout header====

Adilger: /* RAID Sets */ clarify that k=3 for the RAID Set and not the stripe count.

2026-03-11T18:03:53Z

RAID Sets: clarify that k=3 for the RAID Set and not the stripe count.

← Older revision		Revision as of 11:03, 11 March 2026
Line 72:		Line 72:
	For files where the stripe count exceeds the number of stripes in the EC geometry, the concept of a RAID set is introduced. This is necessary when files are widely striped (e.g. 128 data stripes) to allow the EC Geometry to remain reasonable for reconstruction. In such cases, the data stripes are split into sets of <code>k</code> stripes, each of which has its own set of <code>m</code> parity stripes. For example, in a 16-stripe file with <code>--ec 8+2</code> layout there will be 16 data stripes in the data mirror split into two 8+2 RAID sets, with two corresponding 2-stripe EC parity stripes (total 4 stripes) in the EC mirror. This allows the EC parity calculations to remain relatively efficient, while still allowing widely-striped files to have parity.		For files where the stripe count exceeds the number of stripes in the EC geometry, the concept of a RAID set is introduced. This is necessary when files are widely striped (e.g. 128 data stripes) to allow the EC Geometry to remain reasonable for reconstruction. In such cases, the data stripes are split into sets of <code>k</code> stripes, each of which has its own set of <code>m</code> parity stripes. For example, in a 16-stripe file with <code>--ec 8+2</code> layout there will be 16 data stripes in the data mirror split into two 8+2 RAID sets, with two corresponding 2-stripe EC parity stripes (total 4 stripes) in the EC mirror. This allows the EC parity calculations to remain relatively efficient, while still allowing widely-striped files to have parity.

	The following image shows an example RAID-4 layout with 6 data stripes in two 3+1 RAID sets (0, 1, 2 and 3, 4, 5) and two parity stripes (p, q), one for each RAID set. The data stripes 0, 1, 2 use parity stripe p, and data stripes 3, 4, 5 use parity stripe q. The 6 data stripes themselves are in a single component with a RAID-0 layout that make up the data mirror, and the two EC parity stripe are in their own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parities (Ap+Aq, Bp+Bq, Cp+Cq, Dp+Dq) in the parity stripe. It should be noted that this is only an example to illustrate the concept of RAID sets in a constrained space, and in a real system it is much more likely that this file would use a single EC 6+2 geometry to allow recovery from two concurrent failures.		The following image shows an example RAID-4 layout with 6 data stripes in two <code>k=3 m=1</code> RAID sets (0, 1, 2 and 3, 4, 5) and two parity stripes (p, q), one for each RAID set. The data stripes 0, 1, 2 use parity stripe p, and data stripes 3, 4, 5 use parity stripe q. The 6 data stripes themselves are in a single component with a RAID-0 layout that make up the data mirror, and the two EC parity stripe are in their own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parities (Ap+Aq, Bp+Bq, Cp+Cq, Dp+Dq) in the parity stripe. It should be noted that this is only an example to illustrate the concept of RAID sets in a constrained space, and in a real system it is much more likely that this file would use a single EC 6+2 geometry to allow recovery from two concurrent failures.

	[[File:RAID_set_2x3+1.png]]		[[File:RAID_set_2x3+1.png]]

Adilger: /* EC Geometry */ mention "k" and "m" to tie into standard RAID terminology

2026-03-11T18:01:41Z

EC Geometry: mention "k" and "m" to tie into standard RAID terminology

← Older revision		Revision as of 11:01, 11 March 2026
Line 57:		Line 57:
	The Lustre FLR-EC layout will implement redundancy on a per-file basis, with the data and parity being self-contained in the objects allocated to each file. This allows a great deal of flexibility in EC geometries, since they can be selected and tuned on a per-file, per-directory, and per-application basis. A file layout can be mapped conceptually as if each OST object is a "disk" in a traditional RAID device, where each file has a different set of "disks" (objects) that are wholly independent of the layout used by other files. The data stripes ("disks") will be arranged in a RAID-0 format as they are for traditional Lustre striped file layouts. The EC parity will be isolated to their own stripes ("disks"), which results in a [https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_4 RAID-4] style layout (dedicated parity "disks") rather than RAID-5 or RAID-6 where the parity is interleaved with the file data to balance load during parity reconstruction.		The Lustre FLR-EC layout will implement redundancy on a per-file basis, with the data and parity being self-contained in the objects allocated to each file. This allows a great deal of flexibility in EC geometries, since they can be selected and tuned on a per-file, per-directory, and per-application basis. A file layout can be mapped conceptually as if each OST object is a "disk" in a traditional RAID device, where each file has a different set of "disks" (objects) that are wholly independent of the layout used by other files. The data stripes ("disks") will be arranged in a RAID-0 format as they are for traditional Lustre striped file layouts. The EC parity will be isolated to their own stripes ("disks"), which results in a [https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_4 RAID-4] style layout (dedicated parity "disks") rather than RAID-5 or RAID-6 where the parity is interleaved with the file data to balance load during parity reconstruction.

	The following figure shows a RAID-4 layout with 3 data stripes (0, 1, 2) and 1 parity stripe (p). The 3 data stripes themselves are in a component with a RAID-0 layout that make up the data mirror, and the single EC parity stripe is in its own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parity (Ap, Bp, Cp, Dp) in the parity stripe.		The following figure shows a RAID-4 layout with k=3 data stripes (0, 1, 2) and m=1 parity stripe (p). The 3 data stripes themselves are in a component with a RAID-0 layout that make up the data mirror, and the single EC parity stripe is in its own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parity (Ap, Bp, Cp, Dp) in the parity stripe.

	[[File:RAID_4.png]]		[[File:RAID_4.png]]

Adilger: /* Introduction */ include "HLD" in the page so it is more easily found.

2026-03-11T17:55:16Z

Introduction: include "HLD" in the page so it is more easily found.

← Older revision		Revision as of 10:55, 11 March 2026
Line 1:		Line 1:
	==Introduction==		==Introduction==
	The Erasure Coding High Level Design describes details of how the EC feature may be implemented, including the user interfaces, how files with EC feature interact with client-IO layer, the RPC interchange between the client and servers.		The Erasure Coding High Level Design (HLD) describes details of how the EC feature may be implemented, including the user interfaces, how files with EC feature interact with client-IO layer, the RPC interchange between the client and servers.

	==Design Overview==		==Design Overview==

Adilger: /* RAID Sets */ add new copy of picture because of weird display issue

2026-02-27T19:14:45Z

RAID Sets: add new copy of picture because of weird display issue

← Older revision		Revision as of 12:14, 27 February 2026
Line 74:		Line 74:
	The following image shows an example RAID-4 layout with 6 data stripes in two 3+1 RAID sets (0, 1, 2 and 3, 4, 5) and two parity stripes (p, q), one for each RAID set. The data stripes 0, 1, 2 use parity stripe p, and data stripes 3, 4, 5 use parity stripe q. The 6 data stripes themselves are in a single component with a RAID-0 layout that make up the data mirror, and the two EC parity stripe are in their own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parities (Ap+Aq, Bp+Bq, Cp+Cq, Dp+Dq) in the parity stripe. It should be noted that this is only an example to illustrate the concept of RAID sets in a constrained space, and in a real system it is much more likely that this file would use a single EC 6+2 geometry to allow recovery from two concurrent failures.		The following image shows an example RAID-4 layout with 6 data stripes in two 3+1 RAID sets (0, 1, 2 and 3, 4, 5) and two parity stripes (p, q), one for each RAID set. The data stripes 0, 1, 2 use parity stripe p, and data stripes 3, 4, 5 use parity stripe q. The 6 data stripes themselves are in a single component with a RAID-0 layout that make up the data mirror, and the two EC parity stripe are in their own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parities (Ap+Aq, Bp+Bq, Cp+Cq, Dp+Dq) in the parity stripe. It should be noted that this is only an example to illustrate the concept of RAID sets in a constrained space, and in a real system it is much more likely that this file would use a single EC 6+2 geometry to allow recovery from two concurrent failures.

	[[File:~~RAID_set~~.png]]		[[File:RAID_set_2x3+1.png]]

	RAID sets also represent isolated failure domains within a file, allowing the reconstruction of data independently and in parallel across each RAID set. In addition, for widely-striped files where the number of data stripes are allocated on most or all of the OSTs in the filesystem, it allows the data and parity stripes to be distributed across all of the OSTs in the filesystem since the only requirement to maintain redundancy is that the OSTs do not overlap within a single RAID set.		RAID sets also represent isolated failure domains within a file, allowing the reconstruction of data independently and in parallel across each RAID set. In addition, for widely-striped files where the number of data stripes are allocated on most or all of the OSTs in the filesystem, it allows the data and parity stripes to be distributed across all of the OSTs in the filesystem since the only requirement to maintain redundancy is that the OSTs do not overlap within a single RAID set.

Adilger: /* On-disk/wire structure changes for Erasure Coding layout component */ update image descriptions

2026-02-21T00:14:04Z

On-disk/wire structure changes for Erasure Coding layout component: update image descriptions

← Older revision		Revision as of 17:14, 20 February 2026
Line 57:		Line 57:
	The Lustre FLR-EC layout will implement redundancy on a per-file basis, with the data and parity being self-contained in the objects allocated to each file. This allows a great deal of flexibility in EC geometries, since they can be selected and tuned on a per-file, per-directory, and per-application basis. A file layout can be mapped conceptually as if each OST object is a "disk" in a traditional RAID device, where each file has a different set of "disks" (objects) that are wholly independent of the layout used by other files. The data stripes ("disks") will be arranged in a RAID-0 format as they are for traditional Lustre striped file layouts. The EC parity will be isolated to their own stripes ("disks"), which results in a [https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_4 RAID-4] style layout (dedicated parity "disks") rather than RAID-5 or RAID-6 where the parity is interleaved with the file data to balance load during parity reconstruction.		The Lustre FLR-EC layout will implement redundancy on a per-file basis, with the data and parity being self-contained in the objects allocated to each file. This allows a great deal of flexibility in EC geometries, since they can be selected and tuned on a per-file, per-directory, and per-application basis. A file layout can be mapped conceptually as if each OST object is a "disk" in a traditional RAID device, where each file has a different set of "disks" (objects) that are wholly independent of the layout used by other files. The data stripes ("disks") will be arranged in a RAID-0 format as they are for traditional Lustre striped file layouts. The EC parity will be isolated to their own stripes ("disks"), which results in a [https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_4 RAID-4] style layout (dedicated parity "disks") rather than RAID-5 or RAID-6 where the parity is interleaved with the file data to balance load during parity reconstruction.

	The following ~~image~~ shows a RAID-4 layout with 3 data stripes and 1 parity stripe. The 3 data stripes themselves are in a component with a RAID-0 layout that make up the data mirror, and the single EC parity stripe is in its own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST.		The following figure shows a RAID-4 layout with 3 data stripes (0, 1, 2) and 1 parity stripe (p). The 3 data stripes themselves are in a component with a RAID-0 layout that make up the data mirror, and the single EC parity stripe is in its own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parity (Ap, Bp, Cp, Dp) in the parity stripe.

	[[File:RAID_4.png]]		[[File:RAID_4.png]]
Line 71:		Line 71:

	For files where the stripe count exceeds the number of stripes in the EC geometry, the concept of a RAID set is introduced. This is necessary when files are widely striped (e.g. 128 data stripes) to allow the EC Geometry to remain reasonable for reconstruction. In such cases, the data stripes are split into sets of <code>k</code> stripes, each of which has its own set of <code>m</code> parity stripes. For example, in a 16-stripe file with <code>--ec 8+2</code> layout there will be 16 data stripes in the data mirror split into two 8+2 RAID sets, with two corresponding 2-stripe EC parity stripes (total 4 stripes) in the EC mirror. This allows the EC parity calculations to remain relatively efficient, while still allowing widely-striped files to have parity.		For files where the stripe count exceeds the number of stripes in the EC geometry, the concept of a RAID set is introduced. This is necessary when files are widely striped (e.g. 128 data stripes) to allow the EC Geometry to remain reasonable for reconstruction. In such cases, the data stripes are split into sets of <code>k</code> stripes, each of which has its own set of <code>m</code> parity stripes. For example, in a 16-stripe file with <code>--ec 8+2</code> layout there will be 16 data stripes in the data mirror split into two 8+2 RAID sets, with two corresponding 2-stripe EC parity stripes (total 4 stripes) in the EC mirror. This allows the EC parity calculations to remain relatively efficient, while still allowing widely-striped files to have parity.

			The following image shows an example RAID-4 layout with 6 data stripes in two 3+1 RAID sets (0, 1, 2 and 3, 4, 5) and two parity stripes (p, q), one for each RAID set. The data stripes 0, 1, 2 use parity stripe p, and data stripes 3, 4, 5 use parity stripe q. The 6 data stripes themselves are in a single component with a RAID-0 layout that make up the data mirror, and the two EC parity stripe are in their own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST. Data is written in round-robin "slices" (A, B, C, D) in <code>stripe_size</code> units across the stripes, with the corresponding EC parities (Ap+Aq, Bp+Bq, Cp+Cq, Dp+Dq) in the parity stripe. It should be noted that this is only an example to illustrate the concept of RAID sets in a constrained space, and in a real system it is much more likely that this file would use a single EC 6+2 geometry to allow recovery from two concurrent failures.

	[[File:RAID_set.png]]		[[File:RAID_set.png]]

Adilger: /* EC Geometry */ update description based on image

2026-02-20T23:34:49Z

EC Geometry: update description based on image

← Older revision		Revision as of 16:34, 20 February 2026
Line 56:		Line 56:
	==== EC Geometry====		==== EC Geometry====
	The Lustre FLR-EC layout will implement redundancy on a per-file basis, with the data and parity being self-contained in the objects allocated to each file. This allows a great deal of flexibility in EC geometries, since they can be selected and tuned on a per-file, per-directory, and per-application basis. A file layout can be mapped conceptually as if each OST object is a "disk" in a traditional RAID device, where each file has a different set of "disks" (objects) that are wholly independent of the layout used by other files. The data stripes ("disks") will be arranged in a RAID-0 format as they are for traditional Lustre striped file layouts. The EC parity will be isolated to their own stripes ("disks"), which results in a [https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_4 RAID-4] style layout (dedicated parity "disks") rather than RAID-5 or RAID-6 where the parity is interleaved with the file data to balance load during parity reconstruction.		The Lustre FLR-EC layout will implement redundancy on a per-file basis, with the data and parity being self-contained in the objects allocated to each file. This allows a great deal of flexibility in EC geometries, since they can be selected and tuned on a per-file, per-directory, and per-application basis. A file layout can be mapped conceptually as if each OST object is a "disk" in a traditional RAID device, where each file has a different set of "disks" (objects) that are wholly independent of the layout used by other files. The data stripes ("disks") will be arranged in a RAID-0 format as they are for traditional Lustre striped file layouts. The EC parity will be isolated to their own stripes ("disks"), which results in a [https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_4 RAID-4] style layout (dedicated parity "disks") rather than RAID-5 or RAID-6 where the parity is interleaved with the file data to balance load during parity reconstruction.

			The following image shows a RAID-4 layout with 3 data stripes and 1 parity stripe. The 3 data stripes themselves are in a component with a RAID-0 layout that make up the data mirror, and the single EC parity stripe is in its own component that makes up the EC mirror. Each stripe is an OST object that is allocated on a different OST.

	[[File:RAID_4.png]]		[[File:RAID_4.png]]

Adilger: /* On-disk/wire structure changes for Erasure Coding layout component */ use .png images

2026-02-20T23:18:41Z

On-disk/wire structure changes for Erasure Coding layout component: use .png images

← Older revision		Revision as of 16:18, 20 February 2026
Line 57:		Line 57:
	The Lustre FLR-EC layout will implement redundancy on a per-file basis, with the data and parity being self-contained in the objects allocated to each file. This allows a great deal of flexibility in EC geometries, since they can be selected and tuned on a per-file, per-directory, and per-application basis. A file layout can be mapped conceptually as if each OST object is a "disk" in a traditional RAID device, where each file has a different set of "disks" (objects) that are wholly independent of the layout used by other files. The data stripes ("disks") will be arranged in a RAID-0 format as they are for traditional Lustre striped file layouts. The EC parity will be isolated to their own stripes ("disks"), which results in a [https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_4 RAID-4] style layout (dedicated parity "disks") rather than RAID-5 or RAID-6 where the parity is interleaved with the file data to balance load during parity reconstruction.		The Lustre FLR-EC layout will implement redundancy on a per-file basis, with the data and parity being self-contained in the objects allocated to each file. This allows a great deal of flexibility in EC geometries, since they can be selected and tuned on a per-file, per-directory, and per-application basis. A file layout can be mapped conceptually as if each OST object is a "disk" in a traditional RAID device, where each file has a different set of "disks" (objects) that are wholly independent of the layout used by other files. The data stripes ("disks") will be arranged in a RAID-0 format as they are for traditional Lustre striped file layouts. The EC parity will be isolated to their own stripes ("disks"), which results in a [https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_4 RAID-4] style layout (dedicated parity "disks") rather than RAID-5 or RAID-6 where the parity is interleaved with the file data to balance load during parity reconstruction.

	[[File:RAID_4.~~svg~~]]		[[File:RAID_4.png]]

	A significant advantage of using the RAID-4 geometry (RAID-0 + fixed parity stripes) is that this allows adding EC parity to existing Lustre files, and maintaining read compatibility for older clients that do not understand FLR-EC layouts. The RAID-0 data mirror of the file will be written normally in the same manner it was before FLR-EC (potentially already years ago), and then the EC mirror can be created and written afterward to add redundancy to the file. The delay between data writes and EC parity writes can be managed as appropriate for the files. For example, short-lived checkpoint files may not need any EC overhead, but 1/6 or 1/8 or 1/24 of hourly checkpoints for a long-running application (1, 3, 4 times per day) may still want to have EC generated to minimize the loss of application progress in case of a significant failure of the storage.		A significant advantage of using the RAID-4 geometry (RAID-0 + fixed parity stripes) is that this allows adding EC parity to existing Lustre files, and maintaining read compatibility for older clients that do not understand FLR-EC layouts. The RAID-0 data mirror of the file will be written normally in the same manner it was before FLR-EC (potentially already years ago), and then the EC mirror can be created and written afterward to add redundancy to the file. The delay between data writes and EC parity writes can be managed as appropriate for the files. For example, short-lived checkpoint files may not need any EC overhead, but 1/6 or 1/8 or 1/24 of hourly checkpoints for a long-running application (1, 3, 4 times per day) may still want to have EC generated to minimize the loss of application progress in case of a significant failure of the storage.
Line 70:		Line 70:
	For files where the stripe count exceeds the number of stripes in the EC geometry, the concept of a RAID set is introduced. This is necessary when files are widely striped (e.g. 128 data stripes) to allow the EC Geometry to remain reasonable for reconstruction. In such cases, the data stripes are split into sets of <code>k</code> stripes, each of which has its own set of <code>m</code> parity stripes. For example, in a 16-stripe file with <code>--ec 8+2</code> layout there will be 16 data stripes in the data mirror split into two 8+2 RAID sets, with two corresponding 2-stripe EC parity stripes (total 4 stripes) in the EC mirror. This allows the EC parity calculations to remain relatively efficient, while still allowing widely-striped files to have parity.		For files where the stripe count exceeds the number of stripes in the EC geometry, the concept of a RAID set is introduced. This is necessary when files are widely striped (e.g. 128 data stripes) to allow the EC Geometry to remain reasonable for reconstruction. In such cases, the data stripes are split into sets of <code>k</code> stripes, each of which has its own set of <code>m</code> parity stripes. For example, in a 16-stripe file with <code>--ec 8+2</code> layout there will be 16 data stripes in the data mirror split into two 8+2 RAID sets, with two corresponding 2-stripe EC parity stripes (total 4 stripes) in the EC mirror. This allows the EC parity calculations to remain relatively efficient, while still allowing widely-striped files to have parity.

	[[File:RAID_set.~~svg~~]]		[[File:RAID_set.png]]

	RAID sets also represent isolated failure domains within a file, allowing the reconstruction of data independently and in parallel across each RAID set. In addition, for widely-striped files where the number of data stripes are allocated on most or all of the OSTs in the filesystem, it allows the data and parity stripes to be distributed across all of the OSTs in the filesystem since the only requirement to maintain redundancy is that the OSTs do not overlap within a single RAID set.		RAID sets also represent isolated failure domains within a file, allowing the reconstruction of data independently and in parallel across each RAID set. In addition, for widely-striped files where the number of data stripes are allocated on most or all of the OSTs in the filesystem, it allows the data and parity stripes to be distributed across all of the OSTs in the filesystem since the only requirement to maintain redundancy is that the OSTs do not overlap within a single RAID set.