File Level Replication High Level Design - Revision history

Adilger: Move to common Architecture category

2025-09-25T19:21:23Z

Move to common Architecture category

← Older revision		Revision as of 12:21, 25 September 2025
Line 359:		Line 359:
	\|}		\|}

	[[Category:~~Design~~]]		[[Category:Architecture]]
	[[Category:FLR]]		[[Category:FLR]]
	[[Category:Layout]]		[[Category:Layout]]

Adilger: /* Feature List */ add FLR category

2018-06-19T08:03:28Z

Feature List: add FLR category

← Older revision		Revision as of 01:03, 19 June 2018
Line 360:		Line 360:

	[[Category:Design]]		[[Category:Design]]
			[[Category:FLR]]
	[[Category:Layout]]		[[Category:Layout]]

Adilger: fix URL description

2018-06-04T07:25:59Z

fix URL description

← Older revision		Revision as of 00:25, 4 June 2018
Line 1:		Line 1:

	~~The latest version of this file is located at:~~		This page was originally imported from [http://wiki.lustre.org/images/7/79/Opensfs-File-LevelReplicationHighLevelDesign-190118-0121-1606.pdf FLR High Level Design].
	http://wiki.lustre.org/images/7/79/Opensfs-File-LevelReplicationHighLevelDesign-190118-0121-1606.pdf

	== Introduction ==		== Introduction ==

Jxiong: /* Replicated File State */

2018-02-06T04:45:34Z

Replicated File State

← Older revision		Revision as of 21:45, 5 February 2018
Line 153:		Line 153:
	Replicated files transit their state when they are being written and resynchronized. The diagram below shows the FSM to transit the states.		Replicated files transit their state when they are being written and resynchronized. The diagram below shows the FSM to transit the states.

	.~~.. need to figure out how to insert diagram ...~~		[[File:Replication_fsm.png\|631x319px]]

	When the first write comes, and the replicated file is in READ_ONLY state, it will increase the layout generation and pick the primary replica, then change the file state to WRITE_PENDING and mark non-primary replicas STALE. The writing client will then uptodate primary replica’s layout generation, and then notify the MDT to set the replicated file state to WRITABLE. After the replicated file is resynchronized, the file state will go back to READ_ONLY again.		When the first write comes, and the replicated file is in READ_ONLY state, it will increase the layout generation and pick the primary replica, then change the file state to WRITE_PENDING and mark non-primary replicas STALE. The writing client will then uptodate primary replica’s layout generation, and then notify the MDT to set the replicated file state to WRITABLE. After the replicated file is resynchronized, the file state will go back to READ_ONLY again.
Line 166:		Line 166:

	When the replicated file is in READ_ONLY state, all but one of the replicas must be in SYNC state. If it is in WRITE_PENDING or WRITABLE state, only primary replica is in SYNC state, other replicas must be in STALE or OFFLINE state.		When the replicated file is in READ_ONLY state, all but one of the replicas must be in SYNC state. If it is in WRITE_PENDING or WRITABLE state, only primary replica is in SYNC state, other replicas must be in STALE or OFFLINE state.

	==== Wrap it up ====		==== Wrap it up ====

Jxiong: /* IO framework for read */

2018-02-06T04:44:21Z

IO framework for read

← Older revision		Revision as of 21:44, 5 February 2018
Line 101:		Line 101:

	When reading from replica, the IO RPC should be issued with rq_no_delay bit set in ptlrpc_request so that if the object becomes unreachable, the RPC won’t be stuck in the resending list at the RPC layer, but instead it will return with the error code -ENAVAIL. The IO framework should reissue the IO and invoke the replica selection policy to try the next replica.		When reading from replica, the IO RPC should be issued with rq_no_delay bit set in ptlrpc_request so that if the object becomes unreachable, the RPC won’t be stuck in the resending list at the RPC layer, but instead it will return with the error code -ENAVAIL. The IO framework should reissue the IO and invoke the replica selection policy to try the next replica.

			[[File:Read-RPC.png\|300x600px]]

	The diagram above depicts a typical process of sync read. For sync read, the reading process has to wait for the RPC result, therefore if it sees the error -ENAVAIL that indicates the replica is unavailable, it will set cl_io::ci_restart so that the IO can be restarted.		The diagram above depicts a typical process of sync read. For sync read, the reading process has to wait for the RPC result, therefore if it sees the error -ENAVAIL that indicates the replica is unavailable, it will set cl_io::ci_restart so that the IO can be restarted.

	=== Read ahead ===		=== Read ahead ===

Jxiong: /* Resynchronization */

2018-02-06T04:42:38Z

Resynchronization

← Older revision		Revision as of 21:42, 5 February 2018
Line 244:		Line 244:
	The client that is resynchronizing the replicated file is not required to be the same client that wrote that file. Actually, some dedicated clients, named agent clients, could be used to pick files that have elapsed at least quiescent time since the last write. A simple ChangeLog monitor will be provided a list of files (FIDs) that have been marked stale, and can monitor the open state on the MDT. It would also be possible to integrate this resync operation with a policy engine like RobinHood to prioritize if and when to resynchronize the file, or if the replica(s) should be moved other OSTs if the existing ones are unavailable for a long time. This potential integration with RobinHood is out of scope of Phase 1 design and implementation, but a simple ChangeLog monitoring tool will be provided.		The client that is resynchronizing the replicated file is not required to be the same client that wrote that file. Actually, some dedicated clients, named agent clients, could be used to pick files that have elapsed at least quiescent time since the last write. A simple ChangeLog monitor will be provided a list of files (FIDs) that have been marked stale, and can monitor the open state on the MDT. It would also be possible to integrate this resync operation with a policy engine like RobinHood to prioritize if and when to resynchronize the file, or if the replica(s) should be moved other OSTs if the existing ones are unavailable for a long time. This potential integration with RobinHood is out of scope of Phase 1 design and implementation, but a simple ChangeLog monitoring tool will be provided.

	Of course, the file can be resynchronized manually with command lfs ~~layout --~~resync.		Of course, the file can be resynchronized manually with command lfs mirror resync.

			[[File:Replication_resync.png\|682x600px]]

	The flow chart above depicts the process of resynchronization.		The flow chart above depicts the process of resynchronization.
Line 269:		Line 271:

	The replicas that have been successfully synchronized are recorded in cd_nr_replicas and cd_replica_ids field. MDT will clear STALE bit on those synchronized replicas.		The replicas that have been successfully synchronized are recorded in cd_nr_replicas and cd_replica_ids field. MDT will clear STALE bit on those synchronized replicas.

	=== Client Eviction ===		=== Client Eviction ===

Jxiong: /* Wrap it up */

2018-02-06T04:40:58Z

Wrap it up

← Older revision		Revision as of 21:40, 5 February 2018
Line 167:		Line 167:
	The diagram below shows the process of how a writing clients interacts the MDT before writing a replicated file.		The diagram below shows the process of how a writing clients interacts the MDT before writing a replicated file.

	''.~~.. need to figure out how to insert diagram ...''~~		[[File:Replication_write.png\|548x600px]]

	The client sends an RPC called MDT_INTENT_WRITE to the MDT before it writes replicated files. When the MDT receives the MDT_INTENT_WRITE RPC request, and if it turns out that layout has to be changed, it will update the layout synchronously. The reason to update the layout synchronously is that in case the MDT crashes before the update is committed, and if the writing client failed to replay the RPC, it may render corruption to the replicated files.		The client sends an RPC called MDT_INTENT_WRITE to the MDT before it writes replicated files. When the MDT receives the MDT_INTENT_WRITE RPC request, and if it turns out that layout has to be changed, it will update the layout synchronously. The reason to update the layout synchronously is that in case the MDT crashes before the update is committed, and if the writing client failed to replay the RPC, it may render corruption to the replicated files.

Jxiong: /* Layout format */

2018-02-06T04:25:17Z

Layout format

← Older revision		Revision as of 21:25, 5 February 2018
Line 21:		Line 21:
	=== Layout format ===		=== Layout format ===
	Layout format of redundancy is based on the [[Layout Enhancement Solution Architecture]] where composite layout format is defined. Each replica has a unique ID. The state of each replica is maintained separately. The replica state can be modified via layout operation interfaces.		Layout format of redundancy is based on the [[Layout Enhancement Solution Architecture]] where composite layout format is defined. Each replica has a unique ID. The state of each replica is maintained separately. The replica state can be modified via layout operation interfaces.

			[[File:Replication_layout.png\|800x401px]]

	=== RAID-0+1 and RAID-1+0 layout ===		=== RAID-0+1 and RAID-1+0 layout ===

Jxiong: /* Split a specific replica index from a mirrored file */

2018-01-19T01:37:04Z

Split a specific replica index from a mirrored file

← Older revision		Revision as of 18:37, 18 January 2018
Line 43:		Line 43:

	=== Split a specific replica index from a mirrored file ===		=== Split a specific replica index from a mirrored file ===
	<code>lfs ~~layout~~ split <--mirror-id=id> [--destroy\|-d\|-f<new_file>] <file_name></code>		<code>lfs mirror split <--mirror-id=id> [--destroy\|-d\|-f<new_file>] <file_name></code>

	This command will split the specified replica by its id indicated by '''--mirror-id'''. '''file_name''' must be a mirrored file otherwise the command will be returned with -ENOTSUPP. If '''-f new_file''' is specified, a file name '''new_file''' will be created with the layout of the split replica; otherwise the layout of the specified replica will be destroyed.		This command will split the specified replica by its id indicated by '''--mirror-id'''. '''file_name''' must be a mirrored file otherwise the command will be returned with -ENOTSUPP. If '''-f new_file''' is specified, a file name '''new_file''' will be created with the layout of the split replica; otherwise the layout of the specified replica will be destroyed.

Jxiong at 01:36, 19 January 2018

2018-01-19T01:36:41Z

← Older revision		Revision as of 18:36, 18 January 2018
Line 27:		Line 27:

	== Command line tools ==		== Command line tools ==
	~~=== Create RAID-1 file by parameters with lfs setstripe ===~~

	To create RAID-1 file, we have to make sure that different replicas must be on different OSTs, even OSSes and racks. An understanding of cluster topology is necessary to achieve this aim. In the initial implementation the use of existing OST pools mechanism will allow separating OSTs by any arbitrary criteria: i.e. fault domain.

			=== Create mirrored file by parameters similar to lfs setstripe ===
			To create a mirrored file, we have to make sure that different replicas must be on different OSTs, even OSSes and racks. An understanding of cluster topology is necessary to achieve this aim. In the initial implementation the use of existing OST pools mechanism will allow separating OSTs by any arbitrary criteria: i.e. fault domain.
	In practice, we can take advantage of OST pools by grouping OSTs by topological information. Therefore, when creating a replicated file, the users can indicate which OST pool can be used by replicas.		In practice, we can take advantage of OST pools by grouping OSTs by topological information. Therefore, when creating a replicated file, the users can indicate which OST pool can be used by replicas.

	~~lfs setstripe --layout mirror [--component idx] [--preferred] [other setstripe options]]~~ <~~file_name~~>		<code>lfs mirror create <-N[mirror_count]> [--flags=prefer] [other setstripe options] <file_name\|directory></code>

	The above command will create an empty RAID-1 file. The --component option indicates the replica component number that is being specified, and can be repeated multiple times so that multiple replicas can be created at one time. The --prefer option marks that replica as being preferred for writes (e.g. stored on fast/local OSTs). As mentioned above, it is recommended to use the --pool option (one of the lfs setstripe options) with OST pools configured with independent fault domains to ensure different replicas will be placed on different OSTs and/or servers and/or racks, therefore availability and performance can be improved. If the setstripe options are not specified, it is possible to create ~~replicas with objects on the same OST(s) in the Phase 1 implementation, which would remove most of the benefit of using redundancy in that case.~~

	~~=== Extend an existing file to RAID-1 format ===~~

	~~lfs setstripe~~ -~~-layout mirror --extend~~ [~~--read-only\|-r~~] [--prefer] [other setstripe options] <~~raid1_file~~>

	~~This~~ command will ~~append a replica by stripe options into~~ an ~~existing~~ file ~~raid1_file~~. The ~~existing file~~ can ~~already~~ be ~~a replicated file~~, or ~~just a normal~~ file.		The above command will create an empty mirrored file. The -N option indicates the number of replicas should be created with the coming setstripe options. The --flags option sets flags to the replicas that are going to be create (e.g. prefer flag can be used if the mirror is on fast/local OSTs). As mentioned above, it is recommended to use the --pool option (one of the lfs setstripe options) with OST pools configured with independent fault domains to ensure different replicas will be placed on different OSTs and/or servers and/or racks, therefore availability and performance can be improved. If the setstripe options are not specified, it is possible to create replicas with objects on the same OST(s) in the Phase 1 implementation, which would remove most of the benefit of using replication in that case.
			=== Extend an existing file to mirrored format ===
			<code>lfs mirror extend [--no-verify] <-N[mirror_count]> [other setstripe options\|-f <victim_file>] <file_name></code>

	This command will create new volatile file with any optional setstripe options that are specified, or using the defaults inherited from the parent directory or filesystem. The file contents will be copied from the existing file to volatile file and then the layout will be modified to add the volatile file as a new replica. Lease will be held on the existing file to prevent it from being modified during the copy. If the existing file is modified during this operation, -EBUSY will be returned to the ~~resync~~ tool. Administrators can repeat this command later.		This command will append a replica indicated by setstripe options or just take the layout from existing file victim_file into the file file_name. The file_name must be an existing file but it can be a mirrored or normal file.
			This command will create new volatile file with any optional setstripe options that are specified, or using the defaults inherited from the parent directory or filesystem. The file contents will be copied from the existing file to volatile file and then the layout will be modified to add the volatile file as a new replica. Lease will be held on the existing file to prevent it from being modified during the copy. If the existing file is modified during this operation, -EBUSY will be returned to the replication tool. Administrators can repeat this command later.
			The difference between creating a new mirrored file and extending an existing file is that extending the file will make sure the file content has been copied into newly created replica(s). If victim_file is used, the utility will make sure the content of victim_file and file_name have the same content. However, the content verification can be skipped if --no-verify is specified.

	~~The~~ --~~read~~-~~only~~\|-r option can be specified if the file is known to be accessed by older clients that do not directly support File Level Redundancy, which will prevent them from marking the file stale if the file is opened O_RDWR (in particular by Fortran programs), but will prevent the file from being modified. This is similar to marking the file immutable, but it can still be deleted or have attributes such as timestamps and ownership modified.		=== Split a specific replica index from a mirrored file ===
			<code>lfs layout split <--mirror-id=id> [--destroy\|-d\|-f<new_file>] <file_name></code>

	~~=== Split a specific replica index from a RAID-1 file ===~~		This command will split the specified replica by its id indicated by '''--mirror-id'''. '''file_name''' must be a mirrored file otherwise the command will be returned with -ENOTSUPP. If '''-f new_file''' is specified, a file name '''new_file''' will be created with the layout of the split replica; otherwise the layout of the specified replica will be destroyed.

	~~lfs layout --split --component idx <raid1_file> [target_file]~~

	This command will split the specified replica by its ~~component index idx~~. ~~raid1_file~~ must be a ~~RAID-1~~ file otherwise the command will be returned with -ENOTSUPP. If ~~target_file~~ is specified, ~~the~~ file will be created with the layout of the split replica; otherwise the layout of the specified replica will be destroyed.

	=== Replica resynchronize ===		=== Replica resynchronize ===
			<code>lfs mirror resync <file_name></code>

	~~lfs layout --resync <raid1_file>~~		This command is used to resynchronize an out-of-sync mirrored file. If there are no any stale replicas for the replicated file, this command does nothing. Otherwise, this command will first hold the exclusive lease of raid1_file, and then the file will be resynchronized.

	This command is used to resynchronize an out-of-sync ~~RAID-1~~ file. If there are no any stale replicas for the replicated file, this command does nothing. Otherwise, this command will first hold the exclusive lease of raid1_file, and then the file will be resynchronized.

	In terms of resynchronization, the file will be read with normal read, because read must only use uptodate replicas; and then the read contents will be written to stale replicas with a dedicated replica write interface. We’re going to use direct IO for the read and write in this case. After the replicas are synchronized, this command will change the layout to mark the replicas as uptodate.		In terms of resynchronization, the file will be read with normal read, because read must only use uptodate replicas; and then the read contents will be written to stale replicas with a dedicated replica write interface. We’re going to use direct IO for the read and write in this case. After the replicas are synchronized, this command will change the layout to mark the replicas as uptodate.

	=== Verify replicated files ===		=== Verify replicated files ===
			<code>lfs mirror verify <file_name></code>

	~~lfs layout --~~verify ~~<raid1_file>~~		This command is used to verify that each replica of a replicated file contains the exactly same data. This command will pull data from each replica, and then do comparison or calculate checksum, so that we can get some help from hardware, to make sure they’re the same.

	This command is used to verify that each replica of a replicated file contains the exactly same data. This command will pull data from each replica, and then do comparison or calculate checksum, so that we can get some help from hardware, to make sure they’re the same.

	== Read from replicated file ==		== Read from replicated file ==