Backing Up a Lustre File System - Revision history

Adilger: better describe backup options

2022-06-23T18:55:31Z

better describe backup options

← Older revision		Revision as of 11:55, 23 June 2022
Line 1:		Line 1:
	~~{\| class='wikitable'~~
	\|-
	!Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date.
	\|}

	~~Lustre™ provides~~ backups at several levels. Generally, ~~file system~~-level backups are		Lustre allows backups at several levels. Generally, filesystem-level backups are recommended over device-level backups, in order to allow backing up and restoring individual files/directories on demand. However, it is also recommended that regular MDT device-level backups are also done (e.g. weekly), since restoring from a device-level MDT backup (at most a few TB) is '''much''' faster than restoring all of the data in a filesystem (which may be PB of data).
	recommended over device-level backups.

	== ~~File System~~-level Backups ==		== Lustre Filesystem-level Backups ==

	A ~~file system~~-level backup gives you full control over the files to back up and allows		A Lustre filesystem-level backup gives you full control over the files to back up and allows restoration of individual files as needed. Filesystem-level backups are also easier to integrate into existing backup solutions.
	restoration of individual files as needed. ~~File system~~-level backups are also easier to integrate into existing backup solutions.

	~~File system~~ backups are performed from a Lustre client (or many clients working		Lustre filesystem-level backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes, similar to backing up any other file system. However, due to the large size of most Lustre file systems, it is not always possible to complete a full backup in a timely manner. We recommend that you back up subsets of the file system, such as user or project sub-directories, filesets for a single user, and/or incrementally by date.
	parallel in different directories) rather than on individual server nodes, similar to backing up any other file system. However, due to the large size of most Lustre file systems, it is not always possible to complete a full backup. We recommend that you back up subsets of the file system, such as sub-directories ~~of the entire directory~~, filesets for a single user, or ~~files incremented~~ by date.

	For information about device-level ~~or file-level~~ backups or using the Linux Logical Volume Manager (LVM) snapshot feature to make incremental backups of a Lustre file system, see [https://doc.lustre.org/lustre_manual.xhtml#backup_device ~~Chapter 18:~~ ''Backing Up and Restoring a File System''] in the [https://doc.lustre.org/lustre_manual.xhtml Lustre Operations Manual].		== Device-level Image Backups ==

			Device-level image backups use <code>dd</code>, <code>e2image</code>, or <code>zfs send</code> to backup individual MDT or OST devices. These are mostly useful when replacing the hardware of a single target device. However, having MDT device-level backups can be very useful if there is severe corruption of the MDT device. Even if the backup is several days or a week old, restoring this backup would be preferable to total filesystem loss. Device-level image backups can be considerably faster than Target-level Filesystem backups since they can stream at full device speeds, rather than handling millions of individual files

			For information about device-level image backups or using the Linux Logical Volume Manager (LVM) snapshot feature to make incremental backups of a Lustre file system, see [https://doc.lustre.org/lustre_manual.xhtml#backup_device ''Backing Up and Restoring an MDT or OST (ldiskfs Device Level)''] in the [https://doc.lustre.org/lustre_manual.xhtml Lustre Operations Manual].

			== Target-level Filesystem Backups ==

			Target-level filesystem backups use <code>tar</code> or <code>rsync</code> to back up the files and (critically) xattrs from the underlying MDT or OST filesystem. This is useful when replacing the hardware of a single target device, but where the formatting parameters of the filesystem are being changed (e.g. ldiskfs inode ratio, filesystem features, etc). These will likely use less space than a Device-level Image backup, since they do not backup "empty space", but will likely take '''much longer''' to process due to handling millions or billions of individual files on the filesystem in a random order.

			For information about target-level filesystem backups, see [https://doc.lustre.org/lustre_manual.xhtml#backup_fs_level ''Backing Up an OST or MDT (Backend File System Level)''] in the [https://doc.lustre.org/lustre_manual.xhtml Lustre Operations Manual].

Adilger: /* File System-level Backups */ link to correct manual page

2022-06-23T18:33:28Z

File System-level Backups: link to correct manual page

← Older revision		Revision as of 11:33, 23 June 2022
Line 15:		Line 15:
	parallel in different directories) rather than on individual server nodes, similar to backing up any other file system. However, due to the large size of most Lustre file systems, it is not always possible to complete a full backup. We recommend that you back up subsets of the file system, such as sub-directories of the entire directory, filesets for a single user, or files incremented by date.		parallel in different directories) rather than on individual server nodes, similar to backing up any other file system. However, due to the large size of most Lustre file systems, it is not always possible to complete a full backup. We recommend that you back up subsets of the file system, such as sub-directories of the entire directory, filesets for a single user, or files incremented by date.

	For information about device-level or file-level backups or using the Linux Logical Volume Manager (LVM) snapshot feature to make incremental backups of a Lustre file system, see [~~http~~://~~wiki~~.lustre.org/~~manual/LustreManual20_HTML/BackupAndRestore~~.~~html~~#~~50438207_pgfId-5529~~ Chapter 15: ''Backing Up and Restoring a File System''] in the [~~http~~://~~wiki~~.lustre.org/~~manual/LustreManual20_HTML/index~~.~~html~~ Lustre Operations Manual].		For information about device-level or file-level backups or using the Linux Logical Volume Manager (LVM) snapshot feature to make incremental backups of a Lustre file system, see [https://doc.lustre.org/lustre_manual.xhtml#backup_device Chapter 18: ''Backing Up and Restoring a File System''] in the [https://doc.lustre.org/lustre_manual.xhtml Lustre Operations Manual].

KenRawlings: Initial Creation From Migration

2016-06-02T14:52:27Z

Initial Creation From Migration

New page

{| class='wikitable'
|-
!Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date.
|}

Lustre™ provides backups at several levels. Generally, file system-level backups are
recommended over device-level backups.

== File System-level Backups ==

A file system-level backup gives you full control over the files to back up and allows
restoration of individual files as needed. File system-level backups are also easier to integrate into existing backup solutions.

File system backups are performed from a Lustre client (or many clients working
parallel in different directories) rather than on individual server nodes, similar to backing up any other file system. However, due to the large size of most Lustre file systems, it is not always possible to complete a full backup. We recommend that you back up subsets of the file system, such as sub-directories of the entire directory, filesets for a single user, or files incremented by date.

For information about device-level or file-level backups or using the Linux Logical Volume Manager (LVM) snapshot feature to make incremental backups of a Lustre file system, see [http://wiki.lustre.org/manual/LustreManual20_HTML/BackupAndRestore.html#50438207_pgfId-5529 Chapter 15: ''Backing Up and Restoring a File System''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html Lustre Operations Manual].