ZFS Snapshots for MDT backup

Backing up your MDT using zfs snapshots

ZFS snapshots are an easy and lightweight way to backup or migrate your MDT(s) (and OSTs!) to different servers. The process is fairly straight forward:

1. Take a snapshot of the MDT dataset
2. Send the snapshot to another pool/system making sure to use the -p switch on the zfs send command.

example of a resume-able snapshot send to a remote system using ssh:

(You will likely want to use the -s switch on your recv command to support resuming the send if it gets interrupted for some reason.)

zfs snap mdt-pool-000/mdt-0000@backup-20251022
zfs send -vp mdt-pool-000/mdt-0000@backup-20251022 | ssh [email protected] 'zfs recv -s backup-pool/mdt-0000'

That's it. You may notice your MDT consuming more (or less) space depending on the pool geometry of the pool you've sent it to.

Incremental backups

An extremely powerful feature of zfs snapshots and sends is that you can do incremental sends between two snapshots provided the earlier snapshot also exists on the receiving system.

# it is assumed that the earlier snapshot (backup-20251022 here) exists on the receiving side
zfs snap mdt-pool-000/mdt-0000@backup-20251023
zfs send -vp -i mdt-pool-000/mdt-0000@backup-20251022 mdt-pool-000/mdt-0000@backup-20251023 | ssh [email protected] 'zfs recv -s backup-pool/mdt-0000'

this should send the changes to the dataset from when the first snapshot was taken to the second.

This can make the migration to new hardware particularly painless as you can do the initial sync while the filesystem is active and then take a shorter downtime to do the final snapshot and send.

Restoring your MDT from a backup snapshot

Assuming you are restoring to a machine that will have the same nid(s) as the machine the mdt came from the process should be very straight forward:

1. have or create a pool on the new host that is configured as desired
2. send snapshot from backup pool to production pool making sure you use the -p switch for the send command.
3. start the mdt (if desired)

example of a resume-able snapshot send from a remote system using ssh:

# this assumes that the pool mdt-pool-000 exists on the system and that there is not presently a dataset called mdt-0000 on it
ssh [email protected] 'zfs send -p backup-pool/mdt-0000@backup-20251022' | zfs recv -vs mdt-pool-000/mdt-0000

If you are restoring to a machine that has a different nid from the machine the MDT was originally on you'll likely need to follow the process for changing server nids in the lustre manual.

Note about backups and restoration

Any restoration of a MDT from a snapshot is going to be of the MDT at the point in time the snapshot was created. It is highly likely that unless you also took snapshots of your OSTs at the exact same time as the MDT(s) and restored everything at the same time that the MDT(s) and OSTs will be inconsistent. Unless the filesystem was offline when the snapshots were taken you will likely have orphaned or missing objects on the OSTs. You should run a lfsck after any restoration.

While it may not meet the definition of a backup, just taking snapshots (but not sending them anywhere) on your MDT(s) (and OSTs) can give you a mechanism for rolling back the filesystem should you need to, like due to difficulties during an upgrade for example.