Lustre Rsync Replication
Overview
lustre_rsync replicates a Lustre filesystem to another filesystem (Lustre or non-Lustre) using MDT changelogs. After an initial full sync, subsequent runs only process changes recorded since the last sync, making incremental replication efficient.
Prerequisites
- The source filesystem must have changelogs enabled on the MDT.
- The target filesystem must be mounted and writable.
- An initial full sync must be completed before registering the changelog user and starting changelog-based replication.
Step-by-Step Setup
1. Perform Initial Full Sync
This must be done before registering the changelog user. Use standard rsync to copy the entire source filesystem to the target:
rsync -av --xattrs /mnt/source_fs/ /mnt/target_fs/
WARNING: If you register the changelog user and start lustre_rsync without completing the initial sync first, the target will be missing files that existed before the changelog user was registered. The changelog only records changes after registration.
2. Register a Changelog User
Register a changelog user on the source MDT. This creates a consumer that tracks which changelog records have been processed:
lctl --device testfs-MDT0000 changelog_register
This returns a changelog user ID (e.g., cl1). Note this ID — it is required for all subsequent lustre_rsync invocations.
To verify:
lctl get_param mdd.testfs-MDT0000.changelog_users
3. First lustre_rsync Run
lustre_rsync --source=/mnt/source_fs \
--target=/mnt/target_fs \
--mdt=testfs-MDT0000 \
--user=cl1 \
--statuslog=/var/log/lustre_rsync.status \
--verbose
Key options:
--source— mount point of the source Lustre filesystem.--target— mount point of the target filesystem.--mdt— MDT device name (e.g.,testfs-MDT0000).--user— changelog user ID returned bychangelog_register.--statuslog— file to track sync progress. Required for incremental runs.--verbose— optional, prints progress.
4. Subsequent Runs
After the first run, only the --statuslog is required. lustre_rsync reads the status file to determine where it left off:
lustre_rsync --statuslog=/var/log/lustre_rsync.status
Run this periodically (e.g., via cron) to keep the target in sync.
Multiple Targets
lustre_rsync supports replicating to multiple targets simultaneously:
lustre_rsync --source=/mnt/source_fs \
--target=/mnt/target_fs1 \
--target=/mnt/target_fs2 \
--mdt=testfs-MDT0000 \
--user=cl1 \
--statuslog=/var/log/lustre_rsync.status
Each target receives the same set of changes.
Additional Options
| Option | Description |
|---|---|
--dry-run |
Show what would be done without making changes. |
--abort-on-err |
Stop on the first error instead of continuing. |
-D |
Debug mode — verbose logging for troubleshooting. |
--xattr=no |
Skip syncing extended attributes. |
DNE Considerations
For filesystems with multiple MDTs (DNE), you need one changelog user per MDT and one lustre_rsync process per MDT:
# Register on each MDT
lctl --device testfs-MDT0000 changelog_register # returns cl1
lctl --device testfs-MDT0001 changelog_register # returns cl1
# Run lustre_rsync for each MDT
lustre_rsync --source=/mnt/source_fs --target=/mnt/target_fs \
--mdt=testfs-MDT0000 --user=cl1 \
--statuslog=/var/log/lustre_rsync_mdt0.status &
lustre_rsync --source=/mnt/source_fs --target=/mnt/target_fs \
--mdt=testfs-MDT0001 --user=cl1 \
--statuslog=/var/log/lustre_rsync_mdt1.status &
Deregistering a Changelog User
When replication is no longer needed, deregister the changelog user to allow changelog records to be purged:
lctl --device testfs-MDT0000 changelog_deregister cl1
WARNING: Failing to deregister unused changelog users causes changelogs to accumulate indefinitely, consuming MDT space.