Elliswilson: Created page with "== Overview == `lustre_rsync` replicates a Lustre filesystem to another filesystem (Lustre or non-Lustre) using MDT changelogs. After an initial full sync, subsequent runs only process changes recorded since the last sync, making incremental replication efficient. == Prerequisites == * The source filesystem must have changelogs enabled on the MDT. * The target filesystem must be mounted and writable. * An initial full sync must be completed '''before''' reg..."

2026-07-20T15:28:33Z

Created page with "== Overview == <code>lustre_rsync</code> replicates a Lustre filesystem to another filesystem (Lustre or non-Lustre) using MDT changelogs. After an initial full sync, subsequent runs only process changes recorded since the last sync, making incremental replication efficient. == Prerequisites == * The source filesystem must have changelogs enabled on the MDT. * The target filesystem must be mounted and writable. * An initial full sync must be completed '''before''' reg..."

New page

== Overview ==

<code>lustre_rsync</code> replicates a Lustre filesystem to another filesystem (Lustre or non-Lustre) using MDT changelogs. After an initial full sync, subsequent runs only process changes recorded since the last sync, making incremental replication efficient.

== Prerequisites ==

* The source filesystem must have changelogs enabled on the MDT.
* The target filesystem must be mounted and writable.
* An initial full sync must be completed '''before''' registering the changelog user and starting changelog-based replication.

== Step-by-Step Setup ==

=== 1. Perform Initial Full Sync ===

'''This must be done before registering the changelog user.''' Use standard <code>rsync</code> to copy the entire source filesystem to the target:

<pre>
rsync -av --xattrs /mnt/source_fs/ /mnt/target_fs/
</pre>

{{Warning|If you register the changelog user and start <code>lustre_rsync</code> without completing the initial sync first, the target will be missing files that existed before the changelog user was registered. The changelog only records changes ''after'' registration.}}

=== 2. Register a Changelog User ===

Register a changelog user on the source MDT. This creates a consumer that tracks which changelog records have been processed:

<pre>
lctl --device testfs-MDT0000 changelog_register
</pre>

This returns a changelog user ID (e.g., <code>cl1</code>). Note this ID — it is required for all subsequent <code>lustre_rsync</code> invocations.

To verify:

<pre>
lctl get_param mdd.testfs-MDT0000.changelog_users
</pre>

=== 3. First lustre_rsync Run ===

<pre>
lustre_rsync --source=/mnt/source_fs \
--target=/mnt/target_fs \
--mdt=testfs-MDT0000 \
--user=cl1 \
--statuslog=/var/log/lustre_rsync.status \
--verbose
</pre>

Key options:
* <code>--source</code> — mount point of the source Lustre filesystem.
* <code>--target</code> — mount point of the target filesystem.
* <code>--mdt</code> — MDT device name (e.g., <code>testfs-MDT0000</code>).
* <code>--user</code> — changelog user ID returned by <code>changelog_register</code>.
* <code>--statuslog</code> — file to track sync progress. Required for incremental runs.
* <code>--verbose</code> — optional, prints progress.

=== 4. Subsequent Runs ===

After the first run, only the <code>--statuslog</code> is required. <code>lustre_rsync</code> reads the status file to determine where it left off:

<pre>
lustre_rsync --statuslog=/var/log/lustre_rsync.status
</pre>

Run this periodically (e.g., via cron) to keep the target in sync.

== Multiple Targets ==

<code>lustre_rsync</code> supports replicating to multiple targets simultaneously:

<pre>
lustre_rsync --source=/mnt/source_fs \
--target=/mnt/target_fs1 \
--target=/mnt/target_fs2 \
--mdt=testfs-MDT0000 \
--user=cl1 \
--statuslog=/var/log/lustre_rsync.status
</pre>

Each target receives the same set of changes.

== Additional Options ==

{| class="wikitable"
|-
! Option !! Description
|-
| <code>--dry-run</code> || Show what would be done without making changes.
|-
| <code>--abort-on-err</code> || Stop on the first error instead of continuing.
|-
| <code>-D</code> || Debug mode — verbose logging for troubleshooting.
|-
| <code>--xattr=no</code> || Skip syncing extended attributes.
|}

== DNE Considerations ==

For filesystems with multiple MDTs (DNE), you need one changelog user per MDT and one <code>lustre_rsync</code> process per MDT:

<pre>
# Register on each MDT
lctl --device testfs-MDT0000 changelog_register # returns cl1
lctl --device testfs-MDT0001 changelog_register # returns cl1

# Run lustre_rsync for each MDT
lustre_rsync --source=/mnt/source_fs --target=/mnt/target_fs \
--mdt=testfs-MDT0000 --user=cl1 \
--statuslog=/var/log/lustre_rsync_mdt0.status &

lustre_rsync --source=/mnt/source_fs --target=/mnt/target_fs \
--mdt=testfs-MDT0001 --user=cl1 \
--statuslog=/var/log/lustre_rsync_mdt1.status &
</pre>

== Deregistering a Changelog User ==

When replication is no longer needed, deregister the changelog user to allow changelog records to be purged:

<pre>
lctl --device testfs-MDT0000 changelog_deregister cl1
</pre>

{{Warning|Failing to deregister unused changelog users causes changelogs to accumulate indefinitely, consuming MDT space.}}

== See Also ==

* [[Lustre Backup Strategies]]
* [[Backing Up a Lustre File System]]

[[Category:Lustre Systems Administration]]
[[Category:Lustre Backup and Recovery]]

Lustre Rsync Replication - Revision history