Immediate Write Mirror - High Level Design

Migration draft. Copied from the committed local design in paf0186/lustre_design_docs (projects/immediate-write-mirroring/design/immediate_write_mirroring_design.md; source commit feb9fd325f72f2b2f1df12e7355586514822a739), with diagrams committed in 21b52e0. The original Confluence page is 373565003 on wiki.whamcloud.com. Historical discussions and server revisions are archived separately; they are not recreated as native wiki history here. Source-code references are given as plain-text paths and line numbers at Lustre commit 47638add78158a74441ed3e56392f5a720cef764 (Linux references at v6.12) rather than links.

Overview & Design Summary

FLR Today

In the current implementation of FLR (introduced in Lustre 2.11), the system uses a "delayed write" approach for maintaining file mirrors. This means:

Writing to Mirrored Files: When a client writes to a mirrored file, only one primary (preferred) mirror is updated directly during the write operation. The other mirrors are simply marked as "stale" to indicate they're out of sync with the primary mirror.

Today, we have manual synchronization: after a write, the lfs mirror resync command must be run to synchronize the stale mirrors with the primary mirror. This command copies data from the synced mirror to the stale mirrors and removes the stale flag from successfully copied mirrors.

Layout state and staleness are managed through a careful series of layout state changes; see the file-level replication state machine in Core Design Concept in this document.

This delayed write approach was implemented in the first phase of FLR to avoid the complexity of maintaining consistency across multiple mirrors during concurrent writes. By updating only one mirror during writes and marking others as stale, the system maintains a consistent view of the file data, at the cost of requiring explicit synchronization after writes complete.

The current implementation does not provide immediate redundancy, since only a single mirror is up to date until an explicit resync operation is performed. This approach enables things like hot pools synchronization, but the lack of immediate write redundancy severely limits the use cases.

Immediate Mirroring

We have a requirement to do immediate mirroring on Lustre files, where all writes (and related ops) are replicated to multiple mirrors immediately. The infrastructure created for this will also be used for immediate erasure coding.

Whether a mirror takes part in immediate write mirroring is controlled by a per-mirror IMMEDIATE flag in the composite layout (lcme_flags), alongside existing flags like STALE and PREFER. Mirrors without the flag are ordinary FLR mirrors, written lazily via resync. Immediate and non-immediate mirrors can coexist on the same file.

The participants of a write epoch are the IMMEDIATE mirrors that are not STALE when the epoch opens, primary included. Writes and other modifying operations go to every participant. Non-immediate mirrors are not written; when a modifying operation omits a clean non-immediate mirror, its affected components are marked STALE under ordinary FLR rules. Changing which mirrors are IMMEDIATE requires an epoch boundary.

lfs mirror create --immediate marks every mirror the command creates IMMEDIATE, on all of its components; explicit per-mirror component flags allow mixed immediate/non-immediate layouts. Existing files change membership only through the conditional promotion and demotion operations in §4.2. The shorthand implies no minimum number of copies (implementation: lustre-design-docs-iwm.2.3).

The goal is to have redundancy immediately, but during writes only a single mirror (the primary) is available for reads — see Primary Mirror Selection and Read Visibility for the rationale.

Compatibility is capability-based. Clients we can modify (2.18/master and 2.15) treat unknown mirror-state flags as STALE for mirror selection and advertise this with a separate connect capability, so they skip INFLIGHT mirrors even without full IWM support. The MDT refuses to open files with IMMEDIATE mirrors for clients without that capability, whether or not an epoch is active. Capability-bearing clients without full IWM support can read the primary; their writes are denied by default, or, with an optional setting, use ordinary FLR write intents that mark every non-primary mirror STALE and write the primary only. See Old-Client Compatibility.

V1 scope. Data-on-MDT is excluded: layouts combining DoM with IMMEDIATE mirrors are rejected, including on promotion; ordinary DoM is unchanged. NFS re-export, PCC and HSM are supported under the normal layout/AW admission and drain rules, and every IO and lifecycle path they use must be covered (lustre-design-docs-od5.49.14; mmap in lustre-design-docs-od5.36).

The core idea of the design is this:

To write an IWM file, the client first obtains a valid layout, then takes or references a cached Active Writer (AW) lock at the top of the modifying IO, before it can dirty data or obtain or use PW write authority. AW is a CW lock on a separate IBITS bit (ACTIVE_WRITERS) on the same per-file resource. The first AW request opens an epoch: the MDT selects the primary by the usual policy, moves the file to WRITE_PENDING, sets INFLIGHT on the other participants, and makes the active-epoch record durable before write authority becomes usable. Later AW requests join the open epoch. The layout hold is soft and can be revoked without waiting for a flush. The server always takes LAYOUT before ACTIVE_WRITERS, with EX on both, to close an epoch or for any operation needing both; ordinary layout changes within an epoch (component instantiation, SEL extension) revoke LAYOUT alone and leave AW untouched. See AW Lock vs Layout Lock Interaction.

The client sends all writes to all participating mirrors in parallel. A single primary mirror will be selected for reads during this time, since we cannot guarantee all mirrors are identical during writes. This mirror is the "write leader", also used for write ordering (extent locks are taken on this mirror only). All other mirrors are marked INFLIGHT during writes — see Mirror States for the full flag model.

Clients hold AW until their admitted modifying IO has drained, the resulting dirty data has been flushed, and each participant’s updates are durable on OST storage or recorded as failed, with outstanding IO safely resolved. Per-mirror errors are reported to the MDS in the AW cancellation. On error, or for administrative operations, the MDS takes layout EX then AW EX, which forces writers through this release path; it then clears INFLIGHT from clean mirrors and sets STALE on failed ones. Secondary IO has three failure policies (wait indefinitely, finite slow-fail, finite fast-fail) that differ only in timing; administrative abandonment of a dead OST is a separate operation. See Secondary Failure Policy, Write Operation Flow and Active Writer Lock.

Write ordering is enforced by requiring client-side LDLM extent locks on the primary mirror. This means DIO to IWM files must use client-side locks where today it operates locklessly — a performance regression for shared-file DIO workloads. Restoring lockless DIO is a high priority for future work; see Write Ordering Alternatives. Hybrid IO, which turns large buffered IO into DIO, is DIO for this purpose and takes the same client lock. Group locks are allowed unchanged: group-lock holders already get no write ordering between them, and with IWM that includes possible differences between mirrors for overlapping writes, which are not detected or marked STALE.

Client eviction, MDS failover, write leader failure, and other recovery scenarios are covered in Failure Handling.

Relationship to Erasure Coding

Immediate mirroring is separate from FLR Erasure Coding (Confluence page 105093375) (read-only EC), and can be done partly in parallel. IWM will be the foundation of immediate erasure coding — the write duplication infrastructure, Active Writer lock, epoch management, page consistency mechanisms, and error reporting all carry over to EC. Immediate EC will be covered in a separate design document.

Detailed Design

The write flow, AW lock mechanics, client-side IO duplication, and failure handling are each covered in the subsections below. Layout handling proceeds similarly to existing FLR, except that secondary mirrors are marked INFLIGHT rather than STALE during writes — see Mirror States for the full flag model.

FLR Mirroring

FLR Immediate Mirroring State Machine

FLR Immediate Replication

Write Operation Flow

At the top of each write or other OST-modifying IO (see §2.3.2), the client obtains a valid layout first and then takes an AW reference, acquiring the lock or reusing a cached one (see Lock Ordering). This precedes primary PW acquisition/reuse and any dirtying; AW is not first acquired at writeback or BRW submission. Ordinary buffered write() may return after accepting data into the client cache, with normal aggregation and later BRW fan-out to the participating mirrors. The primary mirror remains readable; secondaries are INFLIGHT (see Mirror States).

The client holds AW across this write-permission window and its resulting dirty data and IO. Before releasing AW, it must stop further dirtying under that authority, obtain committed or recorded-failed outcomes for every participating mirror, and safely resolve outstanding IO (see Lock Lifetime). It releases AW with the mirror-error summary (see Error Reporting). Application return, RPC completion, durable commit, AW release, and global epoch closure are distinct events; the agreed application contracts and remaining mechanism questions are recorded in Operation Completion Consistency Model.

In the normal (no-error) case, once all clients release their AW locks, the MDS closes the epoch and transitions the layout back to RDONLY. If any client reports errors, the MDS forces epoch closure — see Epoch Close for details.

Example Scenario

With one client, if there are three mirrors and three writes:

If write 1 errored to mirror 0
If write 2 errored to mirror 1

The client will report errors on mirror 0 and mirror 1 to the MDS as part of AW lock cancellation (see Active Writer Lock for the LVB structure). The MDS would then clear INFLIGHT from mirror 2 (it is now clean) and mark mirrors 0 and 1 as STALE (clearing INFLIGHT, adding STALE). Userspace can try to resync to these mirrors, and if this fails, will need to add new mirrors.

If all mirrors fail writes during an IO and no successful eligible replacement exists, report the failures through the usual IO error and logging paths. On epoch close, retain the existing primary as the sole valid, read-eligible best-effort replica, clear INFLIGHT and mark every other mirror STALE. The primary may be incomplete; selecting it does not turn failed IO into success. Resync can use its current contents to restore redundancy, but cannot reconstruct missing updates. See the selected policy and review note in §2.4.4.

Primary Mirror Selection and Read Visibility

The primary is the only readable participant during an epoch: without MVCC (which Lustre’s storage backends lack), writes reach different mirrors at different times, and concurrent readers could otherwise see inconsistent data. At epoch entry the MDT marks every other participant INFLIGHT. IWM clients, and patched non-IWM clients through their unknown-flags-as-STALE rule, exclude INFLIGHT mirrors from reads and from OST-derived metadata such as file size. The primary is never INFLIGHT. Clients without the safe-selection capability cannot open IMMEDIATE files at all (§2.4.6). Reads already in flight at entry rely on the layout invalidation and revalidation that entry triggers. See Mirror States. However, this is not possible for DIO, which does not use LDLM locks on the client, but instead only uses them locally on each separate server. (Superseded: v1 DIO to IWM files takes client-side extent locks on the primary; see §1.2.)

Active Writer Lock

This section covers the Active Writer (AW) lock: how it is used, what it guarantees, and how the MDS uses it to manage write epochs. Throughout, “flush before release” includes failure resolution: successful mirrors must be durable, and terminally failed mirrors must be reported with their outstanding IO safely resolved.

Lock Semantics

The AW lock is a CW (concurrent write) lock on a dedicated MDS IBITS bit, ACTIVE_WRITERS. Many clients can hold it at once, and the MDS uses its presence to know whether a file has active writers. When the MDS needs exclusive access to both layout and write epoch (epoch close, mirror replacement, resync), it takes layout EX first and then AW EX, which forces every holder to drain admitted IO, flush and release.

Every distributed AW acquisition, whether it opens the epoch or joins an open one, durably records that client’s participation before its write authority becomes usable (§2.2.5; cost in §2.2.3). Taking an IO reference to an already-cached AW grant acquires nothing new and writes nothing.

Epoch identity. An epoch is identified by the file plus the layout generation of the transition that opened it (RDONLY → WRITE_PENDING). Entry already bumps the generation, so the value is unique and increasing per file; the current generation cannot serve because it changes within an epoch. The MDT stores it in the active-epoch record and returns it with the AW grant; the client puts it in the cancellation report header; participation and error records carry it. Since only one epoch per file can be open at a time, the file alone identifies the open epoch on the MDT; the epoch ID matters for messages that outlive their epoch. The MDT discards, and logs, any cancellation report or replayed AW lock whose epoch ID is not the file’s open epoch. For example, an error report retried after an MDT restart, whose lock was cancelled locally and so not replayed, was already covered by the lost-participant rule when that epoch closed and must not be applied to a later one.

Lock Lifetime and Commit Requirements

Admission and coverage. Every modifying IO obtains a valid layout, takes an AW reference, then checks that its soft LAYOUT grant is still valid before it acquires or reuses primary PW write authority or dirties data; if the layout was lost it drops the AW reference and retries from the layout (§2.2.7). So AW cancellation must wait even for an admitted IO that has not dirtied anything yet. Dirty data belongs to the epoch as soon as it is dirtied, not when a BRW is submitted, and keeps the file’s AW obligation alive after the IO ends; a layout refresh within the epoch does not move that obligation to a later epoch. Cached PW reuse and mmap write permission obey the same rule.

Release condition. The client holds AW until nothing more can be dirtied under it and every participant has a resolved outcome: its updates confirmed durable on OST storage, or its failure recorded for the MDS. Recording a failure alone does not permit release: outstanding sends, retries and replay must first complete or be safely cancelled or fenced. A mirror reported successful must be durable; an RPC reply alone is not enough.

This is stronger than for ordinary Lustre writes. Without IWM, a buffered write that reached OST memory but not the journal is lost if the OST crashes, which is data loss. With IWM, if one mirror’s OST commits and another’s does not, the mirrors silently diverge with no signal to the MDS, and that inconsistency can corrupt reads after recovery.

The release barrier. Every AW release, whether forced, voluntary or idle, and even while other clients keep the epoch open, synchronously establishes these outcomes; there is no tunable to disable it, and no secondary timing policy weakens it. Each OST import tracks last_committed. At release:

1. For each OST object written during the epoch on a mirror not already recorded as failed, check whether the highest transno among its successful write replies is ≤ that import’s last_committed.
2. If all remaining updates are committed and failed-mirror IO is safely resolved, release with the accumulated error report.
3. For objects still behind, issue an object-scoped OST_SYNC (osc_sync_base()) and confirm durability. Secondary sync and recovery waits follow the mount’s secondary failure policy. A terminal inability to establish durability is recorded as a mirror failure and its IO safely resolved; release does not require a failed mirror to commit.

This barrier does not make buffered write() synchronous, change the DIO or explicit-sync contracts, or require a synchronous commit of the MDT’s closing transaction.

Existing mechanisms. At 47638add78, every write request stays on the import replay list until the OST commits it, and its rq_commit_cb (brw_commit, osc_request.c:2759-2771, 2922) fires then; unstable-page accounting and soft sync (OBD_BRW_SOFT_SYNC) build on it. That gives commit notification per RPC but no per-file view, so IWM adds per-file tracking: for each OST object the file wrote during the epoch, the highest transno from successful write replies, recorded at reply time. PW lock cancellation (osc_lock_flush) waits for replies, not commits, so it cannot serve as the barrier. The OST sync_lock_cancel setting is not relied on: in effect it defaults to never (ofd_slc_set() sets always when sync_journal=0, but tgt_init() later resets it; ofd_dev.c:3157, tgt_main.c:493), and where enabled it covers only the primary’s OST.

DIO. Successful IWM DIO completion means stable storage, with or without synchronous flags: each participant’s updates are committed or recorded as failed, outstanding activity is resolved, and the primary determines the result. At 47638add78, osc_page_submit() sets OBD_BRW_SYNC on every immediate transfer, including DIO (osc_page.c:302), and OFD sets th_sync whenever a niobuf lacks OBD_BRW_ASYNC (ofd_io.c:1309-1319), so a successful DIO reply is already journal-committed. Secondary DIO RPCs rebuilt by fan-out must carry the same flags. Buffered IO tracking continues after write() returns, through writeback, RPC completion and commit.

Failures. A client that dies without releasing AW is handled by eviction and recovery. For a live client, failures during submission, transport or recovery, the modifying RPC, or commit/sync are reported once terminal under the selected policy; a retryable event that recovers is not a mirror error. Terminal failures lead to STALE through the cancellation report and epoch close.

Caching and retention. The client caches its AW grant so that most writes run in an already-open epoch: each modifying IO validates its layout and takes a reference to the cached grant, with no MDT round trip. An AW enqueue that joins an unchanged epoch reuses it and a still-valid layout grant, without repeating primary selection, WRITE_PENDING/INFLIGHT publication, active-epoch insertion or layout replacement; it does record durable participation. Caching never bypasses layout validation or cancellation gates. Retention uses ordinary LDLM lock-use and recency semantics, with one difference: AW has its own shorter retention timer, proposed at about 60 seconds, so healthy secondaries do not stay INFLIGHT long after writing stops.

Timer expiry starts the normal release; it does not itself release AW or clear INFLIGHT. The client first gates or retires covered PW write authority and writable mappings (reusing unmap_mapping_range() with the AW admission interlock, §2.2.7.3), then drains admitted IO, flushes, and passes the release barrier. A cached PW lock may not allow new dirtying after AW release. The last holder’s release triggers the ordinary rollup that restores secondaries’ read eligibility.

There is no maximum epoch length. Because lock use resets the retention timer, a continuously written file keeps its epoch open and serves reads only from its primary until writing pauses; nothing is at risk, since every AW release is durable on all participants. A forced periodic rollup would drain and sync every writer only for the next write to reopen the epoch. Operations that need a closed epoch (resync, membership changes) force closure through LAYOUT EX → AW EX.

Open items. Exact AW reference points, timer/admission races, mmap fault and PTE handoff, and cached PW/AW lifetime integration (lustre-design-docs-od5.36, lustre-design-docs-od5.8 P1–P3); the retention timer value and controls; confirming that fan-out preserves the DIO sync flags, and testing on both backends (A9).

Error Reporting via Cancellation LVB

When the client releases AW, it reports write results to the MDS in the client-to-MDS cancellation request. This is new protocol: the existing LDLM_CANCEL format carries no LVB. The report is a versioned, length-delimited, extensible message (§3.1). In v1 it carries the epoch ID and a counted list of failed mirror IDs, with no per-stripe detail. Errors are rare, so the list is usually empty; naming mirrors directly needs no numbering shared between client and MDS. The MDS validates each ID against the epoch’s participants (a report naming a mirror that was not written is invalid and gets conservative handling), takes the union of all valid reports for the epoch, and marks each named mirror STALE as a whole. CLIO collects errors from every modifying operation covered by AW, including setattr. Per-stripe error recording is TBD.

Durable acceptance. An error-bearing cancellation is acknowledged only after one MDT transaction has made durable both the reported errors and the removal of that client’s participation. Error records name the file, epoch and mirrors, with no durable association to the reporting client (debug logs may record it); in-memory receipt is not enough. Error records are distinct from participation: the reporter can release AW and retire its participation without waiting for rollup (holding AW until then would block the AW EX that resolves the error), and the error record survives until the closing transaction consumes it. Recovery therefore sees either the old participation, to be reconciled, or retired participation with its errors safely recorded, never participation removed with its errors lost. This transaction does not remove the active-epoch record.

Clean cancellations commit lazily. For zero-error cancellations, participation removal becomes durable with the next regular MDT commit; the MDT neither forces nor waits for a commit before acknowledging. The cost is a short window in which an MDT crash leaves the old participation record, and if that client also fails to reconnect, the file is degraded unnecessarily.

MDT commit cost. Where IWM forces an MDT commit wait, relative to delayed-write FLR:

Event	Forced commit wait?	Cost relative to FLR
Epoch entry (RDONLY → WRITE_PENDING, active-epoch record)	yes	none extra: FLR’s write-intent transition is already synchronous (th_sync=1, mdd_object.c at 47638add78), and the record shares that transaction
Another client joining an open epoch	yes	new: one per client per file per epoch; the AW grant is then cached
Error-bearing AW cancellation	yes	new, rare
Clean AW cancellation	no (lazy)	an ordinary transaction, typically when cached AW expires, off the application IO path
Epoch close / rollup	no	an ordinary transaction

Concurrent forced commits share journal commits (jbd2 batches them; ZFS waits on a txg), so the cost shows mainly as MDT commit pressure rather than application latency.

Terminal errors release promptly. When an error becomes terminal under the selected policy, or administrative abandonment declares a secondary failed, the client resolves the remaining epoch IO and releases AW without waiting for the retention timer, reporting the failed mirrors. Any voluntary release carrying an error triggers rollup immediately, even with other holders (§2.2.4). A retryable event that recovered does not.

Open items.

• Durable error-bearing CANCEL delivery (lustre-design-docs-od5.29). Stock LDLM cancellation does not deliver the report durably: at 47638add78, ldlm_cli_cancel_local() (lustre/ldlm/ldlm_request.c:1382) cancels the local lock before the CANCEL RPC, and ldlm_cli_cancel_req() (lustre/ldlm/ldlm_request.c:1598) treats LUSTRE_ESTALE as success and tolerates other failures. The client must keep the pending report until durable acceptance, and the server must record it before lock destruction and reprocessing can lose its association. A stale or unknown cookie is not proof of acceptance; the epoch ID prevents applying it to a later epoch. Specify retries, lost acknowledgements, persistence failures, and reconnect when the cancelled lock is no longer replayed.
• Packing, limits and per-lock association in batched and early-cancel RPCs (§3.1).

Epoch Close: MDS Processing

The last voluntary AW release triggers rollup, and so does any voluntary release carrying an error report, immediately, whatever other holders remain. A trigger starts closure; it does not mean the epoch is closed. The server always takes LAYOUT EX, then AW EX, even when there were no holders at the trigger. A clean last release and an error-triggered release enter the same sequence:

1. The client drains its work. It finishes admitted modifying IO, flushes the file’s dirty data and resolves mirror IO and durability under its existing protection, then reports failed mirrors in the AW cancellation. It never releases AW with dirty data or unresolved admitted IO, and acquires no new distributed locks to finish old work.
2. The MDT accepts responsibility durably (§2.2.3). The reporter is then done. The epoch, its active-epoch record and all accumulated errors remain.
3. Rollup starts. The MDT requests LAYOUT EX, then AW EX. Until LAYOUT EX is granted, another client may still acquire AW and join the epoch; AW grants are not parked behind a “rollup pending” gate, which would create a circular wait.
4. Remaining writers drain. LAYOUT cancellation invalidates their layout views. A restarting IO drops its AW reference before waiting for a fresh layout, while old dirty data keeps file-level AW protection. AW EX cancels and drains every remaining holder, including late joiners, each finishing with a file-wide flush and no layout reacquisition. Their error reports get the same durable acceptance.
5. The MDT establishes the outcome, with both EX locks held, from all reports and durable error records, using the outcome rules below.
6. One transaction closes the epoch. It publishes the RDONLY layout with the resulting INFLIGHT/STALE flags, consumes the error records it handled, and removes the active-epoch record.

Outcome rules, applied in this order (also used by recovery, §2.2.5):

• A known primary error first. Replace the failed primary with an eligible mirror that has no known failure, and mark the old primary STALE. If none is eligible, report the failures and keep the existing primary as the sole best-effort copy (§2.4.4); this does not mean the failed writes succeeded.
• Then any lost participant. If a participant was evicted or lost, its outcome is unknown: every mirror other than the resulting primary becomes STALE in full, however few mirrors any reports named, and the primary is kept as a best-effort copy.
• Otherwise, ordinary results. Mirrors with reported errors become STALE; the others lose INFLIGHT and are clean. Pre-existing STALE is never cleared by a clean epoch; only resync clears it.

STALE mirrors are visible to lamigo through changelogs, the same mechanism Hot Pools uses, and it attempts resync (§2.4.5).

Atomic close, no extra commit wait. Recovery sees either the pre-close state (active-epoch record and error records) or the complete closing layout with that bookkeeping gone. If the close transaction is lost before it commits, the old evidence remains and recovery repeats rollup; if it commits, the layout carries the outcome. This holds even after every client has released AW, because the active-epoch and error records are not tied to participation. So close needs no dedicated synchronous commit; entry and error-bearing cancellation still do. Later epoch entry, resync and other layout operations must not become durable while losing an earlier close they depend on, nor reinterpret its errors as a later epoch’s. Within one MDT’s backend this ordering holds naturally: ldiskfs synchronous transactions wait for a journal commit and ZFS waits for the transaction group, so a later durable entry or participation record also makes the preceding close durable (osd_trans_stop() in both OSDs at 47638add78). It is not a commit across MDTs or OSTs.

On a hot file closure is expensive: every holder must finish the synchronous release before AW EX is granted. Finite fast- and slow-fail timings bound secondary recovery waits; wait-indefinitely can keep closure waiting until recovery or administrative abandonment. Administrative operations such as mirror replacement and resync use the same sequence.

Lost participants: drain, then version fence. An evicted client cannot be counted as drained: it may still have old-version writes in flight and access to the OSTs. Its outcome is covered by the lost-participant rule above; late writes are rejected by the existing FLR write-era mechanism, not by an MDT-driven extent-lock flush (the MDT has no way to take OST extent locks). FLR’s era boundary is specific: a push carrying LU_LAYOUT_RESYNC raises the OST object’s minimum accepted version, while an ordinary generation update only widens the accepted range (era transition (lustre/lod/lod_object.c:8742), OST range handling (lustre/ofd/ofd_objects.c:608), old-version rejection (lustre/ofd/ofd_io.c:443)). The push travels from the MDT through OSP asynchronously (delivery conditions (lustre/osp/osp_sync.c:244)).

• Only closes with a lost participant are fenced. If every participant drained through AW cancellation, nobody retains write authority and no fence is needed. A lost-participant close, live or in recovery, pushes the era-ending version to every object of every participant mirror, including the primary; the next epoch’s entry then opens a new era above the lost client’s version.
• Close does not wait for delivery. After such a close every non-primary mirror is STALE and the primary is best effort. A late write to a STALE mirror is harmless; a late write to the primary is the application’s own data arriving late, with no other readable copy to diverge from. Publishing the layout, restoring readability and admitting the next epoch do not wait for the push.
• Copy-to-clean operations wait. Divergence becomes possible only when copies are made to agree again: resync, mirror extend, or any copy that clears STALE. Before reading the source mirror, such an operation must confirm the era-ending version is applied on the source objects; a queued OSP record is not confirmation. osp_sync() (osp_dev.c:860) waits for an OST’s pending records, with a timeout and -ENOTCONN when disconnected; a narrower per-object wait may be preferable. Target objects fence themselves, because resync writes carry LU_LAYOUT_RESYNC.
• Existing FLR resync has the same window (lustre-design-docs-od5.49.31). Resync start pushes gen|LU_LAYOUT_RESYNC to the primary asynchronously and copies immediately, with no group lock, lockless DIO reads and no data-version check at resync-done, so a writer evicted from the MDT but still connected to the OSTs can write the primary after resync copied that range. The gate above fixes FLR as well.

Open items. Era-ending sequencing (lustre-design-docs-iwm.6.1): MDT transaction and OSP delivery ordering, writes already prepared before the version check changes (bulk IO, commit, retries, replay), disconnected OSTs, and integration with the atomic close and crash recovery; whether close reuses SYNC_PENDING internally or an equivalent era-ending step. Atomic crash recovery and dependency ordering must be verified on ldiskfs and ZFS.

Recovery After MDS Failover

AW lock state, including which clients hold AW, lives in MDT memory and is lost on restart. To close interrupted epochs safely, the MDT must know which files had open epochs, which clients took part, and which mirror errors were already reported. It keeps that in durable tracking.

The primary of an open epoch is recoverable from the layout itself: INFLIGHT is stored durably in lcme_flags, and the primary is the IMMEDIATE mirror without INFLIGHT. Non-immediate mirrors are never primary candidates merely because they lack INFLIGHT.

Durable tracking. The MDT keeps three logically distinct kinds of record:

• Active-epoch record (file plus epoch ID). Inserted by the opening AW request, in the same transaction that moves the layout to WRITE_PENDING and sets INFLIGHT. Removed only by the atomic closing transaction (§2.2.4).
• Client participation (client, file, epoch). Recorded durably as part of every client’s AW acquisition, including joins of an open epoch, before that client’s write authority is usable. Retired by that client’s cancellation transaction (§2.2.3).
• Mirror-error record (file, epoch, failed mirror IDs). Recorded durably before an error-bearing cancellation is acknowledged; it carries no durable reporting-client association. Consumed only by the atomic closing transaction, so it survives the reporter’s AW release and participation retirement.

Each kind is independently updateable; there is no growing per-file client list rewritten on every join or release. The MDT also keeps an in-memory client → file/epoch lookup, maintained with live participation and rebuilt from the durable records on restart; it is not stored separately. Participation is recorded per client, rather than per file only, so that losing a client degrades only the files it was writing. The cost is one forced MDT commit per client join (see the cost model in §2.2.3); cached AW references add no further writes.

Two alternatives were rejected: scanning all inodes for INFLIGHT mirrors (correct, but prohibitive on large filesystems) and relying on clients’ AW lock replay alone (fails exactly when a client is evicted, the case recovery exists for).

Recovery sequence.

1. A close that became durable needs nothing. Its layout outcome is on disk and its tracking is gone. A close that did not become durable left its active-epoch and error records, so the epoch is simply still open.
2. Load the tracking. Read active-epoch, participation and error records; mark epochs with error records as needing rollup. A recorded error does not need its reporter to reconnect.
3. Finish ordinary lock replay and reconciliation. Rebuild surviving AW holders and match them to participation records. Do not cancel holders while replay is still assembling them, and do not make a missing-holder decision from partial state. A participant that fails to recover is lost and is evicted. An admitted client with no participation record is never assumed not to have written. A replayed AW lock with no matching active-epoch record (same file and epoch ID) is rejected; see below.
4. Then roll up affected epochs with the ordinary close. Epochs with error records, and epochs left with no recovered holders, close through the same LAYOUT EX → AW EX sequence as a live close (§2.2.4), without waiting for recovered holders to go idle. A writer that joins before LAYOUT EX is granted is drained by that same rollup. Independent files are processed by bounded parallel workers; the active records are the authority, so unfinished work is rediscovered after another restart without a progress journal.

Each recovery rollup applies the ordinary outcome rules (§2.2.4): a known primary error first, then the lost-participant rule (which affects only files whose participants did not all recover or release safely), then ordinary per-mirror results; late writes from a lost client are fenced as described there.

Consumed error records and the active-epoch record are removed in the closing transaction itself; they are never deleted early, and never kept just to replay a released reporter’s lock. An active-epoch record for a file that is no longer an IWM file is removed only after its error records are handled.

Replayed AW without a matching active-epoch record. This violates the protocol invariant but must not crash the MDS: it depends on client replay state and index integrity. Reject the replay (the client drops the lock; evict the client if needed), log it with CERROR (an LASSERT may remain in debug builds), and handle the file conservatively. If an active-epoch record exists for a different epoch, the lost-participant rule applies; if none exists, open a recovery rollup that stales every mirror other than the primary. Never create the missing record or treat the file as clean.

Storage. Dedicated local DT indexes are the preferred candidate, not yet a chosen format. DT index operations (lustre/include/dt_object.h:1532) provide keyed lookup, declared insert/delete under a transaction handle, and iteration on both backends (local index creation (lustre/obdclass/local_storage.c:514); ldiskfs IAM (lustre/osd-ldiskfs/osd_handler.c:6029); ZFS ZAP (lustre/osd-zfs/osd_index.c:1769)), with fixed-size unique keys. Constraints found in the source (Lustre 47638add78):

• Sharding: LFSCK’s hashed local trace indexes (lustre/lfsck/lfsck_lib.c:2716) are a precedent for FID-hashed shards in dedicated IWM objects (not shared OI/LFSCK records).
• No ordered scans: ZFS iterator seek (lustre/osd-zfs/osd_index.c:1870) is not implemented for nonzero keys and index negotiation (lustre/osd-zfs/osd_index.c:2024) rejects range and variable/non-unique formats, so recovery scans shards and groups records in memory.
• Same-transaction updates: index mutations and the layout xattr can share one local backend transaction (MDD orphan insertion (lustre/mdd/mdd_orphans.c:82) is a caller-owned precedent), but MDD layout change (lustre/mdd/mdd_object.c:3543) owns its transaction today and must be extended.
• LLOG: catalog append (lustre/obdclass/llog_cat_server.c:443) can join a caller transaction, but record cancellation (lustre/obdclass/llog.c:198) starts its own, so LLOG would need extra bookkeeping to meet the atomic boundaries.

Execution failures within a transaction. A shared transaction handle groups updates but does not undo them if a later mutation fails (ZFS stop (lustre/osd-zfs/osd_handler.c:280) commits an assigned transaction; ldiskfs stop (lustre/osd-ldiskfs/osd_handler.c:2291) closes the handle without rollback), and declarations do not make execution infallible (DT execution fault hooks (lustre/obdclass/dt_object.c:1756)). DT insertion does not overwrite (IAM (lustre/osd-ldiskfs/osd_iam.c:2438) rejects an existing key; ZFS uses zap_add_uint64), so error evidence must be added monotonically, never deleted and rewritten. Only a real commit callback (not DCB_TRANS_STOP) may satisfy the grant and error-ACK durability gates; target last-committed handling (lustre/target/tgt_lastrcvd.c:937) is a precedent for refusing to advance on commit error. Proposed execution order, stopping at the first failure (lustre-design-docs-od5.61):

1. Opening: active-epoch record, then participation, then the opening layout; AW authority becomes usable only after all of it is durable. A join records only participation.
2. Error-bearing cancellation: add every reported error before removing participation; acknowledge only after the whole handoff is durable. On partial failure keep participation, errors and the pending report for idempotent retry.
3. Close: write the closing layout outcome first, then delete consumed errors, then the active-epoch record. Never continue cleanup after a failed outcome write or report success on an execution or commit error.

An abstract check of 20 failure points in this order found no lost evidence, but six close cases leave a closed layout with residual tracking (next item). It assumes each mutation applies wholly or not at all; it is not a backend fault-injection test.

Open items.

• Question for Andreas Dilger: recovery-phase placement. Should recovery rollups run as one additional final phase of recovery, before normal service resumes (Patrick’s tentative preference), or as ordinary operations after recovery, with their locks blocking only the affected files? Includes the recovery hook, when cancellation and drain are available, and service-resumption timing.
• Lock progress during recovery (lustre-design-docs-od5.30). A final-phase rollup cannot just call the ordinary close worker: LDLM reprocessing is suppressed during lock recovery (lustre/ldlm/ldlm_lock.c:2401) and ordinary RPCs wait for a full import (lustre/ptlrpc/client.c:879), so client CANCEL is delayed during replay. Define how LAYOUT/AW callbacks, cancellations, reprocessing, error persistence and drain make progress in that phase without admitting ordinary IO, keeping LAYOUT EX → AW EX.
• Decision needed (Patrick): failure-only close residue. With outcome-first close ordering, a cleanup deletion can fail after the closing layout has executed and committed, leaving a closed layout plus old tracking. Proposal: recognise a durable completed outcome for the same epoch and treat the surviving records as cleanup-only, never re-running outcome selection; reconcile before any later epoch or conflicting layout change, and without unambiguous recognition keep the evidence and block conflicting progress. RDONLY, a current-FID lookup or a missing lock is not enough to recognise it; no outcome or epoch tag encoding is chosen. The alternative is a proven atomic or rollback primitive.
• Detailed design (lustre-design-docs-od5.21, lustre-design-docs-od5.61): physical schema and key encoding; stable client identity across reconnect; duplicate or reacquired grants; replay matching; distinguishing a safely released participant from a lost one; index bootstrap; shard count and versioning; iteration and update races (never hold an index iterator or lock across distributed-lock waits; verify ZAP cursor behaviour under mutation and scan completeness); memory bounds; both-backend crash and fault-injection tests. Per-stripe error records remain TBD.

Flush Semantics on Lock Release

AW cancellation drains admitted modifying IO, including operations that have not dirtied anything yet and DIO with no cached pages, then performs a final file-wide flush of all dirty data, including data never submitted in an RPC, and resolves every outstanding mirror outcome (the release barrier, §2.2.2). Writeback may start earlier, but a dirty-page scan taken before admitted IO has drained is not the barrier, because that IO could still dirty data. AW is never released while dirty data, or an admitted operation that could produce it, remains.

The drain runs under the protection the client already holds. It takes no new layout, AW or extent lock, and existing object mappings and references stay usable until the work is resolved. Local synchronization is allowed; a new distributed-lock dependency in the cancellation path is not.

The client must associate admitted modifying IO, PW write authority, dirty pages, in-flight RPCs and commit/error tracking with the current epoch, across buffered syscall return and layout refresh. RPC reference counts or lo_active_ios alone do not cover idle dirty pages or cached write permission. A file-wide flush avoids needing per-page epoch ownership just to choose what to flush.

There are two release paths with the same completion condition:

• Voluntary release (idle timer or an error, no conflicting request). There is no server-held layout EX, so the client gates new modifying IO and write faults locally and revokes writable mappings with unmap_mapping_range() (§2.2.7.3), accounting for faults already past admission at the locked-page/PTE handoff. Dirty cache and existing PW, object and AW protection are kept. It then flushes, passes the release barrier, and releases with its error report. A terminal failure skips the idle timer. The last voluntary release triggers rollup, and an error-bearing one triggers it immediately (§2.2.4).
• Forced revocation (the MDS holds layout EX and requests AW EX). Layout revocation has already blocked new IO and unmapped writable mappings; a cached AW or PW lock is not a bypass. The client starts draining and flushing at once. The blocking AST completes only when admitted IO has drained, the final flush is done, successful mirrors are durable and failed-mirror IO is resolved; a timer expiring or an error bit being set does not complete that work. Finite fast- and slow-fail budgets apply during forced closure; wait-indefinitely stays unbounded (administrative abandonment is the separate way to act on its outstanding work). An IO that must restart for a fresh layout leaves the old drain before waiting for that layout, while its dirty data stays covered by the file’s AW obligation.

Neither path lets late retries escape the closed epoch and interfere with later IO, resync or mirror reuse, and neither requires a conclusively failed mirror to commit.

Release does not need to discard cached data. unmap_mapping_range() removes file-backed PTEs without discarding dirty pages; vvp_prune() and fake layout invalidation are not used just to revoke mapping authority. (Reusing the complete PW cancellation path would discard pages as a side effect, since osc_lock_flush() writes back and then discards.) Writable mappings may not outlive AW coverage. A secondary failure alone never discards data when the primary succeeded; primary and writeback errors keep normal handling.

Evicted clients cannot be relied on to cancel or flush; they are handled by the lost-participant rule and version fence in §2.2.4, not by a full-extent PW sweep.

Existing paths to reuse, with their limits (47638add78): layout invalidation only invalidates and unmaps; applying a changed layout (cl_conf_set() / lov_conf_set() → lov_layout_change() → cl_object_prune() → vvp_prune()) flushes with CL_FSYNC_LOCAL, which waits for writeback, not commit, and then tears down the cache. The CIT_FSYNC path skips layout refresh in vvp_io_init(), and cl_sync_file_range() can use existing object mappings with ci_ignore_layout.

Open items (lustre-design-docs-od5.8 P1–P3, lustre-design-docs-od5.36): exact IO reference boundaries and the final-flush barrier; the voluntary-release admission gate; mmap fault/unmap races; preservation of old object and PW references through pruning and reconfiguration; local configuration-lock and fan-out callback dependencies (no local wait cycles).

AW Lock vs Layout Lock Interaction

The layout lock tracks what the layout is (stripes, components, mirror membership); the AW lock tracks the write epoch (who is writing and when the epoch closes). They are separate locks that interact closely.

Lock order: LAYOUT before ACTIVE_WRITERS, everywhere. The client’s layout hold is soft; the server takes EX on both, layout first, for any operation needing both. There is no exception.

• Client admission. Obtain LAYOUT, acquire or reference AW at the top of the modifying IO, then check that the LAYOUT grant is still valid and that the layout and AW epoch are compatible, before any PW use or dirtying. If the layout was lost, drop the IO’s AW reference (returning an unused new grant through normal release) and retry from LAYOUT; never wait for a fresh layout while holding the old IO’s AW reference. Old dirty data keeps the file’s AW protection until the normal drain. The same sequence applies to cached AW. Files that do not need immediate mirroring take no AW.
• IO restart on a layout change. An ordinary layout change within an epoch revokes LAYOUT but not the AW grant, which can stay cached; the restarting IO drops its own AW reference and readmits. Dirty data keeps its epoch association.
• Epoch close and administrative operations (resync, mirror replacement or addition, IMMEDIATE membership changes): layout EX, then AW EX, with exclusion held through the operation (§2.2.4). Layout EX alone is not a flush barrier. Structural changes require full page-cache teardown; flag-only changes can use STATE_ONLY_CHANGE (§2.2.7.3). Membership changes follow the conditional rules in §4.2.
• Leaving IWM. Removing immediate mirroring is an epoch boundary: the old PW, dirty-data and AW obligations are resolved before delayed-write authority is granted, and later writes under that layout take no AW. A changed layout never retires pending obligations by itself.
• Non-IWM access follows the capability gate in §2.4.6. Reads by capable clients never force closure. Optional primary-only writes reuse ordinary FLR write intents, closing any open epoch with LAYOUT EX → AW EX first.

Forced close, from the client’s side.

1. The MDS obtains layout EX. Clients’ layout holds and writable mappings are invalidated, and no new IO can obtain a valid layout.
2. The MDS requests AW EX. Each client drains the IO already admitted under AW, even IO that has not dirtied anything.
3. The client performs the final file-wide flush under its existing protection and mappings, with no new locks, and passes the release barrier.
4. It reports its outcomes and releases AW. When every holder has released, the MDS holds both EX locks and finishes the close or administrative operation. Lost clients are handled by the fence in §2.2.4.

The layout gate stops new IO; AW cancellation finishes old IO. The closing flush never refreshes the layout or readmits itself behind the server’s layout EX, which removes the layout-EX / AW-cancellation / layout-CR circular wait.

Epoch entry reply: updated layout plus a replacement LAYOUT lock. The opening AW request changes the layout under MDT layout EX, which revokes the requester’s soft layout lock. The AW enqueue reply therefore returns the updated layout and a replacement LAYOUT grant together with the AW grant. These are two independent locks in one reply, not a merged-bit lock and not LDLM lock conversion; returning layout bytes without a grant would not do. The MDT makes the epoch durable under LAYOUT EX, then hands over the replacement LAYOUT grant followed by AW, serialized so that no incompatible transition can slip in between and make the pair obsolete (sharing one reply does not by itself guarantee this). The client refreshes the layout under the replacement grant and validates that grant, not the revoked one, before PW use or dirtying; if the grant is cancelled before validation, it retries normally even though AW arrived. A join of an unchanged epoch can keep a still-valid layout grant.

This reuses FLR’s existing handoff: ordinary write intents already change the layout, return the layout grant via mdt_intent_lock_replace() with an EX-to-CR downgrade in mdt_intent_layout(), and let the client apply it through ll_layout_refresh() / ll_layout_lock_set() (47638add78). Only the independent AW grant is new.

Open items (lustre-design-docs-iwm.3.2, lustre-design-docs-od5.8 P1–P6): the reply representation for both grants; cancellation, partial or lost replies, failure, retry and reconnect handling; ordered-handoff serialization; exact admission hook placement (ll_file_io_generic() is the likely ordinary-write hook; all paths that bypass it need equivalent coverage, §2.3); local lock dependencies during pruning and fan-out.

Why Separate Locks

AW is a separate IBITS bit, not a mode or flag on the layout lock, for three reasons:

The layout lock is light; the AW lock is heavy. The layout lock can be revoked at any time without flushing; the client simply restarts IO under the new layout, which matters for layouts that change often (PFL component instantiation, SEL). AW revocation forces a full flush and durable commit. On one lock, every layout revocation would become an expensive synchronous flush, and a sequentially written many-component PFL file would close its epoch at every component boundary.

The epoch must span layout changes. Component instantiation and SEL extension revoke LAYOUT but should not close the epoch: the participants and the write session are unchanged. Because ACTIVE_WRITERS is a separate bit, layout EX does not revoke it. The epoch closes only when the MDS takes AW EX (errors, resync, administration) or every client releases AW.

No MDT/OST dependency cycle. AW is held across OST IO, and an AW EX request waits for that IO while holding layout EX, so cancellation must never need a layout refresh; it uses existing mappings and protection. Local configuration guards, object pruning and asynchronous fan-out callbacks still need checking for local cycles.

Combining both bits on one lock with conditional flush semantics was rejected: it reinvents AW as a layout-lock mode and complicates every layout-lock user. LDLM lock conversion (take both bits, later drop LAYOUT) could save a round trip but depends on a lightly exercised feature with subtle recovery behaviour; it remains a possible future optimization. The entry reply above already returns both grants in one round trip.

Behavior During Layout Changes

This covers non-destructive layout changes within an epoch (component instantiation, SEL extension, flag changes that keep the participant set), where only the layout lock is involved and AW is untouched. It does not cover destructive changes, IMMEDIATE membership changes or epoch boundaries.

Component instantiation covers every participant. A write intent instantiates the needed components of every participating IMMEDIATE mirror, each according to its own layout, before fan-out to them; this extends the MDT’s existing primary-mirror instantiation and involves no client-side component creation. A secondary instantiation failure is a secondary modifying-IO failure: that mirror ends the epoch STALE, and a successful primary write is not turned into an application error. A primary instantiation failure follows the primary failure rules. Resync must also instantiate the components it needs; if that keeps failing, resync fails and the mirror stays STALE until a retry succeeds. Partial-success replies and their error recording are implementation work (lustre-design-docs-od5.49.12).

When such a change happens while a client holds both locks:

1. The server takes LAYOUT EX, revoking every client’s layout lock.
2. The client’s layout blocking AST returns immediately, with no flush and no AW involvement, as today.
3. OST writes already in flight complete normally. An ordinary generation change inside a write era only widens the OST’s accepted version range, so they are not rejected (§2.2.4).
4. An IO that needs the new layout drops its AW reference and restarts from the top, then waits for the fresh layout.
5. The AW grant stays held or cached, not pinned by the restarting IO; old dirty data and the epoch are untouched.
6. On restart the IO reacquires LAYOUT (through a write intent if needed), references the AW grant and validates the layout before continuing in the same epoch. If AW was meanwhile cancelled for closure, the old drain finishes first and the IO retries admission.

So layout changes are not epoch boundaries. Errors accumulate across them, and the epoch close handles all of them.

Layout Lock and Page Cache Interaction

Why layout changes are expensive today. When a client reacquires its layout lock after revocation, ll_layout_lock_set() applies the new layout via lov_conf_set(OBJECT_CONF_SET), and lov_layout_change() tears down and rebuilds the lov_object, destroying every sub-object (osc_object). Before teardown, cl_object_prune() → vvp_prune() flushes all dirty pages with cl_sync_file_range() and then drops the whole page cache (ll_truncate_inode_pages_final()): each cached page is attached to one sub-object and cannot outlive it. That is necessary for structural changes. Delayed-write FLR changes layouts only on idle files, so the cost is small. IWM changes them on files under active IO with hot caches on many clients, so dropping the cache at every epoch transition would make buffered IWM IO unacceptably slow.

Non-destructive changes keep their sub-objects: FLR state transitions (RDONLY → WRITE_PENDING, INFLIGHT and STALE flag changes) change only per-component flags, and component instantiation (PFL, SEL) only adds sub-objects. For these, rebuilding the lov_object is unnecessary, but not every cached page can be kept.

Read-eligibility rule. Each cached page belongs to one mirror’s sub-object, and clients choose their read mirror independently (per-client hash plus preference, lov_object.c:770-804), so pages cached from a non-primary mirror are normal. During an epoch, writers lock only the primary and secondary BRWs carry no client lock, so nothing would revoke another client’s pages on a secondary that has just become INFLIGHT; they would go stale. A writer that found such a page would dirty a page owned by a secondary sub-object, which lov_io_sub_init asserts against (lov_io.c:111-116). Today the full prune hides this. So when a state-only change is applied, every mirror that goes from readable to not readable has the pages and extent locks on its sub-objects discarded; mirrors that stay or become readable keep their cache:

• Epoch entry: the secondaries become INFLIGHT, so their sub-objects are pruned; the primary’s pages survive.
• Epoch close and component instantiation: no prune, because INFLIGHT mirrors are never read and so have nothing cached.
• A primary replaced at close loses eligibility, so its sub-objects are pruned.

This needs no new wire data. The prune already runs at layout refresh, not at cancellation: cancellation only invalidates and unmaps (vvp_conf_set(OBJECT_CONF_INVALIDATE)), and the refresh holds both the old and new layout, so the client compares per-mirror eligibility and prunes only the affected sub-objects (lov_delete_raid0 already prunes per sub-object). Between invalidation and refresh, fast read cannot serve stale pages, because ll_read_from_cache declines while the layout generation is CL_LAYOUT_GEN_NONE. Server-side locking on secondary BRWs was considered instead and rejected: it cannot cover the writing client’s own secondary-owned pages, which can be dirtied before any secondary BRW is sent.

The STATE_ONLY_CHANGE flag. The MDS sets a transient flag in the layout when the latest layout_gen bump was non-destructive, only for clients that advertise support through an OBD_CONNECT2 bit. On reacquiring the layout lock:

• If the client’s layout_gen is exactly N-1 and the new layout carries the flag, it updates flags (and adds sub-objects for new components) in place in the existing lov_stripe_md, skips the whole-file cl_object_prune(), and applies the read-eligibility rule.
• Otherwise (flag absent, more than one version behind, or a destructive change) it does the full teardown as today.

The flag describes only the most recent transition, hence the N-1 rule. The optimization is purely opportunistic: old clients never see the flag, clients more than one version behind ignore it, an MDS restart loses it, and its absence for any reason means a full teardown. It is separate from the safe-selection capability and from epoch serialization.

Prior art. CSDC added LAYOUT_INTENT_CHANGE for flag-only layout updates (LCM_FL_INCOMPRESSIBLE; EX-8355, Gerrit 54248). The EX patch defines the wire pieces (LAYOUT_INTENT_CHANGE, LAIF_INCOMPRESSIBLE, OBD_CONNECT2_UPDATE_LAYOUT) and an MDS handler (lod_declare_layout_minor_change()); master (47638add78) has only the wire definitions. That patch still bumps layout_gen and still tears down the client cache, avoiding the cost by deferring the change until no layout lock is held; IWM instead makes the client’s reacquisition cheap. The wire definitions could carry the IWM transitions. Existing client behaviour to build on: lov_conf_set already skips the prune when generation, flags and first-entry pattern are unchanged, and returns -EBUSY while lo_active_ios > 0, deferring a layout change during active IO (lov_object.c:1500-1515).

Mapping revocation reuses existing code. vvp_conf_set(OBJECT_CONF_INVALIDATE) already calls unmap_mapping_range(inode->i_mapping, 0, OBD_OBJECT_EOF, 0) (source (lustre/llite/vvp_object.c:110)). It removes file-backed PTEs across processes, not VMAs or dirty page-cache contents, and private COW data is untouched; a later access faults again. Voluntary and idle AW release use the same primitive (directly or through a factored helper) without clearing a valid layout or pruning the cache; forced closure gets it from layout invalidation. Unmapping does not prevent refaults (Linux allows remapping races (Linux v6.12 mm/memory.c:3556)), so it is coupled with the local AW admission gate, the drain of admitted work and normal page/folio/PTE synchronization, so that no old writable mapping survives the final quiescence point; the final flush follows, under the old PW, object and AW protection. Write refaults go through normal LAYOUT → AW admission; a cached PW lock alone cannot restore write permission. No new VMA tracking or distributed lock is needed.

Open items (lustre-design-docs-od5.36, lustre-design-docs-iwm.3.3): __ll_page_mkwrite() finishes its cl_io but leaves the page locked, and ll_page_mkwrite() returns VM_FAULT_LOCKED (Lustre fault path (lustre/llite/llite_mmap.c:137)); Linux (Linux v6.12 mm/memory.c:5023) installs the PTE afterwards, so an AW IO reference count does not cover that handoff. Integrate and test the locked-page/PTE interlock with AW cancellation (including a fault paused at that point, cached-PW refaults and continuously written shared mappings), and the pcc_page_mkwrite() early return. Implement the per-mirror prune in STATE_ONLY_CHANGE.

Client Implementation

Admission coverage. Every path that can dirty file data or modify OST objects must take AW admission (LAYOUT, AW reference, layout revalidation) before it does so, or be denied for IWM files. At 47638add78 the paths are:

Path	Notes
ll_file_io_generic() (read_iter/write_iter, AIO, io_uring, splice_write, copy_file_range via the splice fallback)	The likely main admission hook
Tiny write (ll_do_tiny_write(), on by default)	Writes into an already-dirty page with no cl_io and no new lock; the page is already under the epoch’s AW obligation, but the path must respect the voluntary-release admission gate
Hybrid IO	Switches large buffered IO to DIO; DIO rules apply (client lock, §1.2)
mmap write faults (ll_page_mkwrite())	Admission at fault entry; locked-page/PTE handoff in §2.2.7.3
PCC read/write/splice/mkwrite early returns	Bypass Lustre’s IO stack; must not dirty an IWM file without admission
cl_setattr_ost() (truncate, time setattr) and cl_falloc()	Admission like writes (§2.3.2)

Fast read serves already-cached pages without cl_io; it is read-only and declines while the layout is invalidated, so it needs no AW. Verifying this coverage is tracked in lustre-design-docs-od5.49.14 (mmap in lustre-design-docs-od5.36).

Write Duplication via BRW Page Fan-Out

The client duplicates every write to all participants (other modifying operations are in §2.3.2). The design choice is the layer at which duplication happens.

IO-level duplication is not viable. For buffered IO it would mean sending pages to the secondaries when the IO is created, effectively direct IO to the secondaries, because a page can be cached and aggregated for only one mirror. That loses asynchronous writes and write aggregation on every secondary. (For DIO it would work, but is not sufficient on its own.)

BRW page fan-out (selected). Duplication happens at the BRW layer, after the primary’s write is fully prepared (pages assembled, bounce pages allocated), so buffered and direct IO are handled the same way: buffered writes aggregate normally in the primary’s page cache and are duplicated when the BRW is built, and DIO is duplicated from the same BRW page array. When the primary’s OSC has built its BRW, it calls up to LOV, which rebuilds each secondary’s RPCs from those pages through that mirror’s own stripe mapping, so mirrors may have different layouts (stripe count and size, components, compression). The call-up runs in the writeback/ptlrpcd work path, where sleeping is allowed, but it allocates and builds IO inside writeback, so memory use and lock dependencies there need care. Duplicating the primary’s RPCs verbatim (RPC duplication) is simpler but requires identical layouts on every mirror, which real deployments cannot guarantee (§4.2).

Qian Yingjin’s prototype (LU-13643, Gerrit 63244) demonstrates the hook point and per-mirror rebuild, including heterogeneous layouts. It is a feasibility demonstration, not an implementation to reuse as-is (patchset 1, 2026-01, no human review, broad CI failures). Where it differs from this design:

Prototype behaviour (Gerrit 63244 patchset 1)	Required by this design
Primary BRW keeps its RPC slot until all secondaries finish (whole interpret deferred)	Free the slot at the primary’s reply; defer only extent/page completion, bounce pages and error collection
Wraps the primary’s encrypted bounce pages as clear text for secondaries	Reuse bounce pages when compression settings match; otherwise compress and encrypt from clear pages
Secondary errors dropped; only the primary’s result survives	Collect per-mirror errors into the AW cancellation report (failed mirror IDs plus epoch ID)
Resend path can free or reuse the shared completion context	Resend-safe refcounting of the shared completion
Every FLR write fans out; no IMMEDIATE/STALE gating	Fan out only to the epoch’s participants
Secondaries write without grant once grant runs out	Handle secondary ENOSPC/EDQUOT when the write is attempted, as a secondary error
Primary extent lock release does not wait for secondaries	No primary extent lock release while a covering secondary attempt is unresolved, on every release path
Secondary DIO flag passed from the server-lock flag	Secondary DIO RPCs carry the same OBD_BRW_SYNC semantics as the primary (§2.2.2)
Secondary writes carry OBD_BRW_NOLOCK (no client or server lock)	Acceptable given the per-mirror prune on read-eligibility loss (§2.2.7.3)

Page Consistency During Write Duplication

The primary’s BRW pages must stay valid and unmodified until every mirror attempt has completed or been safely resolved. Three threats:

• Source page modification. For buffered IO, another write to the same pages while secondary RPCs are in flight would send the secondaries different data from the primary. For DIO, changing the user buffer during a write is already an application error, but every mirror attempt must keep its own references to the source; a BRW page array is not a private copy.
• Competing IO on the same client. An overlapping write must be serialized against duplication in flight, so the secondaries see the same order as the primary.
• Competing IO from other clients (ordering invariant). A primary extent lock is never released while any secondary attempt covering its pages is unresolved. Otherwise client A could release its lock with a secondary write still in flight, client B could take the lock and write the same range to every mirror, and B’s secondary write could land first: the primary ends up A-then-B and the secondary B-then-A, with no error recorded. This holds on every release path (blocking-AST cancellation, LRU cancellation, early lock cancellation, and DIO paths that drop their lock early).

Every mirror RPC for a page must complete or reach a resolved terminal failure, so outcomes are tracked per page or per RPC across mirrors. A successful reply still needs the release barrier’s durability check. A failed mirror owes no successful reply, but its failure is recorded and its IO can no longer affect buffer lifetime or later epochs and resync.

RPC slots and completion are separate. The primary BRW releases its RPC slot when its own reply arrives, as today. Only extent and page completion (ending writeback), bounce-page release and per-mirror error collection wait for every mirror. Holding the slot until the secondaries finish could deadlock: two OSTs, each primary for some files and secondary for others, could fill their slots with primaries waiting on secondaries queued behind them. With the split, a secondary RPC waits only for its own reply. The request stays on the import replay list until committed, independently of the slot. Keeping the primary extent in its in-RPC state until all mirrors finish preserves the ordering invariant. In-flight duplicated data is bounded by kernel writeback throttling and each OSC’s own slot limit.

Buffered IO Completion Semantics

Buffered write() stays asynchronous: it may return once data is in the client cache, before any mirror RPC is sent, with no stable-storage promise and normal aggregation; explicit synchronization has its own contract (§4.1). AW coverage starts at admission and continues at file level through syscall return, delayed writeback and commit tracking; reusing a cached PW lock obeys the same rule, and a layout refresh never leaves accepted data without an epoch.

The fan-out barrier applies to writeback completion, not syscall return: page stability and overlapping-write ordering are kept until every mirror attempt is resolved, including against stores through already-writable mappings (a page lock alone does not stop those). Page completion is separate from durable commit, which the release barrier handles.

Memory-mapped writes use the same writeback fan-out, but write permission is granted earlier: the write fault validates the layout and takes AW at entry. Forced closure invalidates the layout and unmaps writable mappings before requesting AW cancellation; voluntary release unmaps with the local admission gate (§2.2.7.3).

Direct IO Completion Semantics

DIO has no page cache, so page locks cannot serialize it. In v1 a DIO waits for every mirror RPC to complete or reach a resolved terminal failure before returning and releasing its buffer and ordering protection, under all three secondary policies (wait-indefinitely has no timeout). The return code is the primary’s; secondary failures go into the cancellation report. A successful DIO is durable on the primary, and every participant’s updates are committed or recorded as failed, regardless of O_SYNC/O_DSYNC (§2.2.2); it does not wait for other writers or for the epoch to close. Returning while secondary work continues would need protected data copies and ordering against later IO, and is future work.

Implications for Erasure Coding

The BRW page fan-out and page consistency mechanisms are intended to carry over to immediate EC writes, though significant design work remains. EC requires absolute consistency between source data pages and computed parity pages — primary locking provides ordering, and the tree lock (rounded to raidset-aligned extents) would ensure source pages are stable during parity computation. Tentatively, EC parity pages would be dispatched to their target OSTs via the same LOV-level call-up used for IWM secondary mirror fan-out, reusing the completion barrier infrastructure. Parity computation timing, partial-stripe writes, and the interaction between EC raidset geometry and the fan-out path remain to be designed.

Compression and Encryption Across Mirrors

Ideally we would not have to compress or encrypt data more than once when sending to multiple servers Compression is per component in the layout (lcme_compr_type, lcme_compr_chunk_lum_bits), so mirrors can differ in algorithm and chunk size, or be compressed and uncompressed. Because fan-out rebuilds each secondary’s RPCs through its own layout, each secondary’s OSC compresses with its own settings and heterogeneous compression needs no special protocol. One requirement follows from how client-side compression (CSDC) works: chunk alignment is done by the client at IO start, not by the OST. For an unaligned write, lov_io_init computes the leading and trailing partial chunks, the client reads them into cache, and vvp_io_rw_lock rounds the extent lock to whole chunks (TLC tree, CSDC series). That rounding uses the selected mirror’s chunk size today; for IWM it must use the largest chunk size among the participants. Chunk sizes are powers of two, so aligning to the largest aligns to all, and CSDC already computes that maximum across layout entries (ci_max_compr_chunk_bits). The cache then holds whole chunks for every mirror, and no secondary needs a read-modify-write inside writeback.

Bounce-page reuse. Fan-out runs after the primary has built its bounce pages. Encryption IVs come from the logical file block, not the OST object (osc_request.c:1416-1418 at 47638add78), so encrypted pages are the same for every mirror. A secondary whose compression settings match the primary’s, or where neither is compressed, reuses the primary’s bounce pages as-is; one with different settings compresses and then encrypts from the clear pages itself. Compression work is therefore done once per distinct setting.

Other Modifying Operations

Truncate, hole punch, fallocate and some setattr variants also modify OST objects and must be applied to every participant. They carry no data pages, so they fan out at the LOV sub-io level instead of through BRW fan-out.

Operation Classification

Duplicated to every participant (OST object operations):

• Truncate — OST_PUNCH over _size, EOF), mapped to the affected stripes of each mirror. ll_setattr_raw() first sends an MDS RPC (the MDS approves the size change, including ETXTBSY), then cl_setattr_ost() performs the OST side; only the OST side is duplicated.
• Fallocate — preallocation (mode 0, KEEP_SIZE), hole punch (FALLOC_FL_PUNCH_HOLE) and ZERO_RANGE (ll_fallocate() supports these; other modes return -EOPNOTSUPP), all through OST_FALLOCATE.
• Timestamp setattr (ATTR_MTIME, ATTR_ATIME, ATTR_CTIME) — via osc_setattr_async.

Not duplicated by the client:

• chmod stays on the MDS. chown, chgrp and project ID are pushed by the MDT itself to every stripe of every instantiated component of every mirror, including STALE ones, asynchronously through the OSP llog, for quota accounting (LA_REMOTE_ATTR_SET, lod_attr_set at 47638add78). That already covers all mirrors and changes only ownership metadata, so no client fan-out or epoch interaction is needed.
• setstripe and other layout changes interact with IWM through layout and epoch coordination, not duplication.

FIEMAP is a query, not a modifying operation: for IWM files it reports the primary mirror’s allocation only, takes no AW and does not close the epoch. Existing FLR FIEMAP ([source (lustre/lov/lov_object.c:2148)) iterates components across all mirrors, so IWM must select the primary explicitly; FIEMAP_FLAG_SYNC keeps its normal writeback behaviour (source (lustre/llite/file.c:3250)). Non-IWM FLR behaviour is unchanged (implementation: lustre-design-docs-iwm.10.1).

LOV Sub-IO Fan-Out

In FLR today these operations use the CIT_SETATTR path: LOV selects one mirror (lov_io_mirror_init sets lis_mirror_index), creates sub-ios only for its stripes, and a write intent stales the other mirrors; resync copies the result later.

For IWM they execute on every participant immediately. Clean non-immediate mirrors are excluded, with their affected components marked STALE through the MDS write intent before the operation runs. The fan-out point is lov_io_iter_init: it creates sub-ios for the overlapping stripes of every participant, each calling the per-stripe RPC (osc_punch_send for truncate, osc_fallocate_base for fallocate and punch, osc_setattr_async for timestamps). Today lov_foreach_io_layout(ind, lio, ext) takes the mirror implicitly from lio->lis_mirror_index (via lov_io_layout_at()); for IWM the iteration covers every participant. Offsets are translated per mirror (lov_size_to_stripe), so heterogeneous layouts map the same file offset to different object offsets correctly. Since the parameters are known when the cl_io starts, all sub-ios are created up front, with no call-up from OSC.

AW Lock and Epoch Interaction

These operations are admitted exactly like writes (layout, then AW reference, then layout revalidation) and run inside the epoch. Per-mirror errors go into the cancellation report: if a truncate succeeds on the primary but fails on a secondary, that secondary keeps the wrong size and becomes STALE at close, and resync copies the primary.

Before a truncate or punch, the client flushes dirty pages in the affected range; under IWM that flush goes through BRW fan-out, so earlier buffered writes reach every mirror before the truncate sub-ios run.

FLR’s write intents (LAYOUT_INTENT_TRUNC, LAYOUT_INTENT_WRITE for fallocate) remain in v1, both for component instantiation and staling of non-immediate copies and for non-IWM clients where they are allowed (optional primary-only writes, §2.4.6). Folding them into AW admission is a possible future optimization, but it would still have to stale omitted non-immediate components before the operation.

Open item (A7): the exact range-flush ordering, completion and error contract for truncate, punch and fallocate (§4.1).

Implications for Erasure Coding

The LOV sub-io fan-out for truncate/punch/fallocate extends naturally to immediate EC. Where IWM creates sub-ios per-mirror-per-stripe, EC would create sub-ios per-parity-group. Truncate and punch are particularly relevant for EC because they can create partial-stripe boundaries that require parity recomputation — a stripe that was fully allocated may become partially punched, requiring the parity stripe to be updated. The sub-io fan-out provides the dispatch infrastructure; the parity computation logic layers on top.

Error Management

Secondary Timing Policies and Administrative Abandonment

Three secondary IO policies, selected per client mount (not per file or in the layout), with finite slow-fail as the default:

1. Wait indefinitely: keep retrying and recovering with no timeout. Outstanding work can survive an arbitrarily long outage; an unrecoverable error or administrative abandonment still ends it.
2. Slow-fail: a longer finite budget; when it expires, safely resolve outstanding activity and record the secondary failed.
3. Fast-fail: the same model with shorter timings; it gives up sooner on a transient outage, staling the mirror and requiring resync.

All three share the same durability, terminal-failure and safe-resolution rules; slow and fast differ only in timing. Different mounts may choose differently without changing shared mirror-state rules. A finite budget must reach every underlying wait: submission, reconnect, recovery and replay, the modifying RPC, and commit/sync including OST_SYNC during forced closure. Per-RPC timeouts that restart indefinitely do not bound it. Expiry starts terminal resolution; it does not promise that the syscall or fencing finishes by then. Fast-fail is the first development milestone, for predictable fault-injection tests; slow-fail is required for the first supported release.

Terminal errors. A retryable event is not a mirror error if recovery completes the operation and its durability. An error becomes terminal when a finite budget expires, normal processing returns a terminal failure, or an administrator abandons the secondary, at any stage (submission, transport, the RPC, or commit/sync). It then stays recorded for the epoch, and the client promptly resolves its IO and releases AW with the report; the MDS closes the epoch and marks the mirror STALE. A late success of one RPC cannot clear the epoch’s mirror failure.

Safe resolution. A timeout, administrative request or error bit does not by itself make outstanding IO safe: retries must be stopped and activity completed, cancelled or fenced so that it can no longer touch source buffers or interfere with later epochs or resync. Buffer and ordering protection stays until then. No policy weakens the durability required of mirrors reported clean.

Space and quota. Secondary ENOSPC/EDQUOT is handled when the secondary write is attempted, as a secondary error (whole-mirror STALE, primary result unaffected). The primary’s grant and buffering are unchanged, and there is no advance reservation on the secondaries in v1. Implementation: lustre-design-docs-od5.49.8; advance reservation as a later improvement: lustre-design-docs-od5.56.

Application results. A secondary failure never turns a successful primary result into an application error, for DIO, fsync(), fdatasync(), O_SYNC or O_DSYNC. All policies keep the completion barriers: DIO and explicit durability requests need committed-or-recorded-failed outcomes for their scope, and buffered writes keep page protection during fan-out. Slow-fail can stall DIO, sync requests, writeback or epoch close longer than fast-fail; wait-indefinitely can stall them without limit. None waits for epoch close. Primary failures follow the separate primary rules (§2.4.4).

Changing the policy affects new IO only; operations already waiting keep their policy and timings, and are not woken or converted. A change never clears accumulated errors or releases outstanding protection, and does not alter primary IO behaviour.

Administrative abandonment is part of declaring an OST dead: a client-side action targeting that OST that reaches its outstanding secondary work, including wait-indefinitely IO, invocable on one client or triggered on all clients when an OST is declared dead cluster-wide. It is not a per-file protocol or a policy change. Affected files then follow the normal terminal-error and epoch-close rules.

Terminal secondary failures reduce redundancy while the primary stays usable; failed mirrors appear in the changelog and are restored by normal FLR resync. Slow-fail trades longer waits for a chance to avoid that; fast-fail accepts it sooner. Returning from DIO while secondary retries continue, or replicating after completion, would need protected source copies and ordering against later writes, and are future designs.

Open items (lustre-design-docs-od5.33): numeric timings, timer start and accounting, control names, and the policy sampling point for new IO; the abandonment interface and target identity, distribution and acknowledgement, and how disconnected, reconnecting and new clients learn of it (sending a notification is not proof of cluster-wide completion); the dead-OST state’s lifetime, fencing and reactivation, so abandoned work cannot resume against resync or new epochs; an OST that also holds primary objects (primary failures must not be hidden as secondary-only); delivery milestone.

Failure Handling

Mirror States: STALE and INFLIGHT Flags

At epoch entry (the first AW request), the MDT selects the primary by the usual policy, moves the file to WRITE_PENDING and sets INFLIGHT on every other participant. Entry does not clear pre-existing STALE or make an unsynchronized copy eligible, and non-immediate mirrors are not marked INFLIGHT. The AW lock tracks that the epoch is active.

INFLIGHT means temporary exclusion from reads during an epoch, not that some RPC is outstanding. It stays set between writes until the epoch closes, when mirrors that were synchronized at entry are expected to be consistent again. IWM clients exclude INFLIGHT mirrors explicitly, and patched 2.18/master and 2.15 clients exclude them as unknown mirror-state flags. Unpatched clients ignore unknown flags (layout verification checks only sizes and offsets; selection tests only STALE, PREF_RD, INIT and PARITY; lov_ea.c:441-470, lov_object.c:689, 721 at 47638add78), which is why the MDT refuses their opens (§2.4.6).

STALE means the mirror is out of sync by an unknown amount; its contents must not be used until a full resync. It is the same flag delayed-write FLR uses. Participants use INFLIGHT during normal writing, and become STALE only through errors or client loss at close. Pre-existing STALE survives entry and close until resync. With the optional primary-only write setting, every non-primary mirror, IMMEDIATE ones included, becomes STALE before the write is admitted.

Clean non-immediate mirrors. IMMEDIATE describes write participation, not synchronization: a non-immediate mirror may be clean after a resync or after removal from the immediate set. Before admitting any modifying IO that omits such a mirror, including IO by IWM clients on the participants, the MDS marks its affected components STALE under ordinary FLR rules (§4.2).

Participants’ flags are settled at epoch close (§2.2.4), which is separate from staling omitted non-immediate components before IO. STALE mirrors are visible to lamigo through changelogs (§2.4.5).

Write/Update Error from Client

Clients associate errors from every modifying operation with the epoch. Retryable secondary events follow the mount’s secondary policy; one that recovers does not stale a mirror. A terminal error at any stage (submission, transport or recovery, the RPC, or commit/sync), or administrative abandonment, is recorded as a mirror failure. The client then resolves its remaining IO and releases AW without waiting for the retention timer, reporting the error and triggering closure (§2.3.3.1).

Client Eviction from OST

Eviction from an OST can invalidate that mirror’s completeness and commit guarantees at any point, not only at commit. The client reports the mirror failed in its AW cancellation, after safely resolving its outstanding IO, and the mirror becomes STALE at close, for resync by lamigo or other tools. Regaining the connection later does not make an eviction-damaged mirror clean, whatever the policy.

Client Eviction from MDT

If a client holding AW is evicted from the MDT, its outcomes are unknown. The ordered close drains and flushes the responsive writers, not the evicted one. The outcome follows the lost-participant rule (§2.2.4): resolve a known primary error first, then every mirror other than the resulting primary becomes STALE in full, regardless of reported error detail, with the primary kept as a best-effort copy that may lack some of the lost client’s writes. Late writes from the evicted client are fenced by the FLR era-ending version push, not an extent-lock flush; sequencing is open in lustre-design-docs-iwm.6.1.

Client Loss

A client that crashes or otherwise disappears is evicted from both the MDT and the OSTs; the MDT eviction governs, so this is handled exactly as above.

MDS Failover

On MDS restart, durable tracking identifies open epochs, participants and reported errors; after ordinary lock replay and reconciliation, affected epochs are rolled up with the ordinary close rules, and any participant that did not recover is a lost participant on its own files only. See Recovery After MDS Failover.

Primary Mirror Write Failure

If writes to the primary fail, clients still complete the other mirrors’ writes and report as usual, and the MDS closes the epoch normally (§2.2.4). It selects a replacement primary from the eligible participants with no known error (from collected reports and durable error records, by the usual policy) and marks the failed old primary STALE. This happens before any lost-participant rule, which is then evaluated against the new primary. A clean non-immediate copy, such as demoted flash, is not eligible without conditional promotion. The outcome is part of the atomic closing transaction.

When no replacement is eligible (every candidate failed or was already STALE): report the errors through the usual logging and IO error paths, keep the epoch’s existing primary as the sole read-eligible copy, clear INFLIGHT and mark every other mirror STALE; a previously STALE secondary is never rehabilitated. This is best effort: it does not mean the primary is complete or that the failed writes succeeded, and normal access may still hit IO errors. Resync uses the primary’s current contents to restore redundancy but cannot recover missing updates. No new degraded flag or administrator source selection is involved. The rationale is to match the existing best-effort behaviour for ordinary write errors without IWM. The same outcome applies in live closure and recovery.

Replacing Mirrors/Permanent Failure

When a mirror is left STALE after a write failure, lamigo is notified through changelogs, which it already watches for STALE mirrors as part of Hot Pools; no new notification path is needed. Changelog overhead is modest (5-10% for metadata-intensive workloads, less for IO-intensive ones). If resync keeps failing, for example because an OST has failed permanently, lamigo can add a replacement mirror (lfs mirror extend) and the failed mirror can be removed later. No special commands are needed.

These operations quiesce writers through the administrative LAYOUT EX → AW EX path (§2.2.7). Resync takes a lease (LL_LEASE_RESYNC) and its resync request takes LAYOUT EX on the MDT; AW EX then closes any open epoch, draining writers under their existing protection. If that epoch, or an earlier one, lost a participant, resync confirms the era-ending version fence on the source mirror’s objects before copying (§2.2.4). Resync instantiates the components it needs; if that keeps failing, it fails and leaves the mirror STALE.

New writers stall on layout acquisition (and AW where applicable) until the operation completes; that is not a write failure. Adding a mirror and resyncing it are separate commands, so there are two quiesce windows, acceptable for an infrequent recovery operation.

Conflicting Operations During Write Epochs (Resync, Old Clients)

Compatibility policy:

1. Clients we can modify (2.18/master and 2.15) treat an unknown mirror-state flag as making the mirror unusable, as if STALE, so a client that does not understand INFLIGHT skips the mirror rather than reading an inconsistent copy. They advertise this with a new connect capability, distinct from OBD_CONNECT2_FLR_IMMED_MIRROR (full IWM support; already reserved at lustre_idl.h:855). This is a local selection rule: the client never writes STALE into the layout or clears unknown flags. Full IWM clients advertise the capability too.
2. Clients without the capability cannot open a file that has any IMMEDIATE mirror, whatever the file’s state (RDONLY, WRITE_PENDING, with or without AW holders). Existing pre-FLR exclusion is unchanged.

Known flags. A skip-unknown client must treat IMMEDIATE as known membership metadata (which does not imply AW support), or the rule would also exclude the primary. The mask covers mirror-state flags only, not arbitrary component flags or layout formats. INFLIGHT may stay unknown and be skipped; a client that recognizes it must exclude it the same way. Unknown state may not be ignored in read selection, size and attribute queries, or any fallback path that would silently pick an excluded mirror.

Non-open access by clients without the capability. Data access needs an open, so the open denial covers all reads and writes. Metadata paths without an open (stat/ls -l via glimpse, getxattr, lfs getstripe) stay allowed. During an epoch such a client may glimpse an INFLIGHT secondary or fold it into its size (the maximum over non-STALE components of all mirrors, lov_attr_get_composite) and briefly report a size slightly behind or ahead of the primary. That is transient, limited to the epoch, exposes no data, and is accepted. Withholding the layout (which would break stat) and per-client layout rewriting were rejected; old clients are not a design target.

Capable clients without full IWM support. They read the primary and never force epoch closure just to read. Writes are denied by default. An optional setting allows primary-only writes through the existing FLR write-intent sequence: any open epoch is closed with LAYOUT EX → AW EX, the MDT changes the layout and marks every non-primary mirror (IMMEDIATE ones included) STALE, then the primary-only write proceeds under ordinary FLR layout, extent-lock and resync serialization. That deliberately gives up immediate redundancy until resync; it is not participation in an IWM epoch, and a later clean IWM epoch never clears that STALE. The setting never admits clients without the capability.

Promoting IMMEDIATE on an open file is a migration-style protected operation, not a raw flag change: it reuses existing migration quiescence, layout invalidation and revalidation, and serialized publication; closes any open epoch with LAYOUT EX → AW EX; and keeps exclusion through the conditional clean-copy and membership checks (§4.2) until publication. Promotion never clears STALE, and needs no data copy or temporary inode when the intended copies are already clean. Ordinary migration contention, retry and failure handling apply.

Existing descriptors are not grandfathered. Migration preserves file identity and open handles, so applications are not forced to reopen. Supported holders continue through normal layout refresh and AW admission. Holders without the capability must not obtain or use the promoted layout through an old descriptor, cached layout, PW or mmap authority, or reconnect and replay: the capability gate is enforced on layout return and refresh and on IO admission, not only at open. This denies unsupported access; it does not destroy descriptors or evict clients.

Migration precedents at 47638add78: blocking migration (lustre/utils/lfs.c:1085) takes a group lock and swaps layouts after a data-version check; nonblocking migration (lustre/utils/lfs.c:1608) uses a lease and an atomic swap at close; the MDT swap handler (lustre/mdt/mdt_handler.c:1839) takes LAYOUT/XATTR EX. These are precedents, not proof: stock migration provides neither the capability gate nor the AW drain, the blocking path takes its group lock before MDT LAYOUT EX (not automatically compatible with the AW order), and the swap handler checks layout support only for the requester.

Resync closes any open epoch first, whatever the client’s capability, then follows normal resync serialization (§2.4.5); an ordinary read needs neither.

Deferred: state-conditional access for clients without the capability. Allowing them to open files only between epochs would not make their activity vanish when the next epoch opens: they could keep descriptors, layouts, cached PW or mmap authority, dirty data, RPCs and retries without any AW lock, and revoking a soft layout is not a writeback or durability barrier. It would need a server-driven drain and fence mechanism that has not been designed.

Open items (lustre-design-docs-od5.40, lustre-design-docs-iwm.1, checklist P4): the capability’s wire bit and the mirror-state mask; enforcement points and denial errors across open, create-with-layout, layout return and refresh, reconnect and replay, serialized with layout and membership changes; invalidation of reads already in flight at entry; the primary-only write setting’s name, scope and enforcement; for promotion, the group/lease/AW placement, draining legacy PW, mmap, dirty data and IO (including holders and older clients without layout-invalidation support), and making sure migration exclusion never blocks admitted IO or the cancellation flush; partial-component mirror handling.

Future Enhancements and Alternatives

Cancellation LVB Format and Future Stripe Detail

The v1 cancellation report carries mirror-level errors only, but uses an extensible structure from the start, so that per-OST-object detail can be added later for finer repair and immediate EC without changing the mandatory summary or the conservative whole-mirror fallback.

Message structure (v1):

• A versioned header with explicit header and total-message lengths, enough to find the mandatory summary and any extensions, plus the epoch ID (the opening layout generation; §2.2.1).
• A mandatory count followed by that many failed mirror IDs (16-bit, as in lcme_id). A count of zero is a clean report. Membership changes need an epoch boundary, so IDs keep their meaning for the whole epoch.
• An optional sequence of typed, length-delimited detail records. V1 sends none, but the framing exists so a receiver can skip unknown optional records within a supported version and still process the summary. An unsupported top-level version is not covered by that skipping rule.

The receiver validates lengths, the count, the epoch ID, and that each ID is a participant of that epoch, then takes the union of all valid reports; any reported mirror becomes STALE as a whole. A missing or invalid report is never treated as a clean one; it gets conservative handling. Field widths, alignment and message limits are specified with the RPC implementation. The current LDLM_CANCEL request carries no LVB; adding the field, associating reports with the right lock in batched and early cancellations, and packing and validation are v1 protocol work.

Sizing. The summary is a count plus 2 bytes per failed mirror, bounded by the mirror count (LUSTRE_MIRROR_COUNT_DEF 16, LUSTRE_MIRROR_COUNT_MAX 256); a clean report carries only the count.

Future per-stripe detail. Per-stripe error recording is TBD, including how failures are collected, stored durably, aggregated across layout versions, replayed and consumed; v1 durable tracking records failed mirrors only. A future detail record would identify the mirror, the layout mapping that defines its index space, the number of slots, and an error bitmap, under these rules:

• One flat index space per mirror. Concatenate the stripe lists of the mirror’s components in order and number the OST-object slots from zero: with stripe counts n0, n1, ..., local stripe s of component k has index n0 + ... + n(k-1) + s (two components of 2 and 3 stripes occupy 0–1 and 2–4). Components without OST objects contribute nothing. This counts object slots, not file bytes.
• Layout changes. Indices are interpreted against the layout under which the IO was issued. Since an epoch can span layout changes, client and MDS must keep or translate that mapping before combining detail; without a precise translation, fall back to the whole-mirror summary. This does not force epoch closure for component instantiation.
• Merging. A detail-aware receiver may refine repair only with complete, valid detail from every relevant report; missing or unsupported detail never makes a reported mirror look clean.
• Size. A bitmap for N slots is ceil(N / 8) bytes before record overhead (25, 63 and 250 bytes for 200, 500 and 2000 slots). LOV_MAX_STRIPE_COUNT (2000) limits one plain component, not a mirror’s total, so 16 × 2000 is not a realizable worst case; the RPC implementation must derive and enforce a real bound, falling back to the mirror summary when detail does not fit.

Per-extent reporting (byte ranges of failed writes within a stripe) is a further level whose size grows with the number of disjoint writes; it is not part of v1 or the per-object extension.

Write Ordering Alternatives

v1 uses the primary mirror as locking leader, requiring client-side LDLM locks for all IO including DIO — a performance regression for shared-file DIO workloads that normally operate locklessly. Possible future approaches to restore lockless DIO:

Chained RPC Checksums

One option is to have OSTs maintain a chained checksum of committed writes. In the simplest form, this requires identical RPCs to all mirrors, which requires mirrors to have identical layout geometry — a constraint the v1 BRW page fan-out approach was specifically designed to avoid (see IO Duplication). However, the chained checksum concept could be extended to work with heterogeneous layouts by checksumming at the logical file offset level rather than the RPC level — the OST would chain checksums of (offset, length, data_checksum) tuples as they commit, producing a layout-independent ordering fingerprint. This would require the MDS to compare per-stripe chains after mapping them back to file-level extents, adding complexity but removing the identical-layout restriction. When a write epoch opens, the OST would be informed the stripe object is part of an immediate mirror file. It would take the write checksum from each committed write and chain them together as they are committed (write commits can be ordered by their journal transaction number, even if they are occurring in parallel). The result is a single checksum value which encodes both the writes and their ordering. Non-write operations can be included by checksumming their arguments (e.g., the byte range and fallocate op type for fallocate).

The write primary mirror would be the correct ordering, to which the others are compared. If ordering on a secondary mirror disagrees with the primary, that mirror would be left STALE (not un-staled), requiring a full data resync.

Open questions for this approach include: whether to apply this only to overlapping operations (which requires tracking the extent of all data-modifying operations through a write epoch, but avoids false positives from non-overlapping writes), and recovery scenarios when an OST fails mid-epoch and the checksum chain is incomplete.

A related but heavier-weight idea is having OSTs maintain full write commit vectors (ordered lists of committed write extents) and comparing them across mirrors at epoch close. This subsumes chained checksums but requires significantly more per-OST state and MDS-side comparison logic. # Policies and Open Items {#open-problems}

Operation Completion Consistency Model

Application return, RPC completion, durable commit, AW release and epoch closure are distinct events. The v1 contracts:

Boundary	V1 contract
Ordinary buffered write()	May return once data is in the client cache, before any mirror RPC. Normal asynchronous aggregation; no stable-storage promise.
Buffered dirtying / mmap write permission	Each modifying IO takes AW at admission, after layout validation and before PW use or dirtying, not at writeback. Dirty data stays the epoch’s responsibility across syscall return and layout refresh.
Mirror RPC / writeback completion	Every mirror attempt has completed or been safely resolved before source buffers and ordering protection are released. A reply is not proof of durable commit.
DIO completion	Stable storage without synchronous flags: every participant’s updates committed or recorded as failed, outstanding activity resolved. The primary determines the result.
fsync(), fdatasync(), O_SYNC, O_DSYNC	For the operation’s data/metadata scope, committed-or-recorded-failed outcomes with outstanding activity resolved. Primary success is success despite secondary failure.
AW release	Drain admitted IO (including IO that has not dirtied yet), flush the whole file, and synchronously confirm durability on every mirror reported successful, with failures resolved and reported. No opt-out; never with dirty data remaining.

Explicit durability requests keep the usual distinctions between full and data-integrity synchronization, and between file flushes and synchronous writes. Primary writeback and commit errors, relevant earlier buffered-write errors and required metadata errors are reported normally. A secondary-only failure is never surfaced on a later fsync() when it would be invisible for the write itself. V1 has no “durable redundancy” success requirement; an explicitly requested stricter policy could be added later, consistently across APIs.

No global-close barrier. Completing an operation never requires other clients to release AW, the epoch to close, or the final layout to be published; a durability request does not take AW EX or wait for unrelated writers. AW may stay held for later writes. Local commit and error tracking survives operation completion until reported, and durable tracking with conservative recovery covers an outcome lost before reporting. A successful fsync() does not make secondaries readable; only epoch close does.

Combining Immediate and Non-Immediate Mirrors

Real deployments combine immediate and non-immediate mirrors on one file. For example, DDN’s Hot Pools places latent mirrors on HDD that are filled by resync (lamigo) once a file ages out of flash: a file might have two flash mirrors written immediately for availability and two HDD mirrors synced lazily for capacity.

The IMMEDIATE flag alone determines participation; no new “immediacy group” or file-level layout state is needed, and RDONLY → WRITE_PENDING → SYNC_PENDING stay as they are. Non-immediate mirrors are not necessarily STALE: resync can leave HDD clean before promotion, and demotion can leave flash clean but no longer immediate. Before any modifying IO that omits such a copy, its affected components are marked STALE (§2.4.1), which is what makes conditional promotion safe: a copy that missed a write never still appears clean. Adding and removing IMMEDIATE are conditional layout operations at epoch boundaries that change who participates in later epochs, never the participants of a running one.

Tiering lifecycle (e.g. flash to HDD):

1. The file starts with two immediate flash mirrors.
2. HDD mirrors are added without IMMEDIATE; they need resync.
3. When the file ages out:
   1. Resync HDD with ordinary resync. That makes the copies clean at completion, not afterwards.
   2. Conditionally promote HDD. Close any open epoch and resolve its outcomes; then, while excluding modifying IO and competing layout changes, check that the intended HDD copies still exist and are clean over the required extents, and set IMMEDIATE in the same serialized transition. A userspace check followed by an unconditional flag update is not enough. If a write staled the copies meanwhile, promotion fails and the caller resyncs and retries. Promotion never clears STALE.
   3. Keep both tiers immediate during the overlap. Later epochs write all healthy flash and HDD participants, so there is never a window without immediate copies; failures can still stale a participant.
   4. Conditionally demote flash. Close the overlap epoch and process its outcomes. Clear IMMEDIATE on the flash mirrors only if the retained immediate HDD copies still provide the clean coverage and redundancy required; an IMMEDIATE flag alone is not evidence of a healthy copy. Otherwise leave flash immediate. If flash was the primary, an eligible HDD primary is selected before the next epoch.
   5. Remove flash copies separately, after revalidating that the retained copies make removal safe; an earlier successful demotion does not authorize deletion regardless of later failures.

Serialization. Promotion and demotion take LAYOUT EX, then AW EX: invalidate old layout views, drain and flush writers, check conditions, and publish the new membership before admitting writes under it. AW EX alone would not suffice if clients could resume with an old fan-out. Clients that never held AW are handled by the capability gate and the migration-style promotion of open files (§2.4.6).

Progress. Copying, promotion, demotion and removal are separate operations, and stopping between them leaves a valid state (HDD copied but not promoted, both tiers immediate, or flash demoted but not removed). Continuous writes can re-stale HDD between resync and promotion indefinitely, so v1 gives no progress guarantee on a continuously written file; promotion needs a write-quiet interval. A future resync-and-promote operation could hold serialization through promotion (it would still need writers to pause).

Interaction with resync. Resync of a non-immediate mirror takes LAYOUT EX, then AW EX to close any open epoch, and keeps normal resync serialization through the copy; afterwards writers may resume and re-stale the copy before promotion, which is why promotion is conditional.

Administrative redundancy guard. A demotion or removal that would leave fewer than two healthy IMMEDIATE copies requires an explicit force override. Removing or demoting a copy that is already STALE or otherwise not counted does not reduce healthy redundancy and needs no force. The count is taken after the ordered drain, rollup and primary processing, in the same serialized transition as the membership change, counting clean, valid coverage rather than IMMEDIATE flags. Force overrides only this guard: never locking, drain, error resolution, retained-copy validity or safe primary selection, and it cannot clear STALE. Demoting the last IMMEDIATE participant still requires a valid retained copy and resolution of the old AW obligations. The guard does not imply a two-copy minimum at creation and does not prevent failure-induced degradation.

Open items (lustre-design-docs-od5.44, lustre-design-docs-od5.40): conditional promotion and demotion in the layout-management interface and tiering controller, with retries; the force override’s API and CLI encoding, denial errors and where it is enforced; tests of post-drain counts, cleanup of degraded files and force without weakening retained-copy validity.

Heterogeneous Compression Across Mirrors

Heterogeneous compression across immediate mirrors needs no special protocol: each secondary compresses with its own settings when its RPCs are rebuilt, provided client-side chunk alignment uses the largest chunk size among participants. See Compression and Encryption Across Mirrors.

Open Items Summary

Everything still open, by area; details are in the sections cited. The shared protocol checklist (iwm_discussion_checklist.md, P1–P8, A7, A9) tracks the verification work.

• Questions needing a decision
  • Andreas Dilger: recovery rollups as a final recovery phase or as ordinary post-recovery operations; recovery hook, cancellation availability and service-resumption timing (§2.2.5).
  • Patrick: failure-only close residue: how recovery treats a committed closing layout whose bookkeeping cleanup failed (§2.2.5, lustre-design-docs-od5.61).
  • Whether close reuses SYNC_PENDING internally or an equivalent era-ending step (§2.2.4).
• Client admission and AW lifetime (P1–P3; lustre-design-docs-od5.8, lustre-design-docs-od5.36): admission hook placement and coverage of every path (§2.3); IO reference boundaries and the final-flush barrier; voluntary-release gate; mmap fault/PTE handoff and PCC mkwrite; cached PW/AW lifetimes; retention timer value and controls; local lock dependencies in pruning and fan-out.
• Compatibility (P4; lustre-design-docs-iwm.1, lustre-design-docs-od5.40): capability bit and mirror-state mask; enforcement points and errors; old-read invalidation at entry; primary-only write setting; migration-style promotion integration.
• Epoch entry and grants (P5–P6; lustre-design-docs-iwm.3.2): reply representation for the two grants; ordered handoff serialization; lost and partial replies, retry and replay.
• Release, close and recovery (P7–P8; lustre-design-docs-od5.21, lustre-design-docs-od5.29, lustre-design-docs-od5.30, lustre-design-docs-od5.61): durable error-bearing CANCEL delivery; lock progress during recovery; storage schema, identity and shard design; failure-safe execution order; era-ending fence sequencing (lustre-design-docs-iwm.6.1); atomic close and dependency ordering verified on ldiskfs and ZFS; handoff authority after closure.
• Secondary failure policies (lustre-design-docs-od5.33): timings, controls, sampling; abandonment interface, distribution and reconciliation; dead-OST lifetime and reactivation.
• Other operations and durability (A7, A9): truncate/punch/fallocate ordering and error contract; DIO flag preservation through fan-out; both-backend tests.
• Implementation follow-ups with design impact: write-duplication rework of Gerrit 63244 (lustre-design-docs-iwm.4.1); STATE_ONLY_CHANGE with per-mirror prune (lustre-design-docs-iwm.3.3); all-participant component instantiation (lustre-design-docs-od5.49.12); DoM exclusion and admission-path audit (lustre-design-docs-od5.49.14); secondary space accounting (lustre-design-docs-od5.49.8); tiering controls and the redundancy guard (lustre-design-docs-od5.44).
• Existing FLR bug found by this design: resync can mark mirrors in sync after a stale writer’s late write (lustre-design-docs-od5.49.31; §2.2.4).
• TBD by design: per-stripe error recording and repair (§3.1); restoring lockless DIO (§3.2); state-conditional access for clients without the capability (§2.4.6).

Decision Log

Decisions recorded with their dates. The sections above state the current rules; this log records when each was settled and what it replaced. Beads are in the lustre-design-docs tracker.

Date	Decision	Where	Replaces
2026-09-08	Application completion contracts A1–A6 (buffered return, dirty-data ownership, RPC completion, DIO durability, explicit sync with degraded success, no global-close barrier)	§4.1	—
2026-10-02	Interrupted epoch opening uses ordinary recovery: after replay and reconciliation, listed epochs with no recovered holders close through the ordinary close	§2.2.5	A separate interrupted-opening outcome or admission acknowledgement
2026-10-02	The first AW request opens the epoch (primary selection, WRITE_PENDING, INFLIGHT, durable active-epoch record); later requests join	§1.2, §2.4.1	Epoch entry driven by a later write intent
2026-10-02	Strict LAYOUT → ACTIVE_WRITERS ordering on client and server; soft client layout hold; component/SEL changes are layout-only	§1.2, §2.2.7	AW-first ordinary close
2026-10-02	Capability-based compatibility: patched 2.18/master and 2.15 clients skip unknown mirror-state flags and advertise it; others cannot open IMMEDIATE files in any epoch state	§1.2, §2.4.6	Blanket epoch closure on every non-IWM layout request; state-conditional access (deferred)
2026-10-02	Capability-bearing non-IWM clients: writes denied by default; optional setting allows primary-only writes that stale every non-primary mirror via ordinary FLR intents	§1.2, §2.4.6	—
2026-10-02	Epoch close uses LAYOUT EX → AW EX; AW cancellation drains old IO and flushes the whole file without taking new locks	§2.2.4, §2.2.6	AW-first close; layout reacquisition during drain
2026-10-02	Entry reply returns the updated layout and a replacement LAYOUT grant with the independent AW grant; reuse FLR’s layout handoff and client refresh	§2.2.7	Two round trips; LDLM lock conversion
2026-10-05	Durable participation recorded for every client’s AW acquisition, including joins	§2.2.5	FID-only active tracking with filesystem-wide degradation on client loss
2026-10-05	Mirror-error records are distinct from participation, carry no reporter identity, and are consumed only by the atomic close	§2.2.3, §2.2.5	Errors annotated on, or living with, participation
2026-10-05	Recovery order: load tracking, finish replay and reconciliation, then roll up; primary error resolved before the lost-participant fallback	§2.2.5	—
2026-10-05	Last voluntary release triggers rollup; any error-bearing release triggers it immediately; raced joins are drained; no “rollup pending” gate	§2.2.4	—
2026-10-05	Error-bearing cancellation: one MDT transaction records errors and retires participation, durable before ACK	§2.2.3	—
2026-10-05	Close: one MDT transaction publishes the closing layout, consumes handled errors and removes the active-epoch record; no extra synchronous close commit	§2.2.4	Layout publication followed by separate cleanup
2026-10-05	Canonical client admission: LAYOUT, AW, revalidate; on layout loss drop the IO’s AW reference before waiting	§2.2.7	—
2026-10-06	Three secondary failure policies (wait indefinitely, slow-fail, fast-fail) differing only in timing; administrative abandonment is separate	§1.2, §2.3.3.1	Two-mode fast/slow model
2026-10-06	Reuse unmap_mapping_range() for voluntary/idle AW mapping revocation	§2.2.2, §2.2.7.3	A new mapping-revocation scheme
2026-10-06	Secondary policy per client mount, default finite slow-fail; changes affect new IO only	§2.3.3.1	Per-file policy
2026-10-06	Administrative abandonment is an OST-targeted client action, local or triggered cluster-wide when an OST is declared dead	§2.3.3.1	Per-file mirror abandonment
2026-10-06	Promoting IMMEDIATE on an open file is a migration-style protected operation; old descriptors are not grandfathered	§2.4.6, §4.2	Refuse promotion until incapable holders close
2026-10-07	No eligible replacement: keep the existing primary as the sole best-effort copy	§2.4.4, §2.2.5	All mirrors STALE
2026-10-07	Unmatched replayed AW is rejected and handled conservatively, not an LBUG (review F-13)	§2.2.5	LBUG
2026-10-07	Logically separate record kinds; in-memory client lookup rebuilt on restart; recovery rollups by bounded parallel workers with no progress journal	§2.2.5	—
2026-10-07	Storage mechanics deferred to detailed design; dedicated DT indexes are the preferred candidate	§2.2.5	—
2026-10-07	lfs mirror create --immediate marks all created mirrors IMMEDIATE	§1.2	—
2026-10-07	V1 scope: DoM excluded; NFS, PCC and HSM supported under normal admission and drain	§1.2	—
2026-10-07	Hybrid IO takes the DIO client lock; group locks allowed with their existing no-ordering contract (review F-07)	§1.2	—
2026-10-07	Synchronous durability at every AW release; no opt-out	§2.2.2	iwm_sync_on_epoch_close opt-out
2026-10-07	AW cache retention: ordinary LDLM semantics plus a separate shorter timer (about 60 s)	§2.2.2	Dedicated 1–5 s idle timer
2026-10-07	Drain then FLR era-ending version fence for lost participants (no MDT extent-lock flush)	§2.2.4	OST extent-lock flush as the fence
2026-10-07	Fence only lost-participant closes; close does not wait for delivery; copy-to-clean operations confirm the fence first (review F-03)	§2.2.4	—
2026-10-07	Epoch ID is the opening layout generation (review F-04)	§2.2.1	—
2026-10-07	Cancellation report carries failed mirror IDs, not a bitmap (review F-05)	§2.2.3, §3.1	Counted mirror bitmap
2026-10-07	Clean cancellations commit lazily by default; MDT commit cost model (review F-15)	§2.2.3	Optimization to evaluate
2026-10-07	No maximum epoch length (review F-16)	§2.2.2	—
2026-10-07	Release barrier built on per-file transno tracking against last_committed; DIO already durable at reply (review F-25)	§2.2.2	—
2026-10-07	Component instantiation covers every participant; a secondary instantiation failure is a secondary error	§2.2.7.2	Primary-only instantiation
2026-10-07	STATE_ONLY_CHANGE prunes sub-objects of mirrors that lose read eligibility (review F-01)	§2.2.7.3	Keeping all cached pages
2026-10-07	Secondary ENOSPC/EDQUOT handled when attempted; no advance reservation in v1	§2.3.3.1	—
2026-10-07	FIEMAP reports the primary only	§2.3.2.1	—
2026-10-07	Primary RPC slot freed at its reply; only completion waits for secondaries (review F-20)	§2.3.1.1	Prototype’s slot hold
2026-10-07	Ordering invariant: no primary extent lock release while a covering secondary attempt is unresolved (review F-06)	§2.3.1.1	—
2026-10-07	Heterogeneous compression via per-mirror rebuild with chunk alignment to the largest participant chunk; bounce pages reused when settings match (review F-09)	§2.3.1.5	“OST handles RMW”; bounce reuse deferred
2026-10-07	Gerrit 63244 cited as a feasibility demonstration with a deltas table (review F-21)	§2.3.1	“Proof of concept”
2026-10-07	Clients without the capability may still stat IWM files; transient size inaccuracy accepted (review F-11)	§2.4.6	—
2026-10-07	Administrative redundancy guard: demotion or removal leaving fewer than two healthy IMMEDIATE copies needs force	§4.2	Deferred to end of review