Migration draft. Copied from the committed local design in paf0186/lustre_design_docs (projects/immediate-write-mirroring/design/immediate_write_mirroring_design.md; source commit 57398ab674208365679706f5e5adcba48ea67470), with diagrams committed in 21b52e0. The original Confluence page is 373565003 on wiki.whamcloud.com. Historical discussions and server revisions are archived separately; they are not recreated as native wiki history here.

Overview & Design Summary

FLR Today

In the current implementation of FLR (introduced in Lustre 2.11), the system uses a "delayed write" approach for maintaining file mirrors. This means:

Writing to Mirrored Files: When a client writes to a mirrored file, only one primary (preferred) mirror is updated directly during the write operation. The other mirrors are simply marked as "stale" to indicate they're out of sync with the primary mirror.

Today, we have manual synchronization: after a write, the lfs mirror resync command must be run to synchronize the stale mirrors with the primary mirror. This command copies data from the synced mirror to the stale mirrors and removes the stale flag from successfully copied mirrors.

Layout state and staleness is managed through a careful series of layout state changes which are described File-level replication state machine in Core Design Concept in this document.

This delayed write approach was implemented in the first phase of FLR to avoid the complexity of maintaining consistency across multiple mirrors during concurrent writes. By updating only one mirror during writes and marking others as stale, the system maintains a consistent view of the file data, at the cost of requiring explicit synchronization after writes complete.

The current implementation does not provide immediate redundancy, since only on a single mirror is up to date until an explicit resync operation is performed. This approach enables things like hot pools synchronization, but the lack of immediate write redundancy severely limits the use cases.

Immediate Mirroring

We have a requirement to do immediate mirroring on Lustre files, where all writes (and related ops) are replicated to multiple mirrors immediately. The infrastructure created for this will also be used for immediate erasure coding.

Whether a mirror participates in immediate write mirroring is controlled by a per-mirror IMMEDIATE flag in the composite layout (lcme_flags), alongside existing flags like STALE and PREFER. The flag is set at layout creation time (e.g., lfs mirror create or lfs setstripe) or changed later through the conditional promotion/demotion operations described in §4.2. Mirrors without the flag are ordinary FLR mirrors — written lazily via resync — and do not participate in write duplication or AW lock epochs. Immediate and non-immediate mirrors can coexist on the same file.

In the IWM write, fan-out, and epoch descriptions below, “all mirrors” means the eligible immediate participants, including the immediate primary. Non-immediate copies are not brought into the fan-out merely because they are clean, and their staling follows ordinary FLR rules when a modifying operation omits them. Changing the immediate participant set requires an epoch boundary.

The goal is to have redundancy immediately, but during writes only a single mirror (the primary) is available for reads — see Primary Mirror Selection and Read Visibility for the rationale.

Non-IWM clients cannot participate in an active epoch. Before giving such a client a usable layout, the MDS closes any active epoch. After that close, an access/read request receives the completed RDONLY layout with healthy mirrors still clean; a write intent invokes ordinary delayed-write FLR and stales affected secondary components. Existing legacy WRITE_PENDING access outside an IWM epoch follows normal FLR rules. Starting another IWM epoch requires revoking legacy layouts and draining or fencing outstanding legacy writes first. See Old-Client Compatibility.

The core idea of the design is this:

To write an IWM file, the client first acquires the layout lock (sending a write intent to transition the layout from RDONLY to WRITE_PENDING if needed), then takes an Active Writer (AW) lock — a CW lock on a separate IBITS bit (ACTIVE_WRITERS) on the same per-file resource. The layout lock and AW lock are independent: layout changes (component instantiation, SEL extension) revoke LAYOUT but leave ACTIVE_WRITERS untouched, so the write epoch spans layout changes without interruption (see AW Lock vs Layout Lock Interaction).

The client sends all writes to all online mirrors in parallel. A single primary mirror will be selected for reads during this time, since we cannot guarantee all mirrors are identical during writes. This mirror is the "write leader", also used for write ordering (locks are taken on this mirror first). All other mirrors are marked INFLIGHT during writes — see Mirror States for the full flag model.

Clients hold the AW lock until each participating mirror’s updates are committed to OST storage or recorded as failed, with outstanding IO safely resolved. Per-mirror errors are reported to the MDS via the cancellation LVB. On error (or for administrative operations), the MDS takes AW in EX mode, forcing writers to flush or resolve failures and release, then clears INFLIGHT from clean mirrors and sets STALE on failed ones. V1 supports fast-fail and slow-fail secondary policies under this same release rule. See Write Operation Flow and Active Writer Lock.

Write ordering is enforced by requiring client-side LDLM extent locks on the primary mirror. This means DIO to IWM files must use client-side locks where today it operates locklessly — a performance regression for shared-file DIO workloads. Restoring lockless DIO is a high priority for future work; see Write Ordering Alternatives.

Client eviction, MDS failover, write leader failure, and other recovery scenarios are covered in Failure Handling.

Relationship to Erasure Coding

Immediate mirroring is separate from FLR Erasure Coding (Confluence page 105093375) (read-only EC), and can be done partly in parallel. IWM will be the foundation of immediate erasure coding — the write duplication infrastructure, Active Writer lock, epoch management, page consistency mechanisms, and error reporting all carry over to EC. Immediate EC will be covered in a separate design document.

Detailed Design

The write flow, AW lock mechanics, client-side IO duplication, and failure handling are each covered in the subsections below. Layout handling proceeds similarly to existing FLR, except that secondary mirrors are marked INFLIGHT rather than STALE during writes — see Mirror States for the full flag model.

FLR Mirroring

FLR Immediate Mirroring State Machine

FLR Immediate Replication

Write Operation Flow

Before starting a write (or other OST-modifying operation — see §2.3.2), the client acquires the layout lock and then the AW lock (see Lock Ordering). For buffered writes, AW must cover acquisition/use of the primary PW extent write lock, before data can be dirtied; it is not first acquired at writeback. Ordinary buffered write() may return after accepting data into the client cache, with normal aggregation and later BRW fan-out to the participating mirrors. The primary mirror remains readable; secondaries are INFLIGHT (see Mirror States).

The client holds AW across this write-permission window and its resulting dirty data and IO. Before releasing AW, it must stop further dirtying under that authority, obtain committed or recorded-failed outcomes for every participating mirror, and safely resolve outstanding IO (see Lock Lifetime). It releases AW with the mirror-error summary (see Error Reporting). Application return, RPC completion, durable commit, AW release, and global epoch closure are distinct events; the agreed application contracts and remaining mechanism questions are recorded in Operation Completion Consistency Model.

In the normal (no-error) case, once all clients release their AW locks, the MDS closes the epoch and transitions the layout back to RDONLY. If any client reports errors, the MDS forces epoch closure — see Epoch Close for details.

Example Scenario

With one client, if there are three mirrors and three writes:

If write 1 errored to mirror 0
If write 2 errored to mirror 1

The client will report errors on mirror 0 and mirror 1 to the MDS as part of AW lock cancellation (see Active Writer Lock for the LVB structure). The MDS would then clear INFLIGHT from mirror 2 (it is now clean) and mark mirrors 0 and 1 as STALE (clearing INFLIGHT, adding STALE). Userspace can try to resync to these mirrors, and if this fails, will need to add new mirrors.

If all mirrors fail writes during an IO, the file is degraded to its pre-IWM state — all mirrors are marked STALE on epoch close and the file requires resync. This is the same outcome as a total write failure in delayed-write FLR today.

Primary Mirror Selection and Read Visibility

Only one mirror is readable during an IWM epoch: without MVCC (which Lustre’s storage backends lack), writes arriving at different mirrors at different times could expose inconsistent data to concurrent readers. The MDS marks secondary mirrors INFLIGHT at epoch open, and IWM-aware clients exclude them from reads. The primary mirror remains unflagged and readable. This also covers OST-authoritative metadata (notably file size) — IWM-aware clients query only the primary while secondaries are INFLIGHT. Non-IWM clients can ignore unknown component flags: their existing layout grants must be revoked before epoch entry, and any subsequent request for a usable layout closes the epoch before it is granted. See Old-Client Compatibility and Mirror States. However, this is not possible for DIO, which does not use LDLM locks on the client, but instead only uses them locally on each separate server.

Active Writer Lock

This section covers the mechanics of the Active Writer (AW) lock: how it is used, what it guarantees, and how the MDS uses it to manage write epochs. References below to flushing or syncing before release include the failure resolution described here: successful mirrors must be durable, while terminally failed mirrors must be reported and their outstanding IO safely resolved.

Lock Semantics

The AW lock is a CW (concurrent write) lock on a special MDS IBITs bit called ACTIVE_WRITERS. Multiple clients can hold it simultaneously. The MDS uses lock presence to determine whether any writers are active on a file. When the MDS needs exclusive access (epoch close, mirror replacement, resync), it requests the AW lock in EX mode, which forces all clients to flush and release.

Lock Lifetime and Commit Requirements

For buffered IO, the primary PW extent write lock defines the coverage window: AW must be held before that write authority can be used to dirty data, including reuse of a cached PW lock and mmap write permission. Dirty data belongs to that epoch immediately, not when a BRW is eventually submitted. A layout refresh within the epoch does not transfer this obligation to a later epoch. The exact acquisition and admission sequence is part of the unified epoch/layout protocol in §2.2.7.

The client holds AW until no further data can be dirtied under the closing authority and every participating mirror has a resolved outcome: its updates are confirmed durable on OST storage, or its failure is recorded for the MDS. Recording failure alone does not authorize release: outstanding sends, retries, and replay must be completed or safely cancelled/fenced before AW and associated resources are released. Every mirror reported successful must have confirmed durability; an RPC acknowledgment alone is insufficient.

The durability requirement for IWM is stronger than for normal Lustre writes. Without IWM, a buffered write flushed to OST page cache but not yet journal-committed is at risk if the OST crashes — but this is just data loss. With IWM, if one mirror’s OST commits while another does not, the mirrors silently diverge with no signal to the MDS. Mirror inconsistency is qualitatively worse than data loss: it persists silently and can corrupt reads after recovery.

The client confirms journal commit on every mirror it reports successful before releasing AW. Mirrors that cannot meet that requirement follow the selected failure policy. Each OST import tracks last_committed (highest durably committed transno). At AW release:

For each OST written during the epoch on a mirror not already recorded as failed, check whether the last write’s transno ≤ last_committed.
If all remaining updates are committed and failed-mirror IO is safely resolved, release with the accumulated mirror-error summary.
For remaining uncommitted updates, issue OST_SYNC and confirm durability. Secondary sync and recovery waits obey the selected fast-fail or slow-fail policy. A terminal inability to establish durability is recorded as a mirror failure and its outstanding IO safely resolved; release does not require that abandoned mirror to commit successfully.

The normal check costs one comparison per OST; explicit sync is needed when epoch close encounters uncommitted writes. The proposed iwm_sync_on_epoch_close tunable defaults to enabled. Disabling it explicitly opts out of the epoch-close durability guarantee and accepts mirror inconsistency in the client+OST crash window. It must not weaken the stable-storage contract of DIO or explicit durability requests. Neither fast-fail nor slow-fail implicitly disables commit verification.

Interaction with sync_on_lock_cancel: OSTs default to sync_lock_cancel=always (set by ofd_slc_set() when sync_journal=0), which forces journal commit when an extent lock is cancelled — protecting lock handoff between clients. The IWM sync mechanism is complementary: SLC protects other clients from seeing uncommitted data, while the AW release sync protects mirror consistency across crashes.

Failure cases: A client that dies without releasing AW follows conservative eviction/recovery handling. For a live client, failures at submission, transport/recovery, modifying-RPC completion, or commit/sync are reported once terminal under the selected policy. A successfully recovered slow-fail event is not itself a final mirror error. Terminal failures cause STALE handling through the cancellation report and epoch close; they need not occur specifically at commit time.

Successful IWM DIO completion requires stable storage, whether or not the caller supplied synchronous flags. Before completion, each participating mirror’s updates must be committed or recorded as failed, with outstanding activity safely resolved; the primary determines the result. This is an application contract, not an inference from OBD_BRW_SYNC. In the September 2026 source checkout (47638add7815), osc_page_submit() sets OBD_BRW_SYNC, while the OFD write-commit path requests a synchronous transaction when a buffer lacks OBD_BRW_ASYNC. The full DIO flag/commit path, including secondary fan-out and backend completion, still needs verification; any path without sufficient commit confirmation must explicitly obtain it. For ordinary buffered IO, tracking continues after write() returns, through writeback, RPC completion, and commit verification.

The AW lock is acquired via standard LDLM enqueue to the MDS and cached for reuse by subsequent writes to the same file. A proposed brief idle timeout (1-5 seconds) initiates AW flush/writeback; it does not by itself authorize release. The release path must first retire or gate the covered PW write authority and writable mappings, then resolve their dirty data and IO. A retained cached PW lock may not permit new dirtying after AW release without admission under valid AW coverage. This avoids per-write MDS round-trips while bounding idle epoch retention; the PW/AW cache-lifetime mechanism remains to be specified.

Error Reporting via Cancellation LVB

When the client releases its AW lock, it communicates write results to the MDS in the client-to-MDS cancellation request. Adding this cancellation LVB is new protocol work; the existing LDLM_CANCEL format does not carry it. The report uses a versioned, length-delimited, extensible message structure from v1 onward. V1 sends only a mandatory mirror-error summary, with no per-stripe detail. Each reported mirror error causes whole-mirror STALE handling; the MDS combines reports from all clients against the same epoch mirror mapping. CLIO collects errors from all types of modifying operations covered by AW, including writes and setattr.

The mirror summary is a counted bitmap, not a fixed uint16: current Lustre defaults to a mirror limit of 16 but supports up to 256, requiring up to 32 bytes of bitmap payload. Bit positions refer to the epoch’s ordered immediate participant list, not directly to possibly sparse mirror IDs; that mapping must be shared by the MDS and clients and remain stable throughout the epoch. Future optional detail records can refine the report without changing v1’s conservative whole-mirror behavior. See Cancellation LVB Format and Future Stripe Detail for the structure, numbering, and sizing rules.

When an error becomes terminal under the selected policy, the client resolves remaining epoch IO and releases AW without the normal idle-cache delay, reporting failed mirrors through the cancellation LVB. This triggers MDS epoch closure. Successfully recovered slow-fail events do not themselves trigger error-driven closure. Any new writer admitted before MDS EX acquisition must be included in closure under the same commit-or-record-failure rule.

Epoch Close: MDS Processing

The MDS closes a write epoch when it holds no outstanding AW locks for a file. In the normal case, clients finish writing and release naturally. When a client reports errors, the MDS requests AW EX, forcing remaining writers to flush and resolve their epoch outcomes before release. Successful mirrors must be committed; failed mirrors must be reported with outstanding IO safely resolved. The same mechanism handles administrative operations such as mirror replacement and resync. On a hot file this can be expensive: every holder must finish the synchronous release path before EX is granted, and the selected slow-fail policy can prolong recovery waits until success or its give-up condition.

Once the MDS holds exclusive access, it must handle any evicted clients before transitioning the layout. Evicted clients will not receive the AW lock cancellation and may have writes in-flight that land on OSTs after the epoch close — see Client Eviction from MDT and MDS Failover for how this is handled (OST extent lock flush to maximise primary mirror completeness).

After evicted clients are handled, the MDS examines the error reports from all clients’ cancellation LVBs and transitions the layout:

No errors on a mirror: clear INFLIGHT — mirror is now clean and in sync.
Errors on a mirror: clear INFLIGHT, set STALE — mirror requires full resync. lamigo is notified via changelogs (the same mechanism used by Hot Pools today) and attempts recovery — see §2.4.5.
Client eviction (no error report available): assume writes to all mirrors except the primary failed — clear INFLIGHT, set STALE on secondaries.

The layout transitions back to RDONLY. This transition bumps the layout generation (version). Note that the OST-side layout generation check (ff_layout_version) does not by itself fence late-arriving RPCs from evicted clients — the OST only rejects writes older than the minimum generation it has already seen, and it learns new generations from incoming writes rather than from the MDS. The actual fencing of evicted-client writes is provided by the OST extent lock flush described above.

Recovery After MDS Failover

For recovery to succeed after MDS failover or crash, the MDS must determine which files had active IWM write epochs at the time of failure, so it can close those epochs safely. AW lock state (including which clients hold AW locks) is held in memory and lost on restart, so the MDS cannot simply reconstruct the epoch state from its in-memory lock tables.

If any client fails to reconnect during recovery (is evicted), the MDS closes active epochs for that client’s files with errors — see Client Eviction from MDT for the eviction behavior. The primary mirror is always identifiable after a crash: INFLIGHT is durably recorded in the layout xattr (lcme_flags), and the primary is the one without INFLIGHT set (set as part of the durable layout transition to WRITE_PENDING at epoch open). If all clients reconnect successfully, recovery proceeds normally: clients replay their AW locks, the MDS reconstructs the in-memory epoch state, and epochs close as they would in the non-crash case.

The central question is how the MDS identifies which files had active epochs. There are several possible approaches:

Scan all inodes for INFLIGHT mirrors. Correct but prohibitively expensive — a large filesystem could have billions of inodes and only a handful of active IWM files.
Replay from client lock state. Clients replay their AW locks during recovery, which tells the MDS which files have active epochs. But this only works if all clients reconnect — if a client is evicted, its lock state is lost and the MDS has no way to know which files it was writing. This is exactly the case where recovery matters most.
Durable epoch FID set. The MDS maintains a persistent set of FIDs with active write epochs, updated at epoch open/close time. On recovery, the MDS reads this set directly instead of scanning or depending on client replay.

Chosen approach: durable epoch FID set. When a file’s first AW lock is granted (transitioning the layout to WRITE_PENDING and setting INFLIGHT on secondaries), the FID is added to the set. When the epoch closes, the FID is removed. This is one durable write per file entering a write epoch, not per client or per lock grant. The set must support fast insertion/removal and could be very large (potentially millions of concurrent IWM files). The storage and indexing design is deferred to the implementation phase.

During recovery, the MDS must first check the active-epoch FID set against current file state: files that are no longer IWM files (e.g., layout was changed before the crash) should be removed from the set. Then, if any client is evicted, the MDS iterates the remaining set and closes all active epochs with errors. This is conservative: the evicted client may not have held AW locks on all of these files, so some mirrors may be needlessly staled. But it is always safe and correct — mirrors that were genuinely consistent will simply be resynced unnecessarily.

Recovery iteration over the FID set should be parallelisable — each file’s epoch close is independent, and the set could be large. The storage structure chosen for the FID set should not force sequential processing; multiple recovery threads should be able to claim and process entries concurrently.

The key trade-off: v1 does not durably record which clients participate in which files’ epochs. Recording per-file client participation would narrow the blast radius on recovery — staling only files where the evicted client actually held AW locks. But this requires a durable write on every AW lock grant (not just the first per file), and the storage must scale to potentially thousands of clients per file. Possible approaches include per-file xattrs storing client NIDs (limited by xattr size), NID range compression (ineffective for IPv6), and separate per-file extent trees (heavy). The cost-benefit balance is unclear — broad staling is always correct and costs nothing at runtime. Per-file tracking adds per-lock-grant IO cost and on-disk complexity. It is worth revisiting if broad staling proves problematic in practice.

Flush Semantics on Lock Release

AW flush/writeback covers all data dirtied under the client’s PW write authority for the epoch, including data not yet submitted in any RPC. Before release, further dirtying under that authority must be stopped: drain active users and revoke or gate cached PW reuse and writable mmap mappings. A one-time dirty-page scan is not sufficient if those permissions can dirty pages again. New writes must obtain valid AW coverage after admission resumes. The mechanism for doing this is part of the epoch/layout protocol, not a decision to defer AW acquisition until BRW submission.

The AW lock has two release paths with the same completion condition: covered dirtying authority is quiesced, each participating mirror’s updates are committed or recorded as failed, and all outstanding epoch IO is safely resolved. The cancellation LVB distinguishes successful mirrors from failed ones. Neither path requires successful commit on a conclusively abandoned mirror.

Voluntary release (client-initiated, no conflicting lock request): Quiesce the covered write authority, finish epoch writeback, confirm commit for successful mirrors, and resolve failed-mirror IO as described in §2.2.2. Release with the accumulated mirror-error summary. A terminal failure bypasses idle caching and starts this resolution/reporting path promptly.

Forced revocation (MDS requests EX lock, blocking AST fires): Start quiescing and flushing immediately, then follow the selected policy to obtain committed or terminally failed outcomes. Slow-fail may spend time in standard retry/recovery, but its give-up mechanism must remain effective during forced closure. The blocking AST cannot complete AW release until covered dirtying authority is quiesced, successful mirrors are durable, and failed-mirror IO is safely resolved. Expiring a timer or setting an error bit alone does not complete that work.

Both paths enforce the same commit-or-record-failure condition. The client or MDS chooses when flushing begins; the secondary failure policy controls retry persistence. Neither mode permits late retries to escape the closed epoch and interfere with later IO, resync, or mirror reuse.

AW flush/writeback does not semantically require discarding successfully written cached data. However, existing PW cancellation flushes and discards pages, so choosing to reuse that path may invalidate cache as an implementation consequence. Retaining clean pages requires a mechanism that still removes or gates write permission; retaining writable mmap mappings without AW coverage is not valid. Secondary failure alone does not require discarding data when the primary succeeded; primary/writeback errors retain normal error and page-state handling.

Both release paths require the client to associate PW write authority, dirty pages, in-flight RPCs, and commit/error tracking with the current epoch. This association must survive buffered syscall return and layout refresh. RPC reference counts or lo_active_ios alone cannot be assumed to cover idle dirty pages or cached write permission. The choice of PW lifetime coupling, page/range tracking, and flush references remains open.

For evicted clients that cannot respond to lock callbacks, OST extent locks (LCK_PW on the full file extent) can be used to force-flush the evicted client’s dirty data on the OSTs. This mechanism is used during both live eviction and MDS recovery eviction — see MDS Failover for details.

Existing mechanisms provide pieces of AW flush/writeback, not yet a complete protocol. PW cancellation reaches osc_lock_flush(), which writes back a write-locked range and then discards pages. Layout invalidation itself only invalidates the layout and unmaps mmap pages; installing a changed layout through cl_conf_set() / lov_conf_set() can subsequently reach lov_layout_change() → cl_object_prune() → vvp_prune(), which flushes and tears down the cache. CL_FSYNC_LOCAL in that prune path waits for writeback, not the full IWM durability barrier. Existing FLR resync explicitly triggers LL_DV_WR_FLUSH before layout revocation. Decide which pieces AW can reuse, whether PW locks must actually be cancelled, and how layout-driven flushing retains the old epoch association without recursively requesting new AW admission.

AW Lock vs Layout Lock Interaction

The AW lock and the layout lock track distinct concerns. The layout lock tracks what the layout is — stripe configuration, component structure, mirror membership. The AW lock tracks the write epoch — who is writing, what phase they are in, and when the epoch closes. These must not be merged — they are separate concerns — but they interact closely.

Lock ordering: Clients acquire the layout before AW. MDS sequencing depends on the operation; epoch closure and administrative layout changes have different paths:

Client write path: The client takes the layout lock first (learning the current layout). If the layout has immediate mirrors, the client then takes the AW lock and proceeds with write duplication. If the layout does not require immediate mirroring (e.g., delayed-write FLR, or no mirrors), no AW lock is taken.
IO restart on layout change: If the layout changes during IO (layout lock revoked), the client’s IO restarts from the layout lock. The AW lock is not revoked by an intra-epoch layout change — it is a separate IBITS bit and survives independently (see Why Separate Locks). Pending dirty data retains its epoch association through restart. Removing immediate mirroring is instead an epoch-boundary operation: resolve the old PW/dirty-data/AW obligations before granting write authority under the new delayed-write layout. Discovering a different layout is not permission to drop an unresolved AW obligation.
Epoch close (non-structural, MDS-initiated): The MDS takes EX on the AW lock first, forcing all writers to flush dirty pages, report errors via cancellation LVB, and release. Once the epoch is closed, the MDS updates the layout flags (INFLIGHT/STALE) via a layout lock cycle — layout EX second. Because this is a flag-only change, the STATE_ONLY_CHANGE optimization applies: clients re-acquire layout CR without a page cache nuke.
Administrative operations (resync, mirror replacement, mirror add, changes to IMMEDIATE membership): Block new epoch participants and quiesce existing writers before changing the layout. The proposed layout EX → AW EX sequence must satisfy the flush dependency requirement below; layout EX alone is not a flush barrier. Structural changes require full page cache teardown on re-acquisition. Adding or removing IMMEDIATE changes the next epoch’s participants and must follow the conditional transition rules in §4.2.
Non-IWM layout acquisition (including ACCESS and layouts returned on open, not just write intents): Close any active epoch before granting a usable layout, following §2.4.6. A read-only handoff is not itself structural or a reason to stale healthy mirrors. Legacy clients use their existing layout refresh path.

Flush dependency requirement: Closing an epoch must block new writers without preventing existing AW holders from completing flushes and any required layout refresh or IO restart. An MDS operation must not hold layout EX while waiting for AW cancellation if that cancellation needs layout CR. The implementation must serialize epoch close, layout publication, and epoch admission without this circular wait; the concrete mechanism remains to be specified. This also applies when an old client’s layout request triggers closure.

PW coverage and layout reconfiguration: Admission must cover PW acquisition and cached-lock reuse before dirtying, including mmap write faults. During AW flush/writeback, completion of existing dirty data remains authorized by the closing AW; it must not be treated as a new writer that waits behind its own closure. An intra-epoch refresh must preserve pending dirty-data obligations; a membership change or compatibility handoff must resolve them before new write authority becomes usable. The acquire, flush, refresh, and release sequence is one open design problem, tracked in the accompanying iwm_discussion_checklist.md as shared questions P1–P8. Start with what PW cancellation and cl_conf_set() / object pruning need to complete: those dependencies may determine layout/AW ordering, admission, and epoch-publication choices. They are not downstream implementation details of an otherwise settled protocol.

The AW requirement is conditional on the layout under which write authority was granted. An intra-epoch layout refresh preserves existing AW coverage and dirty-data obligations. A transition out of IWM must resolve those obligations before delayed-write authority becomes usable; subsequent writes under that layout do not acquire AW. The changed layout alone cannot retire pending obligations from the previous epoch.

Why Separate Locks

The AW lock lives on a separate IBITS bit from the layout lock (see Lock Semantics), not a mode or flag on the layout lock itself. This separation is deliberate — combining them would create fundamental problems:

The layout lock is light; the AW lock is heavy. The layout lock is designed to be held “lightly” — the client reads the layout, then the lock can be revoked at any time without flushing IO. The client simply restarts the IO under the new layout. This is essential for operations that change the layout frequently (component instantiation in PFL files, self-extending layouts). The AW lock has the opposite semantic: its blocking AST forces a full flush and waits for OST commit before returning. If both semantics lived on the same lock, every layout revocation would require flushing all in-flight IO — turning a cheap IO restart into an expensive synchronous flush. For a many-component PFL file written sequentially, this would create an unnecessary epoch close at every component boundary.

The epoch must span layout changes. Component instantiation, SEL extension, and other layout changes revoke the LAYOUT bits but should not close the write epoch. The mirrors are the same, the write session is ongoing, and nothing about epoch consistency semantics changed. Because ACTIVE_WRITERS is a separate bit, taking EX on LAYOUT does not revoke it — the AW lock survives layout changes naturally. The epoch only closes when the MDS explicitly takes EX on ACTIVE_WRITERS (error handling, resync, admin) or all clients voluntarily release their AW locks.

MDT/OST dependency constraint. AW is held across OST IO, so separating the locks does not by itself prove the dependency graph acyclic. Writers’ CW locks do not conflict, but an MDS AW EX request waits for their flushes. Those flushes may need layout refresh after invalidation, and another client’s IO can trigger closure through a legacy layout request. Both ordinary closure and administrative or compatibility handoffs must meet the flush dependency requirement above.

Alternatives considered: Combining the bits onto one lock with conditional flush semantics (flush only when in an IWM epoch) was considered, but this effectively reinvents the AW lock as a flag on the layout lock while adding mode-dependent complexity to every layout lock consumer. LDLM lock conversion (acquiring both bits, then converting to drop LAYOUT while keeping ACTIVE_WRITERS) could save a round trip but couples the implementation to a lightly-exercised LDLM feature with subtle recovery and race implications. Both are potential v2 optimizations; v1 uses separate locks for simplicity and debuggability.

Behavior During Layout Changes

This section describes non-destructive layout changes within an existing IWM epoch (component instantiation, FLR flag transitions that preserve the immediate participant set) where only the layout lock is involved and AW is left untouched — the STATE_ONLY_CHANGE path described in §2.2.7.3. It does not apply to destructive changes, IMMEDIATE membership changes, epoch boundaries, or a handoff to a non-IWM client. Legacy layout acquisition closes an active epoch before returning the layout, even for a read — see §2.4.6.

When a non-destructive layout change occurs while a client holds both the layout lock and the AW lock (e.g., another client’s write intent triggers component instantiation):

The server takes EX on LAYOUT, revoking all clients’ layout locks via blocking AST.
The layout lock blocking AST fires on the client and returns immediately — no flush, no awareness of AW state. This is unchanged from today’s behavior.
In-flight OST writes either complete normally or are rejected by OSTs (stale layout generation), triggering IO restart.
The client’s IO restart path detects the layout change and restarts the IO from the top.
The AW lock remains held throughout — the epoch is still open.
On restart, the client re-acquires the layout lock (via write intent if needed), gets the new layout, and continues writing under the same AW epoch.

The key property: layout changes are not epoch boundaries. This is safe because ongoing IO will simply be restarted at the end, as it is for any layout change. In-flight writes that land on OSTs with a stale layout generation are rejected, and the client retries them under the new layout. The AW cancellation LVB accumulates errors across layout changes, and the epoch close (whenever it eventually happens) handles them all.

Layout Lock and Page Cache Interaction

Layout lock re-acquisition has a hidden cost: the client’s entire page cache for the file is destroyed. Understanding this mechanism is critical for IWM performance.

The mechanism: When the client re-acquires its CR layout lock after revocation, ll_layout_lock_set() installs the new layout via lov_conf_set(OBJECT_CONF_SET). This calls lov_layout_change(), which tears down and rebuilds the lov_object — destroying all sub-objects (osc_object instances representing individual stripes). Before teardown, cl_object_prune() → vvp_prune() calls cl_sync_file_range() to flush all dirty pages, then ll_truncate_inode_pages_final() to nuke the entire page cache. The nuke is necessary because each cached page (cl_page → osc_page) is pinned to a specific sub-object; when those sub-objects are destroyed, the pages cannot survive.

Why this exists: Layout changes can alter the stripe structure — different OSTs, different components, different stripe counts. The sub-objects for the old layout are invalid after the change. Tearing down and rebuilding is the only safe approach for structural changes.

Why this matters for IWM: Delayed-write FLR only mirrors inactive files — there is no hot page cache during layout transitions, so the nuke has negligible cost. IWM is the opposite: it targets files under active sustained IO with hot page caches on multiple clients. If every epoch transition (setting INFLIGHT, clearing STALE) required a layout lock cycle with page cache nuke, IWM performance under buffered IO would be unacceptable.

Non-destructive layout changes. Not all layout changes alter the stripe structure. Two categories preserve existing sub-objects:

FLR state transitions (RDONLY → WRITE_PENDING, epoch flag updates to INFLIGHT/STALE flags): same components, same stripes, same OSTs. Only per-component flags change.
Component instantiation (PFL extend, SEL): new sub-objects are added for the new component, but existing sub-objects are untouched. Pages cached against existing components remain valid.

For these changes, the page cache nuke is unnecessary — the existing sub-objects are still valid and pages referencing them are still correct.

The optimization: STATE_ONLY_CHANGE flag. To avoid the page cache nuke on non-destructive changes, the MDS sets a transient flag (STATE_ONLY_CHANGE or similar) in the layout when the most recent layout_gen bump was non-destructive. A connect flag (OBD_CONNECT2 bit) gates this — the MDS only sets the flag for clients that advertise support.

On layout lock re-acquisition, the client checks:

If the client’s current layout_gen is exactly N-1 and the new layout carries STATE_ONLY_CHANGE: update flags (and add new sub-objects for component instantiation) in the existing lov_stripe_md in-place, skip cl_object_prune(). The page cache survives.
Otherwise (flag absent, client is more than one version behind, or destructive change): full teardown as today. Fall back to old behavior.

The N-1 constraint is critical: the flag only describes the most recent transition. A client that missed multiple layout changes cannot trust it and must do the full rebuild.

Safety properties:

Old clients never see the flag (connect flag gates it) and always do full teardown. Correct by default.
Stale clients (more than one version behind) ignore the flag and do full teardown. Correct.
MDS reboot loses the transient flag. Clients reconnect without it and do full teardown. Correct.
Flag absent for any reason (bug, race, mixed destructive/non-destructive sequence): full teardown. The optimization is purely opportunistic — absence never causes incorrectness.

Prior art: CSDC compressibility changes. The CSDC (client-side data compression) feature introduced LAYOUT_INTENT_CHANGE as a layout intent opcode for flag-only layout updates (EX-8355, Gerrit 54248). This allows the MDS to change the LCM_FL_INCOMPRESSIBLE flag on a layout without altering the stripe structure. The patch defines the wire protocol (LAYOUT_INTENT_CHANGE opcode, LAIF_INCOMPRESSIBLE flag, OBD_CONNECT2_UPDATE_LAYOUT connect flag) and the MDS-side handler (lod_declare_layout_minor_change()). However, the current implementation still bumps layout_gen and still triggers the full page cache teardown on the client — it avoids the problem by deferring the change until no layout lock is held, rather than making the client-side re-acquisition cheap. The IWM optimization extends this by making the client-side path aware of non-destructive changes, so the layout lock can be re-acquired without tearing down the lov_object or nuking the page cache. The LAYOUT_INTENT_CHANGE wire definitions from CSDC could be reused or extended to carry the IWM state transitions.

Relationship to epoch handoff: Page cache preservation and epoch serialization are separate decisions. Legacy clients always use full teardown on layout refresh; destructive changes also require it. Non-destructive transitions may use STATE_ONLY_CHANGE for eligible IWM-aware clients, including flag-only epoch close. This does not remove the requirement to close the epoch before granting a usable layout to a non-IWM client, or to drain legacy writes before opening the next epoch (see §2.4.6).

Client Implementation

Write Duplication via BRW Page Fan-Out

The client will need to duplicate all writes to all of the mirrors (non-write modifying operations are covered separately in §2.3.2). The key design choice is at what level duplication occurs.

IO-level duplication is not viable. For buffered I/O, IO-level duplication would require sending pages to secondary mirrors at IO creation time — effectively forcing direct I/O to the secondaries — because the pages cannot be left in the page cache to be aggregated independently per mirror. This destroys asynchronous write semantics and write aggregation for secondary mirrors, which is a non-starter from a performance perspective. (For DIO, IO-level duplication would work but is not sufficient on its own.)

The selected approach operates at the BRW/RPC layer, intercepting after the primary mirror’s write has been fully prepared (pages assembled, bounce pages allocated for compression/encryption). This handles both buffered and direct I/O uniformly: for buffered writes, the page cache performs normal write aggregation on the primary mirror and duplication fires at BRW send time from the aggregated pages; for DIO, duplication fires from the same BRW page array built for the primary. In neither case does the duplication layer need to re-enter the IO start path.

BRW page fan-out (selected approach) re-derives per-mirror RPCs from the primary mirror’s assembled BRW page array via a call-up from the OSC layer to the LOV layer. When the primary mirror’s OSC has built its BRW RPC (pages assembled, bounce pages allocated), it calls up to the LOV layer, which fans out the page array to each secondary mirror’s OSC. Each secondary mirror’s RPCs are built through that mirror’s own stripe calculation, so mirrors can have different layouts (different stripe count/size). Note: this call-up path must be careful about the PTLRPC no-sleep requirement — the callback occurs in RPC submission context. This is more complex than simple RPC replay but is necessary to support heterogeneous mirror layouts, which real deployments require (see Combining Immediate and Non-Immediate Mirrors). See Qian Yingjin’s prototype (LU-13643, Gerrit 63244) for a proof-of-concept of this approach.

An alternative considered was RPC duplication, which sends the primary mirror’s RPCs to all mirrors identically. This is simpler but requires all mirrors to have identical layouts (same stripe count and stripe size), since the same RPCs — already addressed to specific OSTs at specific offsets — would be reused verbatim. This restriction is too limiting for real deployments, where mirror layouts may differ.

Page Consistency During Write Duplication

The primary’s BRW pages must remain valid and unmodified until every mirror attempt completes or terminates with outstanding activity safely resolved. There are two consistency threats, and the BIO and DIO paths handle them differently:

Source page modification: For buffered IO, another write to the same pages could modify them while secondary RPCs are in flight, causing the secondaries to receive different data than the primary — silent inconsistency. For DIO, modifying the userspace source buffer during a write is already an application error; the implementation must retain the source references needed by every mirror attempt, without assuming that a BRW page array inherently provides private copies.
Competing IO on the same client: A second write to overlapping byte ranges must be serialized against in-flight duplication to prevent the secondary mirrors from seeing a different write order than the primary.

The following two sections describe how each IO path addresses these threats.

A third requirement follows from write duplication: each mirror RPC for a page must complete or reach terminal failure with outstanding activity safely resolved. Track per-page (or per-RPC) outcomes across mirrors; successful acknowledgments still require the separate durability checks before AW release. A failed mirror does not owe a successful acknowledgment, but its failure must be recorded and its IO must no longer threaten buffer lifetime or later epoch/resync ordering.

Buffered IO Completion Semantics

Ordinary buffered write() retains normal asynchronous behavior: it may return after accepting data into the client cache, before any mirror RPC is sent. It does not wait for mirror writeback or promise stable storage. Normal write aggregation remains in place; synchronous requests have the separate contract in §4.1.

AW coverage begins with the primary PW extent write-lock authority, before pages can be dirtied, and continues across syscall return, delayed writeback, and required commit/error tracking. Reusing a cached PW lock must obey the same coverage rule as acquiring one. A layout refresh cannot leave already accepted data without an epoch owner.

The fan-out barrier applies to writeback completion, not ordinary buffered syscall return. Preserve source-page stability and overlapping-write ordering until every mirror attempt has completed or safely terminated, not just the primary. Extend page/writeback completion accounting accordingly, including protection against mmap modification; a page lock alone must not be assumed to stop stores through an already writable mapping. Once no attempt can use the source data, page/buffer completion is distinct from durable commit: the epoch’s commit/error tracking and AW obligation continue until their own release conditions hold.

Memory-mapped writes use the same writeback fan-out, but acquiring write permission is an earlier event. The write-fault / page_mkwrite path must establish AW-covered PW authority before permitting dirtying. AW flush/writeback must also stop subsequent stores through existing writable mappings, by revoking or write-protecting them with appropriate race handling. How PW cancellation, mapping invalidation, and layout reconfiguration supply this barrier remains an explicit implementation discussion, not a reason to defer AW until writeback.

Direct IO Completion Semantics

DIO does not use the page cache, so page locks are not available for serialization. V1 waits for all mirror RPCs to complete or reach terminal failure, including safe resolution of outstanding activity, before returning and releasing buffer/ordering protection. Both modes preserve this rule: slow-fail can delay DIO during retry/recovery, while fast-fail abandons a secondary sooner. The return code reflects the primary result; secondary failures are recorded for the cancellation LVB (see Secondary Failure Policy). Returning while secondary work continues is a separate future asynchronous-completion design requiring protected data copies and ordering against competing IO.

Successful DIO completion additionally guarantees stable storage for the successful primary write, regardless of O_SYNC or O_DSYNC. For the operation’s updates, every participating mirror must be committed or recorded as failed, with outstanding activity safely resolved. A secondary failure does not turn primary success into an application error. This guarantee is mandatory even if satisfying it requires explicit commit confirmation beyond the write RPC reply. It does not require all writers to release AW or the MDS to close the file-wide epoch.

Implications for Erasure Coding

The BRW page fan-out and page consistency mechanisms developed for IWM are intended to carry over to immediate EC writes, though significant design work remains. EC requires absolute consistency between source data pages and computed parity pages — primary locking provides ordering, and the tree lock (rounded to raidset-aligned extents) would ensure source pages are stable during parity computation. Tentatively, EC parity pages would be dispatched to their target OSTs via the same LOV-level RPC callback used for IWM secondary mirror fan-out, reusing the completion barrier infrastructure. However, the details of parity computation timing, partial-stripe writes, and the interaction between EC raidset geometry and the fan-out path remain to be resolved.

Compression and Encryption Across Mirrors

Ideally we would not have to compress or encrypt data more than once when sending to multiple servers Each secondary mirror’s OSC compresses independently from the original (uncompressed) source pages. Since compression is per-component in the Lustre layout (lcme_compr_type, lcme_compr_chunk_log_bits in lov_comp_md_entry_v1), different mirrors can independently specify compression settings — different algorithms, chunk sizes, or one compressed and one not — and the BRW page fan-out path handles this naturally. The compression work is performed once per mirror, but this is an acceptable cost for IWM’s typical 2-3 mirrors. Different chunk sizes across mirrors pose no correctness issue: if a write is not aligned to the secondary mirror’s chunk boundary, the secondary’s OST handles the read-modify-write internally, as it does for any unaligned compressed write.

A possible future enhancement is bounce page reuse: when all immediate mirrors share identical compression settings, bounce pages built for the primary mirror could be reused for secondaries, avoiding redundant compression/encryption work. This is not planned for v1.

Other Modifying Operations

The preceding section covers write IO duplication via BRW page fan-out — a mechanism specific to data writes, where the page cache and write aggregation require fan-out at the RPC assembly layer. Other modifying operations — truncate, hole punch, fallocate (space preallocation), and certain setattr variants — also modify OST object state and must be applied to all immediate mirrors. These operations do not transfer data pages, so the BRW fan-out path does not apply. Instead, they use a different fan-out mechanism at the LOV sub-io level.

Operation Classification

Modifying operations fall into three categories based on their mirror interaction:

OST object operations (must be duplicated to all mirrors):

Truncate — reduces file size via OST_PUNCH over [new_size, EOF). The LOV maps this to the affected stripes, as with any ranged operation.
Fallocate — covers both space preallocation (mode 0) and hole punch (FALLOC_FL_PUNCH_HOLE). Preallocation changes the physical allocation map without writing data; punch deallocates a byte range, replacing data with a hole. Both use OST_FALLOCATE with the appropriate mode flags.
Time setattr (ATTR_MTIME, ATTR_ATIME, ATTR_CTIME) — sets timestamps on OST objects via osc_setattr_async. Lightweight, no range calculation.

MDS-only operations (no mirror interaction needed):

chmod, chown — pure MDS inode metadata. OSTs are not contacted. No IWM concern.
setstripe — layout changes via MDS ioctl. Not a modifying operation in the IWM sense; layout changes interact with IWM through layout/epoch coordination, not through operation duplication.

Mixed operations:

Truncate is technically mixed: ll_setattr_raw always sends an MDS RPC first (the MDS must approve the size change and check ETXTBUSY), then dispatches cl_setattr_ost for the OST-side truncation. The MDS phase is not mirror-specific; only the OST phase requires duplication.

LOV Sub-IO Fan-Out

In current FLR, truncate, punch, and fallocate go through the cl_io CIT_SETATTR path. The LOV layer selects a single mirror (lov_io_mirror_init sets lis_mirror_index), creates sub-ios only for that mirror’s stripes, and sends the write intent RPC to stale other mirrors on the MDS. The operation executes on one mirror; resync later copies data to bring stale mirrors up to date.

For IWM, these operations execute on all eligible immediate participants in real time. Clean non-immediate copies are excluded, with their affected components marked STALE through the MDS layout/write-intent path before the modifying IO is permitted. The fan-out point is lov_io_iter_init: instead of iterating only the primary mirror’s stripes, it creates sub-ios for every eligible immediate mirror’s stripes that overlap the operation’s byte range. Each sub-io calls the appropriate per-stripe RPC (osc_punch_send for truncate, osc_fallocate_base for punch/fallocate, osc_setattr_async for time setattr). This is analogous to BRW page fan-out for writes — the LOV is the mirror-aware layer that manages per-mirror dispatch — but at the sub-io level rather than the RPC assembly level, because these operations have no page data to fan out.

The existing lov_foreach_io_layout_mirror macro already accepts a mirror index parameter. For IWM, the iteration expands to cover all eligible mirrors. Per-stripe size and offset translation (lov_size_to_stripe) uses each mirror’s own component layout, so heterogeneous mirror layouts (different stripe count/size) work correctly — the same file-level truncation offset maps to different OST object offsets depending on the mirror’s geometry.

The BRW fan-out for writes uses an OSC-to-LOV callback — a concession to the fact that write data must pass through the page cache and BRW assembly before the LOV can replicate it. These operations have no such constraint. The operation parameters (byte range, mode flags) are known at cl_io init time, so the LOV can create all per-mirror sub-ios up front during iter_init. This is a cleaner fan-out pattern: no callbacks, no PTLRPC no-sleep concerns, just parallel sub-io dispatch.

AW Lock and Epoch Interaction

These operations participate in AW lock epochs identically to writes. The client acquires the AW CW lock before the operation (or reuses a cached one), the MDS transitions the layout to WRITE_PENDING on first acquisition, and the operation executes under the epoch. Per-mirror errors are collected and reported via the AW lock cancellation LVB on release — if truncate succeeds on mirror 0 but fails on mirror 1, mirror 1 has the wrong (larger) size and is marked STALE on epoch close. Resync later copies the correct state from the primary.

For IWM-aware clients, the write intent RPC that FLR uses today (LAYOUT_INTENT_TRUNC, LAYOUT_INTENT_WRITE for fallocate) could potentially be subsumed by the AW lock epoch mechanism in a future optimization. Such an optimization must preserve the MDS-side staling of affected non-immediate components before any operation that omits them; duplicating to immediate mirrors alone is insufficient. The MDS must also continue to handle write intents from non-IWM clients (see §2.4.6), so the write intent code path cannot be removed. For v1, both paths coexist.

Page cache flush ordering: Before truncate or punch, the client flushes dirty pages in the affected range. Under IWM, this flush triggers BRW fan-out to all mirrors (via the write duplication path in §2.3.1), ensuring all mirrors have consistent data up to the truncation point before the truncate sub-ios execute. The interaction between this flush and AW lock dirty tracking — specifically, how the client knows which dirty pages belong to the current epoch and must be flushed — is an open design question (see §4.1).

Implications for Erasure Coding

The LOV sub-io fan-out for truncate/punch/fallocate extends naturally to immediate EC. Where IWM creates sub-ios per-mirror-per-stripe, EC would create sub-ios per-parity-group. Truncate and punch are particularly relevant for EC because they can create partial-stripe boundaries that require parity recomputation — a stripe that was fully allocated may become partially punched, requiring the parity stripe to be updated. The sub-io fan-out provides the dispatch infrastructure; the parity computation logic layers on top.

Error Management

Fast-fail and Slow-fail Secondary Policies

V1 supports two switchable policies for secondary failures. Fast-fail is the first development milestone because predictable failure reporting is needed for testing and availability-sensitive use cases. Slow-fail follows in the development sequence and is also required for the first supported release: many deployments need normal Lustre retry/recovery persistence to avoid unnecessary full-mirror resyncs. Development order does not select the production default.

Fast-fail: Use the non-delay failure path for secondary operations and promptly treat an unrecoverable attempt as a terminal mirror failure, without entering extended secondary retry/recovery waits. Existing ci_ndelay behavior is the starting point, but the complete secondary path must be checked: setting that flag alone must not leave hidden reconnect, recovery, replay, or commit/sync waits that defeat the policy. A transient outage may therefore stale the mirror and require resync.
Slow-fail: Use the ordinary Lustre RPC retry, reconnect, recovery, and replay procedures, retaining their normal opportunity to complete the operation and preserve the mirror. These procedures must also have a defined give-up path for IWM secondaries. When the configured give-up condition is reached, or a terminal error is returned, stop retrying, resolve outstanding activity safely, and report the mirror failed. Existing waits that can otherwise continue forever must remain terminable through this policy.

Slow-fail must be able to fail. The implementation needs a configurable give-up policy, such as an overall recovery/time budget or an explicit abort mechanism, that can reach the underlying waits. A per-RPC timeout is insufficient if retries or recovery repeatedly restart it. The exact controls, scope, production default, and any automatic timeout/retry budget remain to be selected; v1 must demonstrate that an unavailable secondary can be abandoned without waiting for it to return. This applies during normal IO and forced epoch closure, including secondary commit confirmation and OST_SYNC. Slow-fail may wait substantially longer than fast-fail, but cannot offer an irrevocable wait-forever path as its only behavior.

Shared failure semantics: A retryable event is not yet a final mirror error if slow-fail subsequently completes the operation and its durability requirements. An error becomes terminal when the selected policy gives up or normal processing returns a terminal failure; this can occur during submission, transport/recovery, the modifying RPC, or commit/sync. Once terminal, the mirror’s error remains recorded for the epoch, and the client promptly begins resolution and AW release/reporting without the idle-cache delay. The MDS closes the epoch and marks the failed mirror STALE. Late success from one RPC cannot clear the epoch’s accumulated mirror failure.

Safe abandonment: A timeout or error bit alone does not make outstanding IO safe. The implementation must stop retries and safely complete, cancel, or fence activity that could still access source buffers or interfere with later epoch IO or resync. Buffer and ordering protection must remain until that condition is met. Both modes enforce the same AW release invariant: every participating mirror’s updates are committed or recorded as failed, and outstanding activity has been safely resolved. Failure policy does not weaken the durability requirement for mirrors reported clean.

Application visibility and completion: Secondary failure does not replace a successful primary result with an application error, including for DIO, fsync(), fdatasync(), O_SYNC, and O_DSYNC. Both modes retain the v1 completion barriers: DIO and explicit durability requests require committed-or-recorded-failed outcomes for their scope, while ordinary buffered write() remains asynchronous and retains the required page/writeback protection during later fan-out. Slow-fail can therefore stall DIO, durability requests, writeback, or epoch close while ordinary recovery proceeds. Fast-fail reduces secondary-induced waiting; it is not an overall syscall deadline. Neither policy requires application completion to wait for global epoch closure. The primary remains the ordering anchor and follows its separate normal failure/restart path (see Primary Mirror Write Failure).

Configuration changes: Switching policy must not clear accumulated errors, release protection around outstanding IO, or strand operations in an uninterruptible recovery wait. The implementation must define when a changed policy takes effect, including how an already-waiting secondary reaches the configured give-up path. These controls govern IWM secondary behavior without silently changing ordinary primary IO policy.

Degradation and resync: A terminal secondary failure reduces redundancy while a successful primary remains usable. The MDS reports failed mirrors through the existing changelog path, and userspace resync restores them using normal FLR recovery. Slow-fail trades longer waits for a chance to avoid this degradation; fast-fail accepts it sooner.

Deferred alternatives: Returning from DIO while secondary retries continue, or queueing replication after operation completion, are separate future designs. They require protected source snapshots and ordering against subsequent modifications. The v1 slow-fail mode performs standard retry/recovery while retaining the existing completion and protection rules.

Failure Handling

Mirror States: STALE and INFLIGHT Flags

During an IWM write epoch, participating immediate secondary mirrors are marked INFLIGHT. The primary (write leader) is not flagged — it remains readable. Non-immediate mirrors do not participate in that epoch and follow the ordinary FLR staling rule below. The AW lock tracks that a write epoch is active.

INFLIGHT — indicates an IWM write epoch is actively in progress on this mirror. This is a new flag. A mirror marked INFLIGHT is being updated by immediate writes and is expected to become consistent when the write epoch closes. IWM-aware clients exclude it from reads until closure. Unknown component flags are not a compatibility barrier: existing non-IWM clients can ignore INFLIGHT during ordinary IO.

STALE — the mirror is an unknown amount of out of sync with the primary. Its contents should not be used and the issue can only be resolved with a full resync. This is the same flag used by delayed write mirroring (FLR today). For participating immediate mirrors, normal IWM writing uses INFLIGHT, not STALE; errors or eviction may leave them STALE at epoch close. Outside an IWM epoch, a legacy write intent uses ordinary delayed-write FLR rules and marks affected secondary components STALE. Old clients understand STALE and can trigger resyncs.

Clean non-immediate mirrors: IMMEDIATE describes write participation, not whether a copy is synchronized. A non-immediate mirror may be clean after resync or after removal from the immediate set. Before permitting a write or other modifying IO that omits that mirror, the MDS must mark its affected components STALE under ordinary FLR rules, including when the modification is performed by IWM clients on other mirrors. It cannot assume all non-immediate mirrors are already stale. See Tiering Transitions.

Backward compatibility: Neither IMMEDIATE nor INFLIGHT can be relied upon to make non-IWM clients reject a mirror. Safety comes from MDS-controlled layout handoff: revoke legacy layout grants before opening an IWM epoch, and close that epoch before granting a usable layout to a non-IWM client. Closure returns the file to RDONLY without staling healthy mirrors merely to allow a legacy read; errors and eviction still follow the close rules. Subsequent modifications follow the participation and staling rules above. See §2.4.6.

The MDS resolves participating immediate mirrors’ epoch flags at epoch close — see Epoch Close for the full rules. This is separate from marking omitted non-immediate components STALE before modifying IO. Mirrors left STALE after a write failure are visible to lamigo via changelogs, triggering recovery (see §2.4.5).

Write/Update Error from Client

Clients associate errors from all modifying operations with the AW epoch. Secondary retryable events follow the selected fast-fail or slow-fail policy; a successfully recovered transient event need not stale a mirror. A terminal error at submission, transport/recovery, modifying-RPC completion, or commit/sync time is recorded as a mirror failure. Reporting is not restricted to commit-time failures. The client then resolves remaining IO and releases AW without the normal idle-cache delay, carrying the accumulated error summary to the MDS and triggering epoch closure. See Secondary Failure Policy for give-up and safe-abandonment requirements.

Client Eviction from OST

A client eviction from an OST can invalidate the affected mirror’s completeness and commit guarantees; it is not necessarily observed only at commit time. The client reports the mirror error to the MDT when it cancels AW, after safely resolving outstanding IO. If consistency cannot be established, epoch close clears INFLIGHT and sets STALE, requiring later resync by lamigo or other userspace tooling. Selecting slow-fail does not turn an eviction-induced consistency failure into a clean result merely because connectivity returns.

Client Eviction from MDT

If a client with an active AW lock is evicted from the MDT, we cannot know whether it completed writes successfully. The MDS force-flushes the evicted client’s caches via OST extent locks (see Flush Semantics) to maximise the chance that in-flight writes complete on the primary mirror, then closes the epoch with all non-primary mirrors marked STALE (clearing INFLIGHT, setting STALE). The flush only benefits the primary’s completeness — secondaries are staled regardless.

Client Loss

Client loss - where a client crashes or similar - is the same as MDT eviction, since such a client is evicted from the MDT and OSTs, and the MDT eviction takes priority over the OST evictions. All mirrors except the primary will have INFLIGHT cleared and STALE set.

MDS Failover

When the MDS crashes, AW lock state is lost. On recovery, the MDS uses its durable active-epoch FID set (see Recovery After MDS Failover) to identify files with active epochs. If all clients reconnect, AW locks are replayed and epochs resume normally. If any client is evicted, eviction proceeds as in Client Eviction from MDT — the only difference is how the MDS discovers which files have active epochs: in-memory lock state (live) vs the durable FID set (recovery).

Primary Mirror Write Failure

In case of a write failure to the primary mirror, the clients will attempt to complete all other writes and inform the MDS as usual. The MDS closes the epoch normally (see Epoch Close): the failed primary is marked STALE, and secondaries that completed without error have INFLIGHT cleared (they are now clean and in sync). The next IWM epoch selects a new primary from eligible non-stale IMMEDIATE mirrors — the MDS selects it by examining the write statuses from active writers, finding a mirror with no errors. A clean non-immediate copy, including demoted flash, is not eligible without conditional promotion. If all mirrors had write failures, all mirrors are marked STALE and the file degrades to its pre-IWM state, requiring a full resync — the same outcome as a total write failure in delayed-write FLR today. See Client Loss in this section for what to do if a client is lost entirely.

This should work transparently with minimal changes.

Replacing Mirrors/Permanent Failure

When a mirror is left STALE after a write failure, lamigo needs to be notified to attempt resync. The existing mechanism is changelogs — lamigo already watches changelogs for files with STALE mirrors as part of Hot Pools, so IWM does not require a new notification path. The changelog overhead is modest (5-10% for metadata-intensive workloads, less for IO-intensive). If resync fails repeatedly (e.g., due to permanent OST failure), lamigo could add a new replacement mirror; the failed mirror may be removed later.

These operations require quiescing active writers. They follow the administrative path and its flush dependency requirement (see §2.2.7): resync uses layout EX (via LL_LEASE_RESYNC), while MDS AW EX closes the epoch and forces writers to flush. Their sequencing must not block layout refresh needed to finish that flush. lamigo does not need any special commands — it uses the normal mirror add (lfs mirror extend) and resync (lfs mirror resync) operations. After the operation completes, both locks are released and clients can resume writing.

This means writers will stall during mirror replacement or resync, but they will not fail — they simply block on AW lock acquisition until the operation completes. Since these are separate commands (first add the mirror, then resync it), there will be two quiesce windows. This is acceptable since mirror replacement is an infrequent recovery operation, not a steady-state path.

Conflicting Operations During Write Epochs (Resync, Old Clients)

Clients that understand FLR (delayed-write mirroring, 2.11+) but do not support IWM need explicit handling during active write epochs. These clients can open mirrored files (exp_connect_flr() passes) but lack OBD_CONNECT2_FLR_IMMED_MIRROR and cannot participate in write duplication. Pre-2.11 clients (pre-FLR) are blocked at file open and are not a concern.

The problem: Non-IWM clients can ignore unknown lcme_flags, including IMMEDIATE and INFLIGHT, during ordinary IO. INFLIGHT alone therefore does not stop them reading partially updated secondary mirrors. Checking only write intents is also insufficient: an old client given an already-writable WRITE_PENDING layout may write without another intent when the relevant secondary components are already STALE. It holds no AW lock, so those writes escape epoch accounting.

Design: Close on layout acquisition; stale on write intent. The compatibility boundary is the grant of a usable layout, not just a request explicitly identified as a write. Apply this policy on every path granting or returning a usable layout to a non-IWM client, including LAYOUT_INTENT_ACCESS, layouts supplied on open, and revalidation after invalidation or recovery:

Check exp_connect_immed_mirror() on the requesting export. If an IWM epoch is active, hold the legacy request and serialize the handoff against new epoch admission. Use active-epoch state, not merely the presence of INFLIGHT on a secondary.
Force epoch closure through AW: stop admitting new participants, flush existing writers, collect cancellation LVB error reports, and handle eviction. Complete the normal close transition to RDONLY, clearing INFLIGHT and setting STALE only where the close rules require it. Healthy mirrors remain clean.
For an access/read request, grant the completed RDONLY layout. This is the actual serialized file state, not a client-specific RDONLY view of a still-active or writable layout. The old client may read any clean mirror. RDONLY forces it to send a write intent if it subsequently wants to modify the file.
On that write intent, apply ordinary delayed-write FLR: select the write primary, transition RDONLY → WRITE_PENDING, and mark affected secondary components STALE under existing FLR rules. The old client writes only the primary, without AW. If the original request was already a write intent, process this transition immediately after epoch closure; no artificial RDONLY round trip is required.

A legacy reader pays for epoch closure but does not itself reduce redundancy or create a resync requirement. A legacy writer is allowed outside the epoch and degrades affected mirrors as delayed-write FLR does today. When no IWM epoch is active, existing delayed-write state is handled normally; an ongoing legacy WRITE_PENDING session is not reset to RDONLY on every acquisition.

Returning to IWM: Before opening an epoch, revoke previously granted legacy layouts, including RDONLY layouts held by readers, so old clients cannot continue accessing secondaries during the epoch. Verify that old reads already in flight cannot return secondary data from an invalidated view without the required restart/revalidation; layout revocation is not itself an IO drain. If delayed-write access was granted, also drain or fence outstanding legacy writes before IWM writes begin. Layout invalidation alone is not a flush or durability barrier, and legacy writers hold no AW lock. This handoff must cover dirty cached data, in-flight IO, and retries under the old writable layout. Mirrors staled by legacy writes remain STALE until resync: a new epoch must not make them clean or count them as synchronized participants. Once these requirements are met, eligible clean immediate mirrors can participate in a new epoch. A later legacy layout acquisition closes that epoch again.

Implementation constraint: Serialize both directions: prevent epoch reopening between close and legacy layout grant, and prevent legacy grants racing epoch entry. Existing AW holders must still be able to finish flushes, including layout refreshes or IO restarts. An unconditional layout EX → AW EX wait is insufficient if AW cancellation requires layout CR. The concrete serialization and legacy-write drain/fence mechanism remains to be specified; see §2.2.7.

Legacy clients use the existing full layout refresh/teardown path. Eligible IWM-aware clients may preserve their page cache for non-destructive flag changes as described in §2.2.7.3; that optimization does not change the handoff or durability requirements.

Sustained mixed-client access: Both legacy reads and writes can cause repeated epoch closures and layout refresh costs. Only legacy writes inherently stale healthy secondary components and require later resync. Old clients are excluded from active IWM epochs, not from writing IWM-configured files altogether. Mixed-version operation remains supported under delayed-write semantics between epochs, subject to the handoff rules above.

Resync also closes any active epoch first, regardless of client capability, then proceeds through normal resync serialization (see Replacing Mirrors/Permanent Failure). This differs from an ordinary legacy read, which needs no resync and does not stale healthy mirrors.

Future Enhancements and Alternatives

Cancellation LVB Format and Future Stripe Detail

Decision: Build the extensible message structure in v1, initially carrying only mirror-level errors. V1 neither collects nor transmits stripe error bitmaps. A future extension can add per-OST-object detail for finer repair and immediate EC, while retaining the same mandatory mirror summary and conservative fallback. Per-stripe degradation and repair semantics remain future work.

Message structure from v1:

A versioned header with explicit header and total-message lengths, sufficient to locate the mandatory summary and any following extensions.
A mandatory counted mirror-error bitmap, associated with the cancelling AW lock and its epoch. Bit i denotes participant i in the epoch’s shared immediate-mirror list, ordered by mirror ID. A bit is not a raw mirror ID. Membership changes require an epoch boundary, so ordinary component/layout changes must not silently renumber this list.
An optional sequence of typed, length-delimited detail records. V1 sends no such records. Their framing is defined from the start so a receiver can skip unknown optional detail within a supported message version and still process the mandatory mirror summary.

The exact packed field widths, alignment, and negotiated message limits are to be specified with the cancellation RPC implementation. The structural decision is settled: v1 is not a bare two-byte message, and future detail is not implied merely by a set mirror-error bit. The v1 receiver validates lengths, counts, and the epoch/mirror mapping before combining reports. An absent or invalid mandatory report must not be interpreted as a successful zero-error report; it follows conservative recovery handling. An unsupported top-level version is not covered by the rule for skipping optional records.

V1 semantics and compatibility: Any set mirror-error bit is sufficient to mark that mirror STALE. The MDS ORs valid reports using the shared mirror mapping. A receiver that does not understand optional stripe detail still treats every mirror indicated by the summary as failed as a whole. A future detail-aware receiver may refine repair only when it has complete, valid detail from all relevant error reports; a mirror-level-only report requires whole-mirror fallback. Missing or unsupported detail can never make a mirror with a reported error appear clean.

Future stripe numbering: one flat index space per mirror. Components are ordered within a mirror. Concatenate their stripe lists in that order and number the resulting OST-object slots from zero; numbering does not restart at each component. If the ordered components have stripe counts n0, n1, ..., local stripe s in component k has flat index n0 + ... + n(k-1) + s. For example, two components with 2 and 3 stripes occupy indices 0–1 and 2–4. This is an object-slot index space, not a file-byte address or a count of repeated stripe-sized data chunks. Components without OST-object slots contribute no entries.

A future detail record identifies the mirror, the layout mapping that defines its flat index space, the total number of slots, and the error bitmap. Component IDs do not have to accompany every reported stripe: both sides derive the component/local-stripe pair from the ordered layout. Heterogeneous component geometries are handled by summing the actual per-component counts.

Layout changes during an epoch: Flat indices are interpreted against the layout mapping under which the affected IO was issued. Since an epoch can span layout changes, the client and MDS must preserve that mapping or translate recorded errors before combining detail from different layouts; old bits must never be interpreted using a silently renumbered current layout. If precise translation is unavailable, retain the mirror-error summary and fall back to whole-mirror handling. This requirement does not force epoch closure for ordinary component instantiation. The exact mapping lifetime and detail aggregation mechanism can be designed with the future stripe-reporting extension; v1 needs only the stable mirror mapping.

Sizing: The mandatory bitmap for M immediate participants uses ceil(M / 8) bytes before header and alignment overhead: 2 bytes for 16 participants or 32 for the current maximum of 256 (LUSTRE_MIRROR_COUNT_DEF and LUSTRE_MIRROR_COUNT_MAX). This is bitmap payload size, not the total cancellation report size.

A future stripe bitmap for N flattened OST-object slots uses ceil(N / 8) bytes before record overhead and alignment. For 200, 500, and 2000 slots, the raw bitmap sizes are 25, 63, and 250 bytes respectively. The final wire size also includes the mandatory summary, header, and each detail record’s metadata and padding.

LOV_MAX_STRIPE_COUNT (2000) limits a plain component’s stripe count; composite allocation also accounts for space used by other components in the shared layout xattr. It is not an independent allowance of 2000 stripes for every mirror or a bound on the sum of a mirror’s component stripe counts. Consequently, multiplying 16 mirrors by 2000 stripes does not establish a realizable worst-case report. A maximum must be derived from supported composite layouts and the chosen record encoding, including any multiple mappings retained across an epoch. The RPC implementation must define and enforce that bound, with conservative mirror-level fallback when precise optional detail cannot be supplied within it.

Transport: The report travels from client to MDS in the cancellation request. Current LDLM_CANCEL has no LVB field. Lustre’s variable-sized RPC field machinery can support the new message, but adding the field, associating reports with the correct lock in batched cancellation, and packing/validating it are v1 protocol work. The report’s explicit lengths support extension; they are not a claim that the existing cancel path already transports this data.

Per-extent reporting (deferred): Byte ranges of failed writes within a stripe are a separate possible level of detail. Their representation can grow with the number of disjoint modifications during sustained failures, whereas an object bitmap remains bounded by the reported layout mapping. Range tracking and its repair semantics are not part of v1 or the proposed per-object extension.

Write Ordering Alternatives

v1 uses the primary mirror as locking leader, requiring client-side LDLM locks for all IO including DIO — a performance regression for shared-file DIO workloads that normally operate locklessly. Possible future approaches to restore lockless DIO:

Chained RPC Checksums

One option is to have OSTs maintain a chained checksum of committed writes. In the simplest form, this requires identical RPCs to all mirrors, which requires mirrors to have identical layout geometry — a constraint the v1 BRW page fan-out approach was specifically designed to avoid (see IO Duplication). However, the chained checksum concept could be extended to work with heterogeneous layouts by checksumming at the logical file offset level rather than the RPC level — the OST would chain checksums of (offset, length, data_checksum) tuples as they commit, producing a layout-independent ordering fingerprint. This would require the MDS to compare per-stripe chains after mapping them back to file-level extents, adding complexity but removing the identical-layout restriction. When a write epoch opens, the OST would be informed the stripe object is part of an immediate mirror file. It would take the write checksum from each committed write and chain them together as they are committed (write commits can be ordered by their journal transaction number, even if they are occurring in parallel). The result is a single checksum value which encodes both the writes and their ordering. Non-write operations can be included by checksumming their arguments (e.g., the byte range and fallocate op type for fallocate).

The write primary mirror would be the correct ordering, to which the others are compared. If ordering on a secondary mirror disagrees with the primary, that mirror would be left STALE (not un-staled), requiring a full data resync.

Open questions for this approach include: whether to apply this only to overlapping operations (which requires tracking the extent of all data-modifying operations through a write epoch, but avoids false positives from non-overlapping writes), and recovery scenarios when an OST fails mid-epoch and the checksum chain is incomplete.

A related but heavier-weight idea is having OSTs maintain full write commit vectors (ordered lists of committed write extents) and comparing them across mirrors at epoch close. This subsumes chained checksums but requires significantly more per-OST state and MDS-side comparison logic.

Open Problems

Operation Completion Consistency Model

The application policies below are agreed. The remaining open work is their implementation and integration with the epoch/layout protocol, not a choice between primary-only and all-mirror syscall completion.

Boundary	V1 contract
Ordinary buffered `write()`	May return after data is accepted into the client cache, before mirror RPC submission. Normal asynchronous aggregation; no stable-storage promise at this boundary.
Buffered dirtying / mmap write permission	Covered by AW from acquisition/use of the primary PW write authority, not first at writeback. Dirty data remains the epoch’s responsibility across syscall return and layout refresh.
Mirror RPC / writeback completion	Every attempt has completed or safely terminated before its source buffers and write-ordering protection can be released. An acknowledgment is not by itself proof of durable commit.
DIO completion	Stable storage is required without synchronous flags. Every participating mirror’s updates are committed or recorded as failed and outstanding activity is safely resolved. The primary determines the application result.
`fsync()`, `fdatasync()`, `O_SYNC`, `O_DSYNC`	For the operation’s required data/metadata scope, obtain committed-or-recorded-failed outcomes and safely resolve outstanding activity. Primary success remains success despite secondary failure.
AW release	Quiesce the covered write authority and resolve all its dirty data and IO, including durability and accumulated failure reporting. This is not the same boundary as one operation completing.

Explicit durability requests: Preserve the usual distinction between full data/metadata synchronization and data-integrity synchronization, and between file flushes and synchronous writes’ operation scope. Primary writeback/commit errors, relevant earlier buffered-write errors, and required metadata errors retain normal application error reporting. A secondary-only failure must not be surfaced as an error on a later fsync() when it would be invisible for the write itself. V1 offers no durable-redundancy success requirement; an explicitly requested stricter policy could be considered in the future, consistently across APIs.

No additional global-close barrier: Application completion does not require all clients to release AW, the file-wide epoch to close, or the MDS to publish the final INFLIGHT/STALE layout. A durability request must complete the work in its defined scope; it must not acquire file-wide AW EX merely to satisfy this contract or wait for unrelated writers to stop. AW may remain held for subsequent writes. Local commit/error tracking must survive operation completion until reported, and the durable active-epoch record plus conservative recovery must protect against loss of an unreported outcome. Secondary read eligibility still follows epoch-close rules; a successful fsync() does not itself make secondaries readable.

Remaining discussion and verification: The accompanying iwm_discussion_checklist.md is the shared agenda. P1–P8 combine PW lifetime, cancellation, mmap write permission, cl_conf_set(), layout/AW serialization, and epoch opening/closing/handoff into one protocol design discussion; see §2.2.7. A7 separately covers truncate, punch, and fallocate contracts. A9 covers DIO/backend commit confirmation, sync data/metadata and error-reporting scopes, and a finite local synchronization boundary under concurrent dirtying. The application policies above remain settled while those questions are answered.

Combining Immediate and Non-Immediate Mirrors

Real deployments will combine immediate and non-immediate mirrors on the same file. For example, DDN’s hot pools feature places latent mirrors on HDD that are only populated once a file ages out of a flash tier — these mirrors are not written at write time but filled in later by resync (lamigo). A file might have two flash mirrors (written immediately for HA) and two HDD mirrors (synced lazily for capacity/archival).

The per-mirror IMMEDIATE flag determines membership in immediate write duplication and AW epoch tracking. Healthy immediate mirrors participate; STALE mirrors require resync before they can count as synchronized copies. Mirrors without IMMEDIATE use ordinary delayed-write FLR. No new “immediacy group” or file-level layout state is needed: the existing RDONLY → WRITE_PENDING → SYNC_PENDING states remain singular.

Non-immediate mirrors are not necessarily STALE. Resync can leave HDD clean before promotion, and demotion can leave flash clean but no longer immediate. Before permitting modifying IO that excludes such a copy, its affected components must be marked STALE, as described in Mirror States. This rule is essential to making conditional promotion safe: a copy that missed a write must not still appear clean.

Adding and removing IMMEDIATE are separate, conditional layout operations at epoch boundaries. They change which mirrors participate in subsequent epochs, rather than changing the participant set underneath existing writers. The ordinary AW flush and error-reporting rules still apply.

The tiering lifecycle would work as follows (e.g., migrating from flash to HDD):

The file starts with two immediate flash mirrors.
Add HDD mirrors without IMMEDIATE; they initially need resync.
When the file ages out and should migrate to HDD:
1. Resync HDD. Use ordinary resync to make the intended destination copies clean. This establishes synchronization at completion, not a guarantee that they remain synchronized after writes resume.
2. Conditionally promote HDD. Close and resolve any active epoch, including its flush/error outcomes. While excluding new modifying IO and competing layout changes, verify that the intended HDD copies still exist and are clean over the required extents, then set IMMEDIATE in the same serialized layout transition. A userspace check followed by an unconditional flag update is insufficient. If an intervening write staled the copies, promotion fails and the caller must resync/retry. Promotion never clears STALE itself.
3. Keep both tiers immediate during the overlap. Subsequent epochs duplicate writes to all healthy participating flash and HDD mirrors. This overlap is intentional; the sequence does not introduce a window with no immediate mirrors. Failures can still stale a participant under the normal IWM rules.
4. Conditionally demote flash. Close the overlap epoch and process its actual flush/error results. Clear IMMEDIATE on the selected flash mirrors only if the retained immediate HDD copies still provide the clean coverage and redundancy required by policy. An IMMEDIATE bit alone is not evidence of a healthy destination. If the check fails, leave flash immediate. If flash was the primary, select an eligible HDD primary before the next epoch.
5. Remove flash copies separately. Revalidate that the current retained copies make removal safe, then use the normal mirror-removal operation. Success of the earlier demotion is not a permanent authorization to delete a copy regardless of subsequent failures.

Serialization: Both promotion and demotion require epoch quiescence plus layout coordination: complete existing writers, invalidate obsolete layout views, check the relevant conditions, and publish the new membership before admitting writes under it. Apply the legacy-write drain/fence rules in §2.4.6 as well; AW cannot drain clients that never held it. AW EX alone is insufficient if clients can resume using an obsolete fan-out. The implementation must also satisfy the no-circular-wait requirement in §2.2.7.

Separate operations and progress: Copying, promotion, demotion, and removal need not be one combined operation. If the controller stops between stages, the file remains in a valid state: HDD is copied but not promoted (and may become stale again), both tiers are immediate, or flash is demoted but not deleted. Continuous writes can, however, repeatedly stale HDD between resync and promotion, preventing promotion indefinitely. The separated v1 operations provide no progress guarantee under continuous writes; retries alone do not guarantee completion. Progress then requires a sufficient write-quiescent interval. A future resync-and-promote operation could retain serialization through promotion to close this gap, but is not part of this v1 sequence and would still need writers to quiesce. It would not need to combine promotion with flash demotion.

Interaction with resync: Resync of a non-immediate mirror first closes any active epoch and then uses normal resync serialization. AW EX provides the writer flush; layout/resync coordination excludes conflicting operations, subject to the flush dependency requirement above. Once ordinary resync releases that serialization, writers may resume and stale the non-immediate copy before promotion. That is precisely why promotion must be conditional and why the separate sequence can fail to make progress on a continuously written file.

Old client compatibility: Clients lacking OBD_CONNECT2_FLR_IMMED_MIRROR are excluded from active IWM epochs, not permanently fenced from writing files with IMMEDIATE mirrors. Any legacy acquisition of a usable layout closes an active epoch first. After that close, a read receives RDONLY with healthy mirrors still clean; a write intent invokes ordinary delayed-write FLR and stales affected secondary components. Existing legacy WRITE_PENDING access outside an IWM epoch follows normal FLR rules. Returning to IWM requires revoking legacy layouts and draining or fencing outstanding legacy writes. See §2.4.6.

Remaining implementation work:

Expose conditional IMMEDIATE promotion and demotion through the layout-management interface and tiering controller, including retry handling and the retained-copy redundancy policy. The ordering and safety conditions above are requirements; the concrete serialization mechanism remains to be specified.

Heterogeneous Compression Across Mirrors (Resolved)

Resolved — heterogeneous compression across immediate mirrors does not require special handling. Each mirror’s OSC compresses independently from the original source pages; the redundant compression cost is negligible for 2-3 mirrors. See Bounce Page Reuse for the full analysis.

Immediate Write Mirror - High Level Design

Overview & Design Summary

FLR Today

Immediate Mirroring

Relationship to Erasure Coding

Detailed Design

Write Operation Flow

Primary Mirror Selection and Read Visibility

Active Writer Lock

Lock Semantics

Lock Lifetime and Commit Requirements

Error Reporting via Cancellation LVB

Epoch Close: MDS Processing

Recovery After MDS Failover

Flush Semantics on Lock Release

AW Lock vs Layout Lock Interaction

Why Separate Locks

Behavior During Layout Changes

Layout Lock and Page Cache Interaction

Client Implementation

Write Duplication via BRW Page Fan-Out

Page Consistency During Write Duplication

Buffered IO Completion Semantics

Direct IO Completion Semantics

Implications for Erasure Coding

Compression and Encryption Across Mirrors

Other Modifying Operations

Operation Classification

LOV Sub-IO Fan-Out

AW Lock and Epoch Interaction

Implications for Erasure Coding

Error Management

Fast-fail and Slow-fail Secondary Policies

Failure Handling

Mirror States: STALE and INFLIGHT Flags

Write/Update Error from Client

Client Eviction from OST

Client Eviction from MDT

Client Loss

MDS Failover

Primary Mirror Write Failure

Replacing Mirrors/Permanent Failure

Conflicting Operations During Write Epochs (Resync, Old Clients)

Future Enhancements and Alternatives

Cancellation LVB Format and Future Stripe Detail

Write Ordering Alternatives

Chained RPC Checksums

Open Problems

Operation Completion Consistency Model

Combining Immediate and Non-Immediate Mirrors

Heterogeneous Compression Across Mirrors (Resolved)

Navigation menu

Search