Lustre Timeout Hierarchy
Introduction
The exchange of requests and replies between hosts forms the basis of the Lustre protocol. One host will send a message containing a request to another and then, typically, await a reply from that other host. Messages are prepared at the PtlRPC (Portal Remote Procedure Call) layer. This message is referred to as a PtlRPC request or, more simply, as an RPC. Once an RPC is prepared at the PtlRPC layer, it is passed down to the lower LNet (Lustre Network) layer. The underlying network protocols used to exchange messages are abstracted away via the LNet software layer. The LNet layer is network-type agnostic. It utilizes a Lustre Network Driver (LND) layer to interface with the driver for a specific network type.
This document covers some issues that may arise with Lustre network messaging. Messages can be lost or may take too long to be handled on the target peer. Meanwhile, some other clients may be waiting for the locked resources. The Lustre Distributed Lock Management (LDLM) layer provides the ability of such locking.
The potential for message loss or excessive handling times introduces a need for some failure detection and mechanisms that can recover from these situations with minimal data loss. For example, when a client is not able to return (cancel) a given LDLM lock, it is evicted from the cluster and as a result all the cached data on that client, that has not yet been flushed to servers, are lost.
The main failure detection mechanisms used are timeouts. Once a timeout is exceeded, some recovery action could be done – a resend of the request, possible via another network path, eviction of the client, etc. These mechanisms are used on all the aforementioned layers (LNet, PtlRPC, LDLM).
Some timeouts have a range of definitions [min; max] and some mechanism to identify a correct timeout value for a particular case. PtlRPC uses a mechanism called adaptive timeouts (AT) which estimate how much time the system spends on some action (network message delivery, RPC server processing time) and adding some extra time to estimate what will be a reasonable amount of time to wait before taking any other action.
The main problem is that various timeout settings are not tuned properly by default and, moreover, the various timeouts are not tied to each other. This document discusses several dependencies between timeouts on different layers and makes suggestions for the default settings in several cluster configurations.
Existing Tunables
LNet Tunables
check_routers_before_use
– When LNet is initialized, assume routers are down and ping them before use. Also applies to routers added after LNet has been initialized (i.e. if new router added via CLI then it will be initialized to down and ping’d before use).
dead_router_check_interval (default 60sec)
– interval between checks for a dead router to switch into alive state.
live_router_check_interval (default 60sec)
– interval between checks for a live router to switch into dead state.
router_ping_timeout (default 50sec)
– timeout for an LNet ping request used to determine route aliveness – In HPE’s Lustre 2.12 (and community Lustre 2.13), the router_ping_timeout is only used on LNet routers to determine how long to wait before determining that a peer is down.
New in Lustre 2.12
In HPE’s Lustre 2.12 (and community Lustre 2.13), the dead_router_check_interval and live_router_check_interval parameters are deprecated and replaced with a new alive_router_check_interval parameter. The alive_router_check_interval parameter functions similarly to the old live_router_check_interval parameter. The dead_router_check_interval was deprecated because the LNet health feature has its own mechanisms for determining the health of peer interfaces, and so it was considered redundant to keep this separate parameter just for routers.
alive_router_check_interval (default 60 seconds)
– interval between checks for a live router to switch into dead state.
lnet_health_sensitivity (default 100)
– Value to decrement the health value of interfaces by on error, or increment on successful recovery.
lnet_transaction_timeout (default 50 seconds)
– Maximum seconds to wait for a response from a peer. Used for the response tracking component of the LNet health feature. – Also serves as the timeout value for all discovery pings (which are used to determine router health).
lnet_retry_count (default 2) – Maximum number of times to retry transmitting a message after the initial transmission fails.
– Note: These retries are at the LNet level, as opposed to any resends that may occur at the LND level.
lnet_lnd_timeout ( (lnet_transaction_timeout – 1) / ( lnet_retry_count + 1 ) ) – Transmission timeout used at the LND layer. Derived from the lnet_transaction_timeout and lnet_retry_count (cannot be set directly). The LND specific timeout parameters (see below) will override this value if they are specified. If the LND specific timeout parameters are set then care should be taken to ensure they are less than or equal to: ( (lnet_transaction_timeout – 1) / ( lnet_retry_count + 1 ) )
LND Tunables
InfiniBand (IB) LND
timeout (default 50sec)
– transmission timeout: LND finds this link failure and triggers reconnect;
– Tx descriptors will be aborted if re-connect fails.
peer_timeout (default 180sec)
– mapped to global peer alive timeout (to make per LND/LND instance tunable);
– checked after each event, if the last alive (succeeded) event + peer_timeout exceeded:
marks peer dead, returns an error to upper layer.
retry_count (default 5, max possible 7)
– The maximum number of times that an RDMA transmission is retried if an error occurs.
– IB driver internals, not used by LNet
This parameter is passed down to IB directly.
rnr_retry_count (default 6, max possible 7)
– The maximum number of times that a send operation from the remote peer should be retried on a connection after receiving a receiver not ready (RNR) error. RNR errors are generated when a send request arrives before a buffer has been posted to receive the incoming data.
This parameter is passed down to IB directly. The maximum delay between these retries could be:
• IB_RNR_TIMER_655_36: Delay of 655.36 milliseconds.
The last 2 timeouts if set to 7, will re-try endlessly, and will be limited by IB LND timeout only.
TCP LND
sock_timeout (default 50sec)
– transmission timeout: LND finds this link failure and triggers reconnect;
– Tx descriptors will be aborted if re-connect fails.
peer_timeout (default 180sec)
– mapped to global peer alive timeout (to make per LND/LND instance tunable);
– checked after each event, if the last alive (succeeded) event + peer_timeout exceeded:
marks peer dead, returns an error to upper layer.
RPC Categories
Let’s gather it all together and look what happens once PtlRPC has prepared an RPC:
Client sends a message (LND + RDMA). The message lands on a Router.
Router forwards the message (LND + RDMA). The message lands on a Server.
Server initiates bulk transfer
Server sends a message (LND + RDMA). The message lands on a Router.
Router forwards the message (LND + RDMA). The message lands on Client.
Note: RDMA may happen in the opposite direction depending on IO (read or write).
Server sends reply message (LND + RDMA). The message lands on a Router.
Router forwards the reply message (LND + RDMA). The message lands on client.
Thus, 6 message hops (LND + RDMA). However, not all RPCs follow this pattern. We’ve identified three types of RPCs:
RPC Type1: No reply
This RPC type has only 2 LNet message hops.
An example is sending of the Blocking AST (BLAST) RPC used to revoke locks that have been granted to clients. The goal is to get a cancel RPC in response to the BLAST, so the reply to the BLAST is not to be considered.
RPC Type2: Reply, but no bulk.
This RPC type has only 4 LNet message hops.
RPC Type3: Reply and bulk.
This RPC has 6 LNet message hops as discussed above.
Sending an LNet Message Scenario
Each time LNet sends a message the following pattern is followed:
Choose the best peer network interfaces (NI)
LNet chooses the best peer NI for the final destination on the message originator (client).
LNet chooses the best peer NI for the next-hop (router), if the destination is on a remote network.
LNet transfers the message
LNet invokes the LND send method appropriate for the selected peer NI.
LND sends a connect message to the selected peer NI (either next-hop or final destination for non-routed configurations), if no connection exists yet (the LNet message is queued until an LND connection is established).
On LND timeout:
In multi-rail configuration, LNet may attempt resends before finalizing message. In this case, the process restarts from step 1.
After all resend attempts are completed (if any) an error is returned to the upper layer, if sent from the message originator. (Upper layer notification may be delayed up to lnet_retry_count * (LND Timeout) seconds)
An error cannot be returned to the message originator, if the connect was sent from a router. LND will repeatedly attempt to establish a connection between the router and next-hop/final destination.
LND sends the message
On LND timeout - the same as in (II.B.1)
Recommended Settings
Rely on LNet Message Timeout
Router check is a good solution to detect router is dead, but if a message is sent it makes no sense to wait for alive_router_check_interval + router_ping_timeout, it could be detected directly by the message timeout (lnet transaction timeout) - this is the main mechanism for us to use.
At the same time the router ping, when used, does not differ much from other messages, so its timeout should be set appropriately:
router_ping_timeout = lnet_transaction_timeout
IB and TCP LND Timeout
In general the message delivery by itself in a healthy network takes 0.01 sec, but depends a lot on the load and HW, so better to think about a couple of re-sends and a total timeout we want to wait for them. Let it be 10sec.
IB LND timeout = 10
Similar settings are to TCP sock_timeout.
LNet Retry Count
When there are multiple paths between two LNet peers, it sometimes makes sense for LNet to attempt to resend a message using an alternate local interface, or an alternate destination interface, or both. In these cases, an error is not propagated to the upper layer until a message has been resent a maximum of lnet_retry_count times. Prior to HPE’s Lustre 2.12.4.1, this feature would attempt to resend messages even if neither the message sender nor the message recipient were configured for multi-rail. Thus, it made sense to set lnet_retry_count to 0 if there were not any multi-rail peers in the cluster. Starting with HPE’s Lustre 2.12.4.1, LNet will only attempt to resend messages if the message sender or the recipient are configured for multi-rail, so it is safe to leave this parameter at its default value:
lnet_retry_count = 2
LNet Transaction Timeout
According to the formula above:
lnet_transaction_timeout = lnet_lnd_timeout * (lnet_retry_count + 1) + 1
However, lnet_lnd_timeout may need to be greater than IB/TCP LND Timeout to cover ACK/REPLY for some messages, thus:
lnet_transaction_timeout >= (IB/TCP LND Timeout) * (lnet_retry_count + 1) + 1
With our recommended LND timeouts for o2iblnd and socklnd, and the default values for the LNet parameters, we have:
10* 3 + 1 = 31 and it is < the default 50
Note: we recommend setting the per-LND timeouts, not lnet_lnd_timeout.
However:
Let's limit LNet transaction timeout by just 1 link failure (see the section 4 above) and the time spent on the recovery of this failure.
‘1 link failure’ rule means in particular that lnet_transaction_timeout should cover exactly 1 failure too, and success on all the other links (5 or even more if it has to wait for ACK/REPLY too) - later is already covered by 1 extra second;
PtlRPC must give LNet (and the LND layer) enough time to do its re-tries first and only after that to timeout.
Thus, the recommendation would be:
lnet_transaction_timeout = (IB/TCP LND Timeout) * (lnet_retry_count + 1) + 1 = 31 and therefore we should automatically have:
lnet_lnd_timeout = IB/TCP LND Timeout
Glossary
To simplify the following calculations of the proposed timeouts, there are some abbreviations are introduced.
LTT - LNet transaction timeout, based on either IB or TCP LND timeout.
AT - adaptive timeout, this is already used in the Lustre code for RPC and NET (= AT_RPC + AT_NET), it gathers the time statistics of an operation and provides an recommended timeout for a next operation
AT_RPC - average RPC processing time on server, includes the time the RPC is sitting in the server queues waiting for the handling;
AT_NET - an average network latency time to get the RPC delivered to the server and a reply back. Includes the time the RPC is sitting in the router queues waiting for the sending to the server. Does not include the server handling time and the time spent on sitting in the server queues.
AT_MIN - the minimal limit for each of AT_RPC and AT_NET.
RT_RPC - the RPC server processing timeout estimation made on top of the measured AT_RPC, the current implementation is at_est2timeout(AT_RPC)
RT_NET - the network latency timeout estimation made on top of measured AT_NET, the current implementation is AT_NET
RT - the total RPC timeout estimation, = RT_RPC+RT_NET
(Note: this is not rq_timeout, which is at_est2timeout(AT_RPC) only!!!)
This includes the network travel time, and is therefore the maximum time from the perspective of the client it is ready to wait, on the basis of measured AT.
CT - the connect RPC timeout, = RT(connect) = CONNECT_INIT + AT_NET(connect)
OT - overhead time, the time the RPC is sitting in the queues on client waiting for sending.
RMT - the recovery maximum time, needed for a recovery from 1 network issue. This will be calculated below.
AT_LWT - an expectation of the lock waiting time based on the gathered AT statistics.
LWT - lock waiting timeout, based on AT_LWT, how large timeout should be given to a client to cancel a lock which has a BLAST sent (returned by ldlm_bl_timeout()). Will be calculated below.
AT_PT - an expectation how much prolong time still needs to be added to LWT to not evict the client too early.
PT - prolong timeout, based on AT_PT, how much time to add to the LWT once an IO is handled on the server and the lock cancel is still pending. Will be calculated below.
A couple of more details in the below sub-sections.
at_rpc_min and at_net_min
The *_min values should designate the minimum amount of time the system should wait in “uncrowded” circumstances. This should cover expected failure cases as well (such as packet drop) that we want the system to provide resilience. (The Adaptive part of AT will increase the actual timeouts above these *_min values as servers and networks get loaded, but is not meant to cover network failures.)
RT = at_est2timeout(AT_RPC) + AT_NET, where each on AT_* >= at_min. At the same time, these 2 parts are different by nature, so it would be good to separate them to at_rpc_min and at_net_min in future (note, they both are referenced as at_min in the code now)
at_net_min
What to be included to at_net_min?
It should obviously cover the RPC type3 (6 LND message hops in the “RPC categories” paragraph above), timeouts with possible re-connects & re-sends, and in addition the time spent on the router. However, let’s not consider the worst ever case.
Recommendations:
According to the section 4 above at_net_min must cover LTT + overhead of sitting in the router queue (meaning that the message delivery by itself in a healthy network takes 0.01 sec)
Let the overhead for the waiting in the router queues to be 5sec as well
I.e. the recommended at_net_min = LTT + ROUTER = 31 + 5 = 36sec
at_rpc_min
What is to be included to at_rpc_min?
This is the processing time on server, this (AT) statistics is properly gathered and delivered on client.
The recommended at_rpc_min is 5sec.
Currently recommended settings
Currently the existing tunings are slightly different, at_rpc/at_net are still not implemented, they both refer to at_min. As the overall RPC timeout should not be larger than at_rpc_min + at_net_min, we can just make them equal:
at_min = (at_rpc_min + at_net_min) / 2 = (5+36) / 2 = 21 sec
Timeout Use-Cases
Let’s consider possible real use cases according to the LNet sending scenarios described above.
Network flap or similar
It does not matter here if this is the direct client link or after router, because only an LNet re-send is needed. The scenario:
LNet sends a message;
On IB LND timeout: LND sends re-connect;
LNet re-sends the message
So the overhead is:
IB LND timeout + 1 message (LND connect)
Router is dead – Router switch is needed
This case is about the direct client link problem. The scenario:
LNet sends a message;
On IB LND timeout: LND sends re-connect;
On IB LND timeout:
LNet marks the peer as DEAD
returns an error to PtlRPC;
PtlRPC sends re-connect (another router)
PtlRPC re-sends the RPC
So the overhead here is LTT and PtlRPC re-connect (which is a full RPC not just 1 LNet message, i.e. 4 LNet messages in a routed environment, RPC type2 above):
LTT + AT(connect)
Note: AT is used for connect RPC, not RT, i.e. not the timeout time the client is going to wait, but the measured operational time for this RPC.
However, the router may have several ports and only 1 of them is dead. It is not clear at the beginning what the case is (dead port or router), thus to avoid getting even larger timeouts, it involves a requirement:
Requirement 1:
Cluster is to be configured in a way that on a _network_ error the PtlRPC re-connect would always take a different router (if possible)!!! not just another port on the same router.
A NID is dead – A NID switch is needed
The difference from the previous cases is that this is not the direct client link problem, the message is delivered on the router and fails to get delivered on the server. Thus, whenever LNet is trying to do on the router does not matter, it is all covered by the PtlRPC timeout.
The scenario:
PtlRPC sends an RPC;
On RPC timeout: PtlRPC sends re-connect (the same router, the same NID);
On RPC timeout: PtlRPC sends re-connect (another router, another NID);
PtlRPC re-sends the RPC;
So the overhead here is an original RPC timeout, the PtlRPC connect timeout, another PtlRPC connect handling time.
As seen above, the first RPC does not get a network error so does not change the router. However, at the reconnect RPC timeout, it is already a good time to do it. At the same time, the server may have several NIDs, and only one of them may die. It is not clear at the beginning what the case is (dead link to the server or the server NID), thus to avoid getting even larger larger timeouts, it involves another requirement.
Requirement 2:
Cluster is to be configured in a way that on a timeout from a Connect RPC, the next PtlRPC connect would always take a different router and different NID (if possible)!!!
Having said this, the RMT could be calculated as the maximum of the issues above:
RMT=max(LND timeout + LND connect, LTT + AT(connect), RT(rpc) + RT(connect) + AT(connect))=[the latest is the largest]= RT(rpc)+RT(connect)+AT(connect)
LWT, lock waiting timeout
The main question of this document is how large ldlm_callback_timeout should be comparing with RPC timeout?
Ldlm_callback_timeout covers the BLAST rpc, IO or lock CANCEL – timeout is accounted up to the first of them, thus IO initiate “prolong” and the next cycle of the timeout starts. Also, their replies are not involved, only the delivery and the service time.
Each of these RPC (their compound LNet parts) may timeout, however BLAST means there are other clients which are waiting for the resource and even if the client is alive but does not want to cancel the lock, it must be evicted. Therefore, only a reasonable amount of time to be given to such a client, precisely for a recovery from 1 failure only.
The AT statistics is already gathered for the sequence of BLAST without a reply +IO or +CANCEL without a reply - exp_bl_lock_at per each export.
AT_LWT=AT(exp_bl_lock_at) + RMT(BLAST or IO or CANCEL)
The longest recovery time would be the IO as it has bulk messages.
AT_LWT
= at_get(exp_bl_lock_at) + RT(IO) + RT(connect) + AT(connect)
= at_get(exp_bl_lock_at) +
RT_RPC(IO) + RT_NET(IO) + ← RT(IO)
INIT_CONNECT + RT_NET(connect) + ← RT(connect)
AT_RPC(connect) + AT_NET(connect) ← AT(connect)
The connect and BLAST RPCs are both belong to the rpc type2 category above, in a healthy network their AT_NET’s could be considered as about the same if sent through the same router. Routers could be different, however it is implied the HW, setting are the same and the load is more or less balanced on routers and therefore the statistics is roughly the same, thus we could replace AT_NET(connect) by AT_NET(BLAST), which in its turn has its AT statistics on the server. Let’s do the same for IO.
AT_LWT =
at_get(exp_bl_lock_at) + RT_RPC(IO) + INIT_CONNECT + AT_RPC(connect) + 3*RT_NET(BLAST)
All these stats exist on server.
This is the expected time of getting CANCEL RPC back for a BLAST request having 1 failure scenario. However, we need to make a timeout on base of this expectation, i.e. to add some extra time, e.g. to pass the final value through at_est2timeout() as well:
LWT = at_est2timeout( at_get(exp_bl_lock_at) + AT_RPC(IO) + INIT_CONNECT + AT_RPC(connect) + 3 AT_NET(BLAST) )
Prolong Timeout (PT)
The lock timeout does not lead to the client eviction if at the time of the timeout some IO presents on the server under this lock - it is clear the client is alive and is doing something for getting the request lock cancelled, we should give the client some more time. At the same time, to avoid a case the lock timeout happens immediately after IO RPC completes, by the time the IO reply is to be sent, the timeout is to be prolonged. Moreover, the original lock callback timeout is calculated until any of the involved RPC comes to the server, thus it is supposed to prolong the lock on the same value – PT is not different from LWT in this regard, thus:
PT = LWT
Aries Networks
Aries network has a unique feature, 60sec of quiescent time (QT). This is the time needed to stop the network and do a hardware re-route around failed parts, this time is pretty fixed by Aries HW behavior. If the network is quiesced, all the LNet messages are just frozen for 60s and resent after that.
Once quiescent period finishes, message timeouts are re-set, so LNet messages do not timeout. PtlRPC works as usually, i.e. if a timeout occurs, the RPC is resent.
The Aries network uses an internal timeout of 60 seconds. The resliability and resiliency characteristics of the Aries network allow us to disable LNet retries (set lnet_retry_count = 0), so that overall LTT is a reasonable value.
LTT = (GNILND timeout) * (lnet_retry_count + 1) + 1 = 60 * 1 + 1 = 61.
AT_NET_MIN = LTT + ROUTER = 61 + 5 = 66
AT_RPC_MIN = 5
AT_MIN = (at_rpc_min + at_net_min) / 2 = (5 + 66) / 2 = 36
For LWT let’s consider only 1 network recovery or 1 network re-configuration:
LWT=at_est2timeout( AT(exp_bl_lock_at )+RMT(IO) )
Whereas the RMT describes the recovery period, the quiescent time should replace the original IO and connect timeouts (as only QT is larger than an RPC timeout, a re-connect is required):
RMT(IO) = max(RT(IO) + RT(connect) + AT(connect), QT)
LWT = at_est2timeout( at_get(exp_bl_lock_at) +
max(RT(IO) + INIT_CONNECT + AT_NET(connect), QT) + ← RT(IO) + RT(connect)
AT_RPC(connect) + AT_NET(connect)) ← AT(connect)
LWT = at_est2timeout( at_get(exp_bl_lock_at) +
max(AT_RPC(IO) + INIT_CONNECT + 2 AT_NET(BLAST), QT) +
AT_RPC(connect) +AT_NET(BLAST))
Again, PT=LWT
Overall Recommendations
Non-routed environment
IB LND timeout = TCP sock_timeout = 10 sec
lnet_retry_count = 2
lnet_transaction_timeout = 31
at_min = 21sec (or if at_net_min & at_rpc_min are introduced 36 and 5 correspondingly)
ldlm_bl_timeout() should return
= at_est2timeout( at_get(exp_bl_lock_at) + AT_RPC(IO) + INIT_CONNECT + AT_RPC(connect) + 3 AT_NET(BLAST) )
but never less than ldlm_enqueue_min
prolong_timeout() should return the same as ldlm_bl_timeout()
Routed environment
IB LND timeout (= TCP sock_timeout) = 10sec
lnet_retry_count = 2
lnet_transaction_timeout = router_ping_timeout = 31
at_min = 21sec (or if at_net_min & at_rpc_min are introduced 36 and 5 correspondingly)
On a network error, the PtlRPC re-connect must always take a different router.
On a connect RPC timeout, the next PtlRPC connect must always take a different router and different NID.
ldlm_bl_timeout() should return
= at_est2timeout( at_get(exp_bl_lock_at) + AT_RPC(IO) + INIT_CONNECT + AT_RPC(connect) + 3 AT_NET(BLAST) )
prolong_timeout() should return the same as ldlm_bl_timeout()
Aries Networks (routed with quiescent periods)
GNILND Timeout = 60 sec
lnet_retry_count = 0
lnet_transaction_timeout = router_ping_timeout = 61 sec
at_net_min = 66 sec (if at_net_min implemented) at_rpc_min = 5 sec (if at_rpc_min implemented)
at_min = 36 sec
On a network error, the PtlRPC re-connect must always take a different router.
On a connect RPC timeout, the next PtlRPC connect must always take a different router and different NID.
Quiescent_time (QT) = 60sec
Quiescent_time_extra (QT_extra) = 10sec
ldlm_bl_timeout() should return
= at_est2timeout( at_get(exp_bl_lock_at) + max(AT_RPC(IO) + INIT_CONNECT + 2 AT_NET(BLAST), QT) + AT_RPC(connect) + AT_NET(BLAST))
prolong_timeout() should return the same as ldlm_bl_timeout()
Future Changes
Introduce at_rpc_min and at_net_min instead of a common at_min
Introduce QT & QT_extra module parameters.
Remove router_ping_timeout as it doees not differ by nature from the standard lnet_transaction_timeout.
Open Questions
Q1. IB LND timeout = 8sec, with 2 retries, is it OK? To be tested on scale to confirm...
Q2. LWT_MAX is not defined – what is a reasonable upper limit for lock callback timeout?
It can become 6 * at_max + INITIAL_CONNECT = 3605sec now – too large.
Q3. We have used at_min = 40 on Aries for a very long time. Even though the calculation above shows we could lower it to 36, is it worthwhile to qualify that change and roll it out to the field?
Q4. lnet_retry_count can be set to 0 if MR is disabled. Is it worth it to have separate guidance for MR/non-MR configurations?
Q5. Does at_min on a server need to be >= at_min on clients? What about servers connected to multiple client systems over different networks. Should server-side tunables be the max of any/all client systems?