User talk:Hornc

From Lustre Wiki
Jump to navigation Jump to search

Welcome to Lustre Wiki! We hope you will contribute much and well. You will probably want to read the help pages. Again, welcome and have fun! Webmaster (talk) 10:42, 6 May 2015 (PDT)

Understanding Lustre Evictions

An eviction in Lustre is a server's way of revoking resources from a client that the server perceives as misbehaving. Servers take this action to ensure the file system remains usable when a client fails or otherwise misbehaves. When a client is evicted all of its locks are invalidated. As a result, any of the client’s cached inodes are invalidated and any dirty pages must be dropped.

Servers can evict clients if clients do not respond to certain server requests in a timely manner, or if clients do not communicate with the server at regular intervals. Evictions of the latter sort are carried out by a server feature called the ping evictor.

The ping evictor exists to prevent the problem of cascading timeouts #1. Since message timeouts are used to determine a connection’s state a problem occurs where de- pendencies between requests result in implicit dependencies between connections. If a client holds a resource and dies the server would not realize this until a conflicting request for the resource is made. The server would then need to wait for a message timeout to detect the failed client. This can cascade as the number of resources held by a client increases or the number of dead clients increases. The ping evictor helps with this problem by proactively detecting failed clients and reclaiming their resources via eviction.

Idle clients are expected to communicate with Lustre servers at regular intervals by sending servers a Portal RPC (PtlRPC) ping. The interval is equal to obd_timeout / 4 where obd_timeout is a config- urable Lustre parameter and defaults to 100 seconds for a 25 second ping interval (Note, Cray has historically used a 300 second obd_timeout). The ping evictor on a server keeps track of these client pings. If a particular client has failed to deliver a ping within 1.5*obd_timeout seconds1, and the server has not seen any other RPC traffic from that client, then the ping evictor will evict the client. Clients can exhibit misbehavior for a number of reasons including: client side bugs, a kernel panic or oops, heavy load, and serious Lustre errors known as LBUGs. In addi- tion, component failures external to the client may result in client evictions. We’ll discuss this topic in more depth in section V.

A client will learn that it has been evicted the next time it connects to the server. When the client learns of its eviction it must drop all locks since they’ve been invalidated by the server, and it must drop all dirty pages since it no longer has the requisite locks.

Clients may be unaware of their eviction if they do not have any outstanding user request and previous requests were buffered. This behavior is POSIX semantics, so appli- cations need to check return codes or use fsync(2). Unaware users may call this silent data corruption.

References

1.

Peter Braam, Alex Tomas. (2008, February 9). The cascading timeouts problem and the solution [Online]. Available: http://wiki.old.lustre.org/images/f/f0/Cascading-timeouts-hld.pdf