Lustre Service Thread Tuning

Overview

Lustre servers use service threads to process incoming RPCs from clients. The number of threads directly affects how many requests a server can handle concurrently. Too few threads cause queuing and high latency; too many threads waste memory and cause excessive context switching.

This page covers thread tuning for OSS and MDS services, CPU binding with CPTs, and guidelines for when to adjust thread counts.

For general tuning information, see Lustre Tuning.

OSS Thread Tuning

Default Thread Calculation

The OSS calculates its default thread count based on available memory and CPUs:

• Base: 1 thread per 128 MB of RAM per CPU core
• Minimum: 2 threads
• Maximum: 512 threads (hard limit)

On a server with 256 GB RAM and 32 cores, the default would be calculated as:

(256 * 1024 / 128) * 32 = up to 512 threads (capped at max)

In practice, Lustre applies internal heuristics and the actual count is typically lower. Check the current thread count:

lctl get_param ost.OSS.ost_io.threads_started

Setting OSS Threads at Load Time

Set the thread count via module parameters in /etc/modprobe.d/lustre.conf:

options ost oss_num_threads=128

This sets the number of ost_io service threads. The value takes effect when the ost module is loaded (at mount time).

You can also set minimum and maximum bounds:

options ost oss_max_threads=256

Runtime Thread Tuning

Adjust threads dynamically without a restart:

# View current values
lctl get_param ost.OSS.ost_io.threads_min
lctl get_param ost.OSS.ost_io.threads_max
lctl get_param ost.OSS.ost_io.threads_started

# Increase the maximum (threads will be started on demand)
lctl set_param ost.OSS.ost_io.threads_max=256

# Increase the minimum (threads start immediately)
lctl set_param ost.OSS.ost_io.threads_min=64

Note: You can increase threads_min to immediately start more threads. However, threads cannot be reduced below the current threads_started count without a restart.

Other OSS Services

The ost_io service handles bulk I/O and is the primary tuning target. Other OSS services have their own thread pools:

lctl get_param ost.OSS.ost.threads_started         # metadata-like RPCs
lctl get_param ost.OSS.ost_seq.threads_started      # sequence allocation
lctl get_param ost.OSS.ost_create.threads_started   # object precreation

These rarely need tuning unless you see specific bottlenecks in the service request queues.

MDS Thread Tuning

Default Thread Calculation

The MDS uses a similar calculation but with a lower default maximum:

• Default maximum: 64 threads
• Absolute hard maximum: 1024 threads

For large clusters (thousands of clients), the default 64 threads may be insufficient.

Check current values:

lctl get_param mds.MDS.mdt.threads_started

Setting MDS Threads at Load Time

options mds mds_num_threads=128

Add this to /etc/modprobe.d/lustre.conf on the MDS node.

Runtime MDS Thread Tuning

lctl get_param mds.MDS.mdt.threads_min
lctl get_param mds.MDS.mdt.threads_max
lctl get_param mds.MDS.mdt.threads_started

lctl set_param mds.MDS.mdt.threads_max=256
lctl set_param mds.MDS.mdt.threads_min=128

Other MDS Services

lctl get_param mds.MDS.mdt_readpage.threads_started   # readdir RPCs
lctl get_param mds.MDS.mdt_setattr.threads_started    # setattr RPCs
lctl get_param mds.MDS.mdt_fld.threads_started        # FLD lookups
lctl get_param mds.MDS.mdt_seq.threads_started        # sequence service

The mdt_readpage service may need tuning for workloads with many large directory listings.

CPU Binding with CPTs

Lustre uses CPU Partition Tables (CPTs) to bind service threads to specific CPU cores. This improves cache locality and reduces cross-NUMA memory access.

Configuring CPTs

Set CPT options in /etc/modprobe.d/lustre.conf:

# Bind MDS threads to specific CPUs (comma-separated CPU lists or ranges)
options lnet cpu_pattern=""

# Bind MDS to specific CPTs
options mds mds_num_cpts=2

# Bind OSS I/O threads to specific CPTs
options ost oss_cpts="[0,1]"
options ost oss_io_cpts="[0,1]"

CPT Syntax

CPTs are specified as bracket-enclosed CPU lists:

# Two CPTs, each with specific cores
options lnet cpu_pattern="[0-15] [16-31]"

This creates two CPTs:

• CPT 0: cores 0–15
• CPT 1: cores 16–31

Service threads are then distributed across the specified CPTs.

Viewing CPT Configuration

cat /sys/kernel/debug/lnet/cpu_partition_table

This shows how CPUs are grouped and which CPTs are in use.

Monitoring Thread Utilization

Check whether threads are busy (a sign you may need more):

# View request queue depth
lctl get_param ost.OSS.ost_io.req_waittime
lctl get_param ost.OSS.ost_io.req_active

# Thread utilization
lctl get_param ost.OSS.ost_io.threads_started
lctl get_param ost.OSS.ost_io.threads_busy

If threads_busy is consistently close to threads_started, threads are saturated and more may be needed.

When to Increase Threads

• High thread utilization — threads_busy consistently near threads_started.
• Growing request queues — req_waittime increasing, clients experiencing latency.
• Many clients — large clusters with hundreds or thousands of clients generating concurrent RPCs.
• Mixed workloads — combining metadata-heavy and I/O-heavy workloads on the same server.

When NOT to Increase Threads

• CPU saturation — if CPUs are already at 100%, adding threads increases contention without improving throughput. Profile with top, mpstat, or perf.
• Memory pressure — each thread consumes memory for its stack and request buffers. On memory-limited servers, more threads can trigger OOM conditions.
• Storage bottleneck — if the underlying storage (RAID, SSDs) is the bottleneck, more threads just queue more requests without improving I/O rates. Check with iostat.
• Lock contention — very high thread counts can increase contention on internal locks, actually reducing performance. This is rare but can occur above 256+ threads.

Summary of Key Parameters

Parameter	Location	Default	Description
oss_num_threads	/etc/modprobe.d/lustre.conf	auto	OSS ost_io thread count at load time
oss_max_threads	/etc/modprobe.d/lustre.conf	512	OSS ost_io maximum thread count
mds_num_threads	/etc/modprobe.d/lustre.conf	auto (max 64)	MDS thread count at load time
threads_min	lctl set_param	varies	Minimum threads (runtime adjustable)
threads_max	lctl set_param	varies	Maximum threads (runtime adjustable)
oss_cpts	/etc/modprobe.d/lustre.conf	all CPTs	CPTs used for OSS services
oss_io_cpts	/etc/modprobe.d/lustre.conf	all CPTs	CPTs used for ost_io service specifically
mds_num_cpts	/etc/modprobe.d/lustre.conf	all CPTs	Number of CPTs for MDS services

See Also

• Lustre Tuning
• Lustre Health Checks
• Lustre IO Monitoring