Projects

Trivial Projects

Simple "get your feet wet" coding projects are tracked in JIRA using the easy keyword. These could be attempted by anyone with a knowledge of C programming, and some interest in learning Lustre or Linux kernel programming. Slightly more complex projects can be found in JIRA using the medium keyword and will typically require at least some understanding of the Lustre code.

There are also a large number of issues tracked in the Lustre Coverity Project that may be suitable for developers who want to become more familiar with the code. Note however, that some/many of the issues may not be as obvious to fix as they are presented by the Coverity tool (e.g. just initializing an uninitialized variable may not be the correct fix in some cases), but they will at least point you at code that needs to be fixed. If you are interested in working on some of these issues, please request access to the project.

Current Projects

Feature	Feature Summary	Point of Contact	Tracker	Target Release
File Level Redundancy (FLR) - Erasure Coding	Provides a more space-efficient method for adding data redundancy than mirroring.	James Simmons (ORNL), Bobijam Xu (Whamcloud)	LU-10911	2.17
Metadata Writeback Cache	Overcome link latency for modifying metadata operations esp. noticeable in interactive type of workloads	Qian Yinjin, Oleg Drokin (Whamcloud)	LU-10938	2.17
Enhanced_Adaptive_Compression_in_Lustre	Transparent data compression for the Lustre client and server, primarily utilizing the computational resources on the client nodes to perform compression and decompression operations when possible. This allows compressed data to be sent over the network, as well as stored on the OSTs and MDTs.	Artem Blagodarenko (Whamcloud), Anna Fuchs, Michael Kuhn (Universität Hamburg)	LU-10026	2.17
Asynchronous flocks	Add asynchronous flocks	Andriy Skulysh (HPE)	LU-2525	2.17
Enable folio allocation support for BIO	Accelerate BIO by leveraging Linux folios	Shaun Tancheff (HPE)	LU-17916	2.17
opendir+readdir	Read directory with open	Alexey Lyashkov (HPE)	LU-18448	2.17

Future Projects

Feature	Feature Summary	Point of Contact	Tracker
File Level Redundancy (FLR) - Immediate Write Mirroring	Immediate mirrored writes from the client to multiple OSTs.
Lustre Metadata Redundancy	Redundancy of Lustre services running on MDT0000 and directories		LU-12310
HSM scalability	Improved scalability of HSM based on Lustre llogs		TBD
Improved MDT performance with ChangeLogs	Improved metadata performance when using ChangeLogs		LU-18218
HSM action restartability	Enable copy tools to restart incomplete HSM requests		TBD
HSM import	Enable HSM import with DoM component		LU-16352
File join	Enable file merging by combining multiple layouts		LU-18461
Efficient DoM Migration	Efficiently migrate DoM component to OSTs		LU-13612

Potential Projects

These projects are potential areas of development that are looking for an interested party to work on or sponsor another developer to do. Many of them have more detailed descriptions, but it is worthwhile to contact the lustre-devel mailing list to discuss the project before starting implementation.

Feature	Feature Summary	Complexity	Tracker
ioctl() number cleanups	Clean up Linux IOC numbering to properly use "size" field so that mixed 32- and 64-bit kernel/userspace ioctls work correctly. Attention needs to be paid to maintaining userspace compatibility for a number of releases, so the old ioctl() numbers cannot simply be removed.	1	LU-6202
Updated man pages	Update the online manual pages for Lustre user tools and the lustreapi library. Split the existing lfs.1 and lctl.8 man pages into separate pages for each sub-command, describing options and providing usage examples.	2	LU-4315
Improve testing Efficiency	Improve the performance, efficiency, and coverage of the acceptance-small.sh test scripts.	3
STATX_PROJID	Add support for a new STATX_PROJID flag to the upstream kernel statx() interface for ext4 and XFS. Include this in Lustre and have "lfs find" use this interface.	4	LU-12480
Error message improvements	Review and improve the Lustre error messages to be more useful and follow the Lustre coding style (see Lustre_Coding_Style_Guidelines#Console_Error_Messages. A larger project is to change the core Lustre error message handling to generate better structured error messages so that they can be parsed/managed more easily.	4
Improve QOS Round-Robin object allocator	Improve LOV QOS allocator to always do weighted round-robin allocation, instead of degrading into weighted random allocations once the OST free space becomes imbalanced. This evens out allocations continuously, avoids crazy/bad OST allocation imbalances when QOS becomes active, and allows adding weighting for things like current load, OST RAID rebuild, etc.	6	LU-9
RTDS(Real-Time Dynamic Striping)	A policy based Lustre striping framework	6	LU-9809
NFSv4 ACL Support	Add handling of NFSv4 ACLs to Lustre	6	LU-13862
RPC Replay Signatures	Allow MDS/OSS to determine if client can legitimately replay an RPC, by digitally signing it at processing time and verifying the signature at replay time.	6	LU-15250
Simplified Interoperability	Add functionality so servers can force clients quiesce all opertaions, drop all locks, and flush unreplied RPCs. This allows administrators to halt operations on clients without having to unmount the filesystem. This prevents unintended operations on the server (e.g. during maintenance), reduces recovery operations if a server is being cleanly shut down, and avoids RPC replay issues across major Lustre server version upgrades. Open file recovery should be done by generating new open RPCs on the client, rather than replaying RPCs stored in the replay list, which also allows removing the special-case handling of open RPCs that are already committed. This is partially implemented by the LU-8900 snapshot RPC write barrier, but this only freezes the filesystem for new writes, and does not address the issue of replaying open file handles.	6	LU-5703
Virtual Lustre Block Device	Lustre object lloop driver previously exported a virtual block device to userspace for a file, bypassing the filesystem. Code partly worked but was removed from Lustre due to correctness issues and potential performance problems. It needs to be rewritten to work with newer kernels, by providing a new "backend" for the kernel loop driver and avoid data copies to avoid deadlocks under memory pressure.	6	LU-6585
Swap on Lustre	Implement kernel swapping to Lustre for diskless clients. This needs to work properly under memory pressure, or it will be useless when swap is actually needed.	7
ZFS ZIL Support	Add support for the ZFS Intent Log (ZIL) to the Lustre osd-zfs	8	LU-4009
Local object zero-copy IO	Efficient data IO between a client and a local OST object; optimization to support local clients. Likely implemented as a fast-path connection between the OSC and the local OFD/OSD. Read cache should be kept on the OSD instead of at the client VFS level, so that the cache can be shared among all users of this OST.	9	LU-10191
External HSM Coordinator	Add capability to run HSM coordinator in user space, which could allow 3rd-party control over HSM action queue	7	LU-10968
HSM control of data migration	Allow HSM Coordinator to control data migration (`lfs migrate`), which could allow copy tools to move data in parallel	5	LU-6081
Prioritize HSM requests	Add ability to prioritize HSM requests, especially HSM CANCEL	5	LU-8324 LU-14363

Past Projects

Feature	Feature Summary	Point of Contact	Tracker	GA Release
Imperative Recovery	Servers notify clients (via MGS) where a failed target has restarted to reduce waiting time during recovery.	Jinshan Xiong (Intel)	LU-19	2.2
LFSCK Phase 1 - OI Scrub Solution Architecture	Check and repair Object Index on ldiskfs filesystems via OI Scrub to allow file-level backup/restore of MDT filesystems.	Fan Yong (Intel)	LU-957	2.3
LFSCK Phase 1.5 - FID-in-Dirent and LinkEA Consistency	Enhance LFSCK to work with DNE filesystems, checking the Lustre FID stored in MDT directory entries and the parent LinkEA entries for hard links.	Fan Yong (Intel)	LU-1866	2.4
Shared Secret Key Encryption	Allow shared-secret key encryption and authentication for Lustre network traffic without the use of an external keyserver	Andrew Korty, Jeremy Filizetti (Indiana University)	LU-3289	2.6
LFSCK Phase 2 - MDT-OST consistency	Enhance LFSCK to check layout consistency between MDT file layouts and OST objects.	Fan Yong (Intel)	LU-1267	2.6
LFSCK Phase 3 - DNE consistency	Enhance LFSCK to work with DNE filesystems, including remote directory entries, and OST orphan handling for multiple MDTs.	Fan Yong (Intel)	LU-2307	2.6
UID/GID Nodemap	Allow mapping UID/GID and server-side configuration on a per-client/per-cluster basis	Joshua Walgenbach, Kit Westneat (Indiana University)	LU-4647	2.7
Dynamic LNET Configuration	Introduces a user space script which will read routes from a config file and add those routes to LNET dynamically with the lctl utility. This allows the support of very large routing tables	Amir Shehata (Intel)	LU-2456	2.7
LFSCK Phase 4 - Performance Tuning	Enhance LFSCK performance and efficiency.	Fan Yong (Intel)	LU-6361	2.8
Multiple metadata RPCs	Support of multiple metadata modifications per client (in last_rcvd file) to improve the multi-threaded metadata performance of a single client	Grégoire Pichon (Bull/Atos)	LU-5319	2.8
DNE Phase IIb	Asynchronous Commit of cross-MDT updates for improved performance. Remote rename and remote hard link functionality.	Wang Di (Intel)	LU-3534	2.8
Kerberos Revival	Fix up existing Kerberos code so it is tested working again.	Sébastien Buisson (Bull/Atos)	LU-6356	2.8
Filesystem default OST pool	Allow a default filesystem-wide default OST pool to be specified.	Lai Siyao (Intel)	LU-7660, LU-7335	2.9
UID/GID mapping	Map UID/GID for remote client nodes to local UID/GID on the MDS and OSS. Allows a single Lustre filesystem to be shared across clients with administrative domains.	Stephen Simms (Indiana University)	LU-3291	2.9
Subdirectory Mounts	Ability for client to mount subdirectories of a Lustre filesystem	Wang Shilong (DDN)	LU-28	2.9
Server side advise and hinting	Add support new APIs and utilities for server/storage side advise of accessing file for server cache	Li Xi (DDN)	LU-4931	2.9
Large Bulk IO	Increase the OST bulk IO maximum size to 16MB or larger for more efficient disk IO submission.	Shuichi Ihara (DDN)	LU-7990	2.9
Shared Key Crypto	Allow node authentication and/or RPC encryption using symmetric shared key crypto with GSSAPI. Avoids complexity in configuring Kerberos across multiple domains.	Stephen Simms (Indiana University)	LU-3289	2.9
Multi-Rail LNet	Use multiple LNet network interfaces concurrently to improve reliability and performance	Amir Shehata (Intel), Olaf Weber (SGI)	LU-7734	2.10
TBF policy enhancement	An enhancement of NRS/TBF policy to support complex TBF policy with NID/JOBID expressions	Li Xi (DDN)	LU-5620 LU-7470 LU-8236	2.10
Simplified Userspace Snapshots	Allow snapshots of ZFS targets to be mounted as a coherent filesystem	Fan Yong (Intel)	LU-8900	2.10
Layout Enhancement	Add support for composite layouts on a single file, for File Level Replication, Data on MDT, PFL, etc.	Jinshan Xiong, Niu Yawei (Intel)	LU-3480	2.10
Progressive File Layouts	Allow composite file layouts to be instantiated incrementally during file writes	Jinshan Xiong, Niu Yawei (Intel)	LU-8998	2.10
Project Quota	Allow specifying a "project" or "subtree" identifier for files for accounting to a project, separate from UID/GID.	Shuichi Ihara (DDN)	LU-4017	2.10
NRS Delay policy	Use NRS for fault injection. Intentionally delay request processing to simulate server load.	Chris Horn (Cray)	LU-6283	2.10
Patchless Server	Remove Lustre kernel patches to allow Lustre servers to be more easily ported to new kernels, and to be built against vendor kernels without changing the vendor kernel RPMs.	Oleg Drokin (Intel)	LU-20	2.10
Asynchronous Lock ahead	Allow user space to request LDLM extent locks in advance of need. Intended to optimize shared file IO.	Patrick Farrell (Cray)	LU-6179	2.11
Data on MDT	Allow small files to be stored directly on the MDT for reduced RPC count and improved performance.	Mikhail Pershin (Intel)	LU-3285	2.11
File Level Redundancy (FLR) - Phase 1 Delayed Mirroring	Files can be mirrored on multiple OSTs in the same namespace, with delayed resync of files using userspace tools.	Jinshan Xiong (Intel)	LU-3254	2.11
Lustre Audit with Changelogs	Introduce Lustre Audit feature to be able to track all accesses to files residing on Lustre	Sebastien Buisson (DDN)	LU-9727	2.11
Lazy size on MDT	Lazy size on MDT (LSOM) provides ability of "Lazy" file size from MDS rather than size from OSS today.	Li Xi (DDN)	LU-9538	2.12
Config save/edit/restore	Ability to backup/edit/restore the client/MDS/OSS config llog files after a writeconf for config recovery if the config llog becomes corrupted, and to avoid losing filesystem tunable parameters (including all of the OST pool definitions) stored in the config llog if a writeconf is done.	Ben Evans (Cray)	LU-4939	2.12
Persistent Client Cache	A framework to cache files in storage local to the Lustre client.	Li Xi, Yingjin Qian (Whamcloud)	LU-10092	2.13
Self Extending Layouts	Avoid ENOSPC by leveraging extension spaces in layout	Vitaly Fertman (Cray)	LU-10070	2.13
Foreign Layouts	Allow linking files/directories to arbitrary identifiers (e.g. DAOS datasets)	Bruno Faccini (Intel)	LU-11376	2.13
Data on MDT Phase 2	DNE+DoM optimization, migration, read-on-open, stat() and readdir() returning file data, glimpse-ahead, file create improvements.	Mike Pershin (Whamcloud)	LU-10176	2.13
File heat	Add support new attribute "file heat" of objects to track "hot" files, for client and OFD	Li Xi (Whamcloud)	LU-10602	2.13
OST Pool Quotas	Limit OST pool usage via quotas	Sergey Chermencev (Cray)	LU-11023	2.14
Client-side data encryption	Encrypt files and directories (or possibly just filenames in directories) on the client before sending to the server. This avoids sending unencrypted data over the network, or ever having the data in plaintext on the server (in case of separate decryption from network plus encryption on disk).	Sebastien Buisson (Whamcloud)	LU-12275	2.14
OSP multiple modify requests to MDT	Improve performance of cross-MDT modify operations.	Grégoire Pichon (Atos)	LU-6864	2.15
Directory bulk operations	Bulk metadata readdir/stat interface to speed up "ls -l" operations. Send back requested inode attributes for all directory entries as part of the extended dirent data. Integrate with any proposed API for this on the client. Needs Large Readdir RPCs to be efficient over the wire, since more data will be returned for every entry.	Qian Yinjin (Whamcloud)	LU-14139	2.16