Projects: Difference between revisions

From Lustre Wiki
Jump to navigation Jump to search
m (→‎Potential Projects: add URL for batched directory lookups)
(38 intermediate revisions by 5 users not shown)
Line 1: Line 1:
__TOC__
__TOC__
== Trivial Projects ==
Simple "get your feet wet" coding projects are tracked in JIRA using the [https://jira.whamcloud.com/issues/?jql=labels%20%3D%20easy%20and%20resolution%20%3D%20unresolved '''easy''' keyword].  These could be attempted by anyone with a knowledge of C programming, and some interest in learning Lustre or Linux kernel programming.  Slightly more complex projects can be found in JIRA using the [https://jira.whamcloud.com/issues/?jql=labels%20%3D%20medium%20and%20resolution%20%3D%20unresolved '''medium''' keyword] and will typically require at least some understanding of the Lustre code.


== Current Projects ==
== Current Projects ==
Line 5: Line 8:
{| class="wikitable sortable"
{| class="wikitable sortable"
|-
|-
! Feature !! Feature Summary !! Point of Contact !! Tracker !! Target Date (YYYY-MM)
! Feature !! Feature Summary !! Point of Contact !! Tracker !! Target Release
|-
| [[File_Level_Redundancy_Solution_Architecture#Phase_4:_Erasure_Coded_Striped_Files|File Level Redundancy (FLR) - Phase 3 Erasure Coding]] || Provides a more space-efficient method for adding data redundancy than mirroring. || Bobijam Xu (Whamcloud) || [https://jira.whamcloud.com/browse/LU-10911 LU-10911] || 2.15
|-
|-
| [[Enhanced Adaptive Compression in Lustre]] || Introduce compression for the Lustre client and server || Anna Fuchs [https://wr.informatik.uni-hamburg.de/people/michael_kuhn Michael Kuhn] (Universität Hamburg) || [https://jira.hpdd.intel.com/browse/LU-10026 LU-10026] || 2019-02
| [[External HSM Coordinator]] || Add capability to run HSM coordinator in user space, which could allow 3rd-party control over HSM action queue || Ben Evans (Whamcloud) || [https://jira.whamcloud.com/browse/LU-10968 LU-10968] || 2.15
|-
|-
| [[File Level Replication High Level Design|File Level Redundancy Phase 1]] || Solution Architecture and HLD, Exclusive Open, Mirror Layout, Layout Modification Method, Read Only Mirror || Jinshan Xiong (Intel) || [https://jira.hpdd.intel.com/browse/LU-3254 LU-3254] || 2017-09
| [[HSM control of data migration]] || Allow HSM Coordinator to control data migration (`lfs migrate`), which could allow copy tools to move data in parallel || Ben Evans (Whamcloud) || [https://jira.whamcloud.com/browse/LU-6081 LU-6081] || 2.15
|-
|-
| [[Lustre Audit with Changelogs]] || Introduce Lustre Audit feature to be able to track all accesses to files residing on Lustre || Sebastien Buisson (DDN) || [https://jira.hpdd.intel.com/browse/LU-9727 LU-9727] || 2017-10
| [[Enhanced Adaptive Compression in Lustre]] || Introduce compression for the Lustre client and server || Anna Fuchs [https://wr.informatik.uni-hamburg.de/people/michael_kuhn Michael Kuhn] (Universität Hamburg) || [https://jira.whamcloud.com/browse/LU-10026 LU-10026] || 2.15
|-
|-
| [[ZFS ZIL Support]] || Add support for the ZFS Intent Log (ZIL) to the Lustre osd-zfs || Alex Zhuravlev (Intel) || [https://jira.hpdd.intel.com/browse/LU-4009 LU-4009]||2017-09
| [[Metadata Writeback Cache]] || Overcome link latency for modifying metadata operations esp. noticeable in interactive type of workloads || Oleg Drokin, Qian Yinjin (Whamcloud) || [https://jira.whamcloud.com/browse/LU-10938 LU-10938] || 2.16
|-
|-
|}
|}
Line 23: Line 28:
! Feature !! Feature Summary !! Point of Contact !! Tracker
! Feature !! Feature Summary !! Point of Contact !! Tracker
|-
|-
| [[RTDS(Real-Time Dynamic Striping)]] || A policy based Lustre striping framework || Li Xi (DDN) || [https://jira.hpdd.intel.com/browse/LU-9809 LU-9809]
| [[RTDS(Real-Time Dynamic Striping)]] || A policy based Lustre striping framework || Li Xi (Whamcloud) || [https://jira.whamcloud.com/browse/LU-9809 LU-9809]
|-
| [[Persistent Client Cache]] || A new framework of providing a group of local cache on Lustre client. || Li Xi (DDN) || [https://jira.hpdd.intel.com/browse/LU-10092 LU-10092]
|-
| [[File Level Replication High Level Design|File Level Redundancy Phase 2]] || Data Redundancy with immediate asynchronous write from client. || Jinshan Xiong (Intel) || [https://jira.hpdd.intel.com/browse/LU-3254 LU-3254]
|-
| [[File Level Replication High Level Design|File Level Redundancy Phase 4]] || Data Redundancy with erasure coding of files. || Jinshan Xiong (Intel), Andreas Dilger (Intel) || [https://jira.hpdd.intel.com/browse/LU-3254 LU-3254]
|-
| [[OSP multiple modify requests to MDT]] || Improve performance of cross-MDT modify operations. || Grégoire Pichon (Atos) || [https://jira.hpdd.intel.com/browse/LU-6864 LU-6864]
|-
| [[OFD Read Cache]] || Server side persistent read cache with SSD, NVMe. One of storage cache tier between OSS Readcache with server memory and Storage || Li Xi (DDN) ||
|-
|-
| [[Fileheat based Policy engine ]] || Add support new attribute "file heat" of objects to track "hot" files, for OFD read cache as a policy engine || Li Xi (DDN) ||
| [[OSP multiple modify requests to MDT]] || Improve performance of cross-MDT modify operations. || Grégoire Pichon (Atos) || [https://jira.whamcloud.com/browse/LU-6864 LU-6864]
|-
|-
| [[Directory Level Snapshot]] || Directory level snapshot (DL-SNAP) is designed for user level file backups. DL-SNAP will be implemented by using copy-on-write mechanism on top of ldiskfs without modification of disk format. || Kenichiro Sakai (Fujitsu) ||
| [[Directory Level Snapshot]] || Directory level snapshot (DL-SNAP) is designed for user level file backups. DL-SNAP will be implemented by using copy-on-write mechanism on top of ldiskfs without modification of disk format. || Kenichiro Sakai (Fujitsu) || [https://jira.whamcloud.com/browse/LU-11512 LU-11512]
|-
|-
| [[Lazy size on MDT]] || Lazy size on MDT (LSOM) provides ability of "Lazy" file size from MDS rather than size from OSS today. || Yingjin Qian (DDN) || [https://jira.hpdd.intel.com/browse/LU-9538 LU-9538]
| [[File_Level_Redundancy_Solution_Architecture#Phase_2:_Immediate_Redundancy_and_Optimized_Resync|File Level Redundancy (FLR) - Phase 2 Immediate Write Mirroring]] || Immediate mirrored writes from the client to multiple OSTs. || Bobijam Xu (Whamcloud) ||
|}
|}


Line 56: Line 51:
| [[Updated man pages]]
| [[Updated man pages]]
||Update the online manual pages for Lustre user tools and the lustreapi library.  Split the existing lfs.1 and lctl.8 man pages into separate pages for each sub-command, describing options and providing usage examples.
||Update the online manual pages for Lustre user tools and the lustreapi library.  Split the existing lfs.1 and lctl.8 man pages into separate pages for each sub-command, describing options and providing usage examples.
|| 2 ||[https://jira.hpdd.intel.com/browse/LU-4315 LU-4315]
|| 2 ||[https://jira.whamcloud.com/browse/LU-4315 LU-4315]
|-
|-
| [[Improve testing Efficiency]]
| [[Improve testing Efficiency]]
||Improve the performance, efficiency, and coverage of the acceptance-small.sh test scripts.  As a basic step, printing the duration of each test script in the acceptance-small.sh test summary would tell us where the testing time is being spent.
||Improve the performance, efficiency, and coverage of the acceptance-small.sh test scripts.  As a basic step, printing the duration of each test script in the acceptance-small.sh test summary would tell us where the testing time is being spent.
|| 3 ||[https://bugzilla.lustre.org/show_bug.cgi?id=23051 b=23051]
|| 3 ||[https://bugzilla.lustre.org/show_bug.cgi?id=23051 b=23051]
|-
| [[Config save/edit/restore]]
||Need to be able to backup/edit/restore the client/MDS/OSS config llog files after a writeconf.  One reason is for config recovery if the config llog becomes corrupted.  Another reason is that all of the filesystem tunable parameters (including all of the OST pool definitions) are stored in the config llog and are lost if a writeconf is done.  Being able to dump the config log to a YAML-formatted plain text file, edit it, and then restore it would make administration considerably easier.
|| 3 ||[https://bugzilla.lustre.org/show_bug.cgi?id=17094 b=17094]
|-
|-
| [[Error message improvements]]
| [[Error message improvements]]
||Review and improve the Lustre error messages to be more useful.  A larger project is to change the core Lustre error message handling to generate better structured error messages so that they can be parsed/managed more easily.
||Review and improve the Lustre error messages to be more useful and follow the Lustre coding style (see [[Lustre_Coding_Style_Guidelines#Console_Error_Messages]].  A larger project is to change the core Lustre error message handling to generate better structured error messages so that they can be parsed/managed more easily.
|| 4 ||
|| 4 ||
|-
|-
| [[Improve QOS Round-Robin object allocator]]
| [[Improve QOS Round-Robin object allocator]]
||Improve LOV QOS allocator to always do weighted round-robin allocation, instead of degrading into weighted random allocations once the OST free space becomes imbalanced.  This evens out allocations continuously, avoids crazy/bad OST allocation imbalances when QOS becomes active, and allows adding weighting for things like current load, OST RAID rebuild, etc.
||Improve LOV QOS allocator to always do weighted round-robin allocation, instead of degrading into weighted random allocations once the OST free space becomes imbalanced.  This evens out allocations continuously, avoids crazy/bad OST allocation imbalances when QOS becomes active, and allows adding weighting for things like current load, OST RAID rebuild, etc.
|| 5 ||[https://jira.hpdd.intel.com/browse/LU-9 LU-9]
|| 5 ||[https://jira.whamcloud.com/browse/LU-9 LU-9]
|-
| [[ZFS ZIL Support]]
||Add support for the ZFS Intent Log (ZIL) to the Lustre osd-zfs
|| 6 ||[https://jira.whamcloud.com/browse/LU-4009 LU-4009]
|-
| [[NFSv4 ACL Support]]
||Add handling of NFSv4 ACLs to Lustre
|| 6 ||[https://jira.whamcloud.com/browse/LU-13862 LU-13862]
|-
|-
| [[RPC Replay Signatures]]
| [[RPC Replay Signatures]]
Line 79: Line 78:
|-
|-
| [[Simplified Interoperability]]
| [[Simplified Interoperability]]
||Add functionality so servers can force clients to drop all locks, and flush unreplied RPCs.  This reduces recovery if a server is being cleanly shut down, and avoids RPC replay issues across Lustre server upgrades.  Open file recovery should be done by generating new open RPCs on the client, rather than replaying RPCs stored in the replay list, which also allows removing the special-case handling of open RPCs that are already committed.
||Add functionality so servers can force clients quiesce all opertaions, drop all locks, and flush unreplied RPCs.  This allows administrators to halt operations on clients without having to unmount the filesystem.  This prevents unintended operations on the server (e.g. during maintenance), reduces recovery operations if a server is being cleanly shut down, and avoids RPC replay issues across major Lustre server version upgrades.  Open file recovery should be done by generating new open RPCs on the client, rather than replaying RPCs stored in the replay list, which also allows removing the special-case handling of open RPCs that are already committed.
|| 6 ||[https://bugzilla.lustre.org/show_bug.cgi?id=18496 b=18496] [https://jira.hpdd.intel.com/browse/LU-5703 LU-5703]
|| 6 ||[https://bugzilla.lustre.org/show_bug.cgi?id=18496 b=18496] [https://jira.whamcloud.com/browse/LU-5703 LU-5703]
|-
| [[Policy Engine Proposal]] || New policy engine proposal for FLR, HSM, and others || 6 ||
|-
|-
| [[Virtual Lustre Block Device]]
| [[Virtual Lustre Block Device]]
||Lustre object lloop driver exports block device to userspace, bypassing filesystem. Code partly works and is currently part of Lustre, but has correctness issues and potential performance problems.  It needs to be ported to newer kernels.
||Lustre object lloop driver previously exported a virtual block device to userspace for a file, bypassing the filesystem. Code partly worked but was removed from Lustre due to correctness issues and potential performance problems.  It needs to be rewritten to work with newer kernels, by providing a new "backend" for the kernel loop driver and avoid data copies to avoid deadlocks under memory pressure.
|| 6 ||[https://jira.hpdd.intel.com/browse/LU-6585 LU-6585]
|| 6 ||[https://jira.whamcloud.com/browse/LU-6585 LU-6585]
|-
|-
| [[Swap on Lustre]]
| [[Swap on Lustre]]
||Depends on the Lustre block device. Has problems when working under memory pressure, which makes it mostly useless until those problems are fixed.
||Depends on the Lustre lloop block device. This needs to work properly under memory pressure, or it will be useless when swap is actually needed.
|| 7 ||[https://bugzilla.lustre.org/show_bug.cgi?id=5498 b=5498]
|| 7 ||[https://bugzilla.lustre.org/show_bug.cgi?id=5498 b=5498]
|-
|-
| [[Directory readdir+]]
| [[Directory bulk operations]]
||Bulk metadata readdir/stat interface to speed up "ls -l" operations.  Send back requested inode attributes for all directory entries as part of the extended dirent data.  Integrate with any proposed API for this on the client.  Needs Large Readdir RPCs to be efficient over the wire, since more data will be returned for every entry.
||Bulk metadata readdir/stat interface to speed up "ls -l" operations.  Send back requested inode attributes for all directory entries as part of the extended dirent data.  Integrate with any proposed API for this on the client.  Needs Large Readdir RPCs to be efficient over the wire, since more data will be returned for every entry.
|| 7 ||[https://bugzilla.lustre.org/show_bug.cgi?id=17845 b=17845]
|| 7 ||[https://jira.whamcloud.com/browse/LU-14139 LU-14139]
|-
|-
| [[Small file IO aggregation]]
| [[Small file IO aggregation]]
||Small file IO aggregation (multi-object RPCs), most likely for writes first, and possibly later for reads in conjunction with statahead.
||Small file IO aggregation (multi-object RPCs), most likely for writes first, and possibly later for reads in conjunction with statahead.
|| 7 ||[https://bugzilla.lustre.org/show_bug.cgi?id=944 b=944]
|| 7 ||[https://bugzilla.lustre.org/show_bug.cgi?id=944 b=944]
|-
| [[Lustre Snapshots|Integrated Lustre Snapshots]]
||Allow Lustre to internally mount and manage ZFS snapshots and other datasets within a single namespace
|| 7 ||
|-
| [[Client-side data encryption]]
||Encrypt files and directories (or possibly just filenames in directories) on the client before sending to the server.  This avoids sending unencrypted data over the network, or ever having the data in plaintext on the server (in case of separate decryption from network plus encryption on disk).  It seems possible to leverage the existing GSSAPI sptlrpc_cli_wrap_bulk() and friends to do bulk data encryption/decryption on the client using a per-file key (itself stored encrypted by the users' key(s) with the file on the MDS), and salted with the OST object ID or stripe index and object offset, rather than a per-session key and then not decrypting the data at the server before writing it to disk.
|| 7 ||[]
|-
|-
| [[Local object zero-copy IO]]
| [[Local object zero-copy IO]]
Line 116: Line 109:
{| class="wikitable sortable"
{| class="wikitable sortable"
|-
|-
! Feature !! Feature Summary !! Point of Contact !! Tracker !! Merge Date (YYYY-MM) !! Merge Release
! Feature !! Feature Summary !! Point of Contact !! Tracker !! GA Release
|-
| [[Imperative Recovery]] || Servers notify clients (via MGS) where a failed target has restarted to reduce waiting time during recovery. || Jinshan Xiong (Intel) || [https://jira.whamcloud.com/browse/LU-19 LU-19] ||2.2
|-
| [[LFSCK Phase 1 - OI Scrub Solution Architecture]] || Check and repair Object Index on ldiskfs filesystems via OI Scrub to allow file-level backup/restore of MDT filesystems. || Fan Yong (Intel) || [https://jira.whamcloud.com/browse/LU-957 LU-957] ||2.3
|-
| [[LFSCK_Phase_1.5_-_FID-in-Dirent_LinkEA_Consistency_Solution_Architecture|LFSCK Phase 1.5 - FID-in-Dirent and LinkEA Consistency]] || Enhance LFSCK to work with DNE filesystems, checking the Lustre FID stored in MDT directory entries and the parent LinkEA entries for hard links. || Fan Yong (Intel) || [https://jira.whamcloud.com/browse/LU-1866 LU-1866] ||2.4
|-
| [[Shared_Secret_Key_Authentication_And_Encryption|Shared Secret Key Encryption]] || Allow shared-secret key encryption and authentication for Lustre network traffic without the use of an external keyserver || Andrew Korty, Jeremy Filizetti (Indiana University) || [https://jira.whamcloud.com/browse/LU-3289 LU-3289] ||2.6
|-
| [[LFSCK_Phase_2_-_MDT-OST_Consistency_Solution_Architecture|LFSCK Phase 2 - MDT-OST consistency]] || Enhance LFSCK to check layout consistency between MDT file layouts and OST objects. || Fan Yong (Intel) || [https://jira.whamcloud.com/browse/LU-1267 LU-1267] ||2.6
|-
| [[LFSCK_Phase_3_-_MDT-MDT_Consistency_Solution_Architecture|LFSCK Phase 3 - DNE consistency]] || Enhance LFSCK to work with DNE filesystems, including remote directory entries, and OST orphan handling for multiple MDTs. || Fan Yong (Intel) || [https://jira.whamcloud.com/browse/LU-2307 LU-2307] ||2.6
|-
| [[UID/GID_Mapping|UID/GID Nodemap]] || Allow mapping UID/GID and server-side configuration on a per-client/per-cluster basis || Joshua Walgenbach, Kit Westneat (Indiana University) || [https://jira.whamcloud.com/browse/LU-4647 LU-4647] ||2.7
|-
| [[Dynamic LNET Configuration]] || Introduces a user space script which will read routes from a config file and add those routes to LNET dynamically with the lctl utility. This allows the support of very large routing tables || Amir Shehata (Intel) || [https://jira.whamcloud.com/browse/LU-2456 LU-2456] ||2.7
|-
| [[LFSCK_Phase_4_-_Performance_and_Cleanups_Solution_Architecture|LFSCK Phase 4 - Performance Tuning]] || Enhance LFSCK performance and efficiency. || Fan Yong (Intel) || [https://jira.whamcloud.com/browse/LU-6361 LU-6361] ||2.8
|-
| [[Multiple metadata RPCs]] || Support of multiple metadata modifications per client (in last_rcvd file) to improve the multi-threaded metadata performance of a single client || Grégoire Pichon (Bull/Atos) || [https://jira.whamcloud.com/browse/LU-5319 LU-5319] ||2.8
|-
| [[DNE Phase IIb]] || Asynchronous Commit of cross-MDT updates for improved performance. Remote rename and remote hard link functionality. || Wang Di (Intel) || [https://jira.whamcloud.com/browse/LU-3534 LU-3534] ||2.8
|-
| [[Kerberos Revival]] || Fix up existing Kerberos code so it is tested working again. || Sébastien Buisson (Bull/Atos) || [https://jira.whamcloud.com/browse/LU-6356 LU-6356] ||2.8
|-
| [[Filesystem default OST pool]] || Allow a default filesystem-wide default OST pool to be specified. || Lai Siyao (Intel) || [https://jira.whamcloud.com/browse/LU-7660 LU-7660],  [https://jira.whamcloud.com/browse/LU-7335 LU-7335] ||2.9
|-
| [http://wiki.opensfs.org/index.php?title=Contract_SFS-DEV-002#Summary UID/GID mapping] || Map UID/GID for remote client nodes to local UID/GID on the MDS and OSS. Allows a single Lustre filesystem to be shared across clients with administrative domains. || Stephen Simms (Indiana University) || [https://jira.whamcloud.com/browse/LU-3291 LU-3291] ||2.9
|-
| [[Subdirectory Mounts]] || Ability for client to mount subdirectories of a Lustre filesystem || Wang Shilong (DDN) || [https://jira.whamcloud.com/browse/LU-28 LU-28]||2.9
|-
| [[Server side advise and hinting]] || Add support new APIs and utilities for server/storage side advise of accessing file for server cache || Li Xi (DDN) ||[https://jira.whamcloud.com/browse/LU-4931 LU-4931] ||2.9
|-
| [[Large Bulk IO]] || Increase the OST bulk IO maximum size to 16MB or larger for more efficient disk IO submission. || Shuichi Ihara (DDN) || [https://jira.whamcloud.com/browse/LU-7990 LU-7990] ||2.9
|-
| [http://wiki.opensfs.org/index.php?title=Contract_SFS-DEV-002#Summary Shared Key Crypto] || Allow node authentication and/or RPC encryption using symmetric shared key crypto with GSSAPI. Avoids complexity in configuring Kerberos across multiple domains. || Stephen Simms (Indiana University) || [https://jira.whamcloud.com/browse/LU-3289 LU-3289] ||2.9
|-
|-
| [[LFSCK Phase 3 - DNE consistency]] || Enhance LFSCK to work with DNE filesystems, including remote directory entries, and OST orphan handling for multiple MDTs. || Fan Yong (Intel) || [https://jira.hpdd.intel.com/browse/LU-2307 LU-2307] ||2014-11||2.6
| [[Multi-Rail LNet]] || Use multiple LNet network interfaces concurrently to improve reliability and performance || Amir Shehata (Intel), Olaf Weber (SGI) || [https://jira.whamcloud.com/browse/LU-7734 LU-7734] ||2.10
|-
|-
| [[Dynamic LNET Configuration]] || Introduces a user space script which will read routes from a config file and add those routes to LNET dynamically with the lctl utility. This allows the support of very large routing tables || Amir Shehata (Intel) || [https://jira.hpdd.intel.com/browse/LU-2456 LU-2456] ||2014-11||2.7
| [[TBF policy enhancement]] || An enhancement of NRS/TBF policy to support complex TBF policy with NID/JOBID expressions || Li Xi (DDN) || [https://jira.whamcloud.com/browse/LU-5620 LU-5620] [https://jira.whamcloud.com/browse/LU-7470 LU-7470] [https://jira.whamcloud.com/browse/LU-8236 LU-8236] ||2.10
|-
|-
| [[LFSCK Phase 4 - performance tuning]] || Enhance LFSCK performance and efficiency. || Fan Yong (Intel) || [https://jira.hpdd.intel.com/browse/LU-6361 LU-6361] ||2015-05||2.8
| [[Lustre Snapshots|Simplified Userspace Snapshots]] || Allow snapshots of ZFS targets to be mounted as a coherent filesystem || Fan Yong (Intel) || [https://jira.whamcloud.com/browse/LU-8900 LU-8900] ||2.10
|-
|-
| [[Multiple metadata RPCs]] || Support of multiple metadata modifications per client (in last_rcvd file) to improve the multi-threaded metadata performance of a single client || Grégoire Pichon (Bull/Atos) || [https://jira.hpdd.intel.com/browse/LU-5319 LU-5319] ||2015-05||2.8
| [[Layout Enhancement High Level Design|Layout Enhancement]] || Add support for composite layouts on a single file, for File Level Replication, Data on MDT, PFL, etc. || Jinshan Xiong, Niu Yawei (Intel) || [https://jira.whamcloud.com/browse/LU-3480 LU-3480]||2.10
|-
|-
| [[DNE Phase IIb]] || Asynchronous Commit of cross-MDT updates for improved performance. Remote rename and remote hard link functionality. || Wang Di (Intel) || [https://jira.hpdd.intel.com/browse/LU-3534 LU-3534] ||2015-05||2.8
| [[Progressive File Layouts]] || Allow composite file layouts to be instantiated incrementally during file writes || Jinshan Xiong, Niu Yawei (Intel) || [https://jira.whamcloud.com/browse/LU-8998 LU-8998] ||2.10
|-
|-
| [[Kerberos Revival]] || Fix up existing Kerberos code so it is tested working again. || Sébastien Buisson (Bull/Atos) || [https://jira.hpdd.intel.com/browse/LU-6356 LU-6356] ||2015-06||2.8
| [[Project Quota]] || Allow specifying a "project" or "subtree" identifier for files for accounting to a project, separate from UID/GID. || Shuichi Ihara (DDN) || [https://jira.whamcloud.com/browse/LU-4017 LU-4017] ||2.10
|-
|-
| [[Filesystem default OST pool]] || Allow a default filesystem-wide default OST pool to be specified. || Lai Siyao (Intel) || [https://jira.hpdd.intel.com/browse/LU-7660 LU-7660],  [https://jira.hpdd.intel.com/browse/LU-7335 LU-7335] ||2016-07||2.9
| [[NRS Delay policy]] || Use NRS for fault injection. Intentionally delay request processing to simulate server load. || Chris Horn (Cray) || [https://jira.whamcloud.com/browse/LU-6283 LU-6283] ||2.10
|-
|-
| [http://wiki.opensfs.org/index.php?title=Contract_SFS-DEV-002#Summary UID/GID mapping] || Map UID/GID for remote client nodes to local UID/GID on the MDS and OSS. Allows a single Lustre filesystem to be shared across clients with administrative domains. || Stephen Simms (Indiana University) || [https://jira.hpdd.intel.com/browse/LU-3291 LU-3291] ||2016-06||2.9
| Patchless Server || Remove Lustre kernel patches to allow Lustre servers to be more easily ported to new kernels, and to be built against vendor kernels without changing the vendor kernel RPMs. || Oleg Drokin (Intel) || [https://jira.whamcloud.com/browse/LU-20 LU-20] ||2.10
|-
|-
| [[Subdirectory Mounts]] || Ability for client to mount subdirectories of a Lustre filesystem || Wang Shilong (DDN) || [https://jira.hpdd.intel.com/browse/LU-28 LU-28]||2016-06||2.9
| [[Asynchronous Lock ahead]] || Allow user space to request LDLM extent locks in advance of need. Intended to optimize shared file IO. || Patrick Farrell (Cray) || [https://jira.whamcloud.com/browse/LU-6179 LU-6179] || 2.11
|-
|-
| [[Server side advise and hinting]] || Add support new APIs and utilities for server/storage side advise of accessing file for server cache || Li Xi (DDN) ||[https://jira.hpdd.intel.com/browse/LU-4931 LU-4931] ||2016-09||2.9
| [[Data on MDT]] || Allow small files to be stored directly on the MDT for reduced RPC count and improved performance. || Mikhail Pershin (Intel) || [https://jira.whamcloud.com/browse/LU-3285 LU-3285] || 2.11
|-
|-
| [[Large Bulk IO]] || Increase the OST bulk IO maximum size to 16MB or larger for more efficient disk IO submission. || Shuichi Ihara (DDN) || [https://jira.hpdd.intel.com/browse/LU-7990 LU-7990] ||2016-09||2.9
| [[File_Level_Redundancy_Solution_Architecture#Phase_1:_Delayed_Mirroring|File Level Redundancy (FLR) - Phase 1 Delayed Mirroring]] || Files can be mirrored on multiple OSTs in the same namespace, with delayed resync of files using userspace tools. || Jinshan Xiong (Intel) || [https://jira.whamcloud.com/browse/LU-3254 LU-3254] ||2.11
|-
|-
| [http://wiki.opensfs.org/index.php?title=Contract_SFS-DEV-002#Summary Shared Key Crypto] || Allow node authentication and/or RPC encryption using symmetric shared key crypto with GSSAPI. Avoids complexity in configuring Kerberos across multiple domains. || Stephen Simms (Indiana University) || [https://jira.hpdd.intel.com/browse/LU-3289 LU-3289] ||2016-12||2.9
| [[Lustre Audit with Changelogs]] || Introduce Lustre Audit feature to be able to track all accesses to files residing on Lustre || Sebastien Buisson (DDN) || [https://jira.whamcloud.com/browse/LU-9727 LU-9727] || 2.11
|-
|-
| [[Multi-Rail LNet]] || Use multiple LNet network interfaces concurrently to improve reliability and performance || Amir Shehata (Intel), Olaf Weber (SGI) || [https://jira.hpdd.intel.com/browse/LU-7734 LU-7734] ||2017-02||2.10
| [[Lazy size on MDT]] || Lazy size on MDT (LSOM) provides ability of "Lazy" file size from MDS rather than size from OSS today. || Li Xi (DDN) || [https://jira.whamcloud.com/browse/LU-9538 LU-9538] || 2.12
|-
|-
| [[TBF policy enhancement]] || An enhancement of NRS/TBF policy to support complex TBF policy with NID/JOBID expressions || Li Xi (DDN) || [https://jira.hpdd.intel.com/browse/LU-5620 LU-5620] [https://jira.hpdd.intel.com/browse/LU-7470 LU-7470] [https://jira.hpdd.intel.com/browse/LU-8236 LU-8236] ||2017-03||2.10
| [[Config save/edit/restore]] ||Ability to backup/edit/restore the client/MDS/OSS config llog files after a writeconf for config recovery if the config llog becomes corrupted, and to avoid losing filesystem tunable parameters (including all of the OST pool definitions) stored in the config llog if a writeconf is done. || Ben Evans (Cray) || [https://jira.whamcloud.com/browse/LU-4939 LU-4939] || 2.12
|-
|-
| [[Lustre Snapshots|Simplified Userspace Snapshots]] || Allow snapshots of ZFS targets to be mounted as a coherent filesystem || Fan Yong (Intel) || [https://jira.hpdd.intel.com/browse/LU-8900 LU-8900] ||2017-03||2.10
| [[Persistent Client Cache]] || A framework to cache files in storage local to the Lustre client. || Li Xi, Yingjin Qian (Whamcloud) || [https://jira.whamcloud.com/browse/LU-10092 LU-10092] || 2.13
|-
|-
| [[Layout Enhancement High Level Design|Layout Enhancement]] || Add support for composite layouts on a single file, for File Level Replication, Data on MDT, PFL, etc. || Jinshan Xiong, Niu Yawei (Intel) || [https://jira.hpdd.intel.com/browse/LU-3480 LU-3480]||2017-03||2.10
| [[Self Extending Layouts]] || Avoid ENOSPC by leveraging extension spaces in layout  || Vitaly Fertman (Cray) || [https://jira.whamcloud.com/browse/LU-10070 LU-10070] || 2.13
|-
|-
| [[Progressive File Layouts]] || Allow composite file layouts to be instantiated incrementally during file writes || Jinshan Xiong, Niu Yawei (Intel) || [https://jira.hpdd.intel.com/browse/LU-8998 LU-8998] ||2017-03||2.10
| [[Foreign Layouts]] || Allow linking files/directories to arbitrary identifiers (e.g. DAOS datasets)  || Bruno Faccini (Intel) || [https://jira.whamcloud.com/browse/LU-11376 LU-11376] || 2.13
|-
|-
| [[Project Quota]] || Allow specifying a "project" or "subtree" identifier for files for accounting to a project, separate from UID/GID. || Shuichi Ihara (DDN) || [https://jira.hpdd.intel.com/browse/LU-4017 LU-4017] ||2017-03||2.10
| [[Data on MDT Phase 2]] || DNE+DoM optimization, migration, read-on-open, stat() and readdir() returning file data, glimpse-ahead, file create improvements. || Mike Pershin (Whamcloud) || [https://jira.whamcloud.com/browse/LU-10176 LU-10176]||2.13
|-
|-
| [[NRS Delay policy]] || Use NRS for fault injection. Intentionally delay request processing to simulate server load. || Chris Horn (Cray) || [https://jira.hpdd.intel.com/browse/LU-6283 LU-6283] || 2017-04||2.10
| [[File heat]] || Add support new attribute "file heat" of objects to track "hot" files, for client and OFD || Li Xi (Whamcloud) || [https://jira.whamcloud.com/browse/LU-10602 LU-10602] || 2.13
|-
|-
| Patchless Server || Remove Lustre kernel patches to allow Lustre servers to be more easily ported to new kernels, and to be built against vendor kernels without changing the vendor kernel RPMs. || Oleg Drokin (Intel) || [https://jira.hpdd.intel.com/browse/LU-20 LU-20] || 2017-03 || 2.10
| [[OST Pool Quotas]] || Limit OST pool usage via quotas || Sergey Chermencev (Cray) || [https://jira.whamcloud.com/browse/LU-11023 LU-11023] || 2.14
|-
|-
| [[Asynchronous Lock ahead]] || Allow user space to request LDLM extent locks in advance of need. Intended to optimize shared file IO. || Patrick Farrell (Cray) || [https://jira.hpdd.intel.com/browse/LU-6179 LU-6179] || 2015-07 || 2.11
| [[Client-side data encryption]] || Encrypt files and directories (or possibly just filenames in directories) on the client before sending to the server.  This avoids sending unencrypted data over the network, or ever having the data in plaintext on the server (in case of separate decryption from network plus encryption on disk). || Sebastien Buisson (Whamcloud) || [https://jira.whamcloud.com/browse/LU-12275 LU-12275] || 2.14
|-
|-
| [[Data on MDT]] || Allow small files to be stored directly on the MDT for reduced RPC count and improved performance. || Mikhail Pershin (Intel) || [https://jira.hpdd.intel.com/browse/LU-3285 LU-3285] || 2017-09 || 2.11
|}
|}


[[Category: Development]]
[[Category: Development]]

Revision as of 13:20, 30 March 2021

Trivial Projects

Simple "get your feet wet" coding projects are tracked in JIRA using the easy keyword. These could be attempted by anyone with a knowledge of C programming, and some interest in learning Lustre or Linux kernel programming. Slightly more complex projects can be found in JIRA using the medium keyword and will typically require at least some understanding of the Lustre code.

Current Projects

Feature Feature Summary Point of Contact Tracker Target Release
File Level Redundancy (FLR) - Phase 3 Erasure Coding Provides a more space-efficient method for adding data redundancy than mirroring. Bobijam Xu (Whamcloud) LU-10911 2.15
External HSM Coordinator Add capability to run HSM coordinator in user space, which could allow 3rd-party control over HSM action queue Ben Evans (Whamcloud) LU-10968 2.15
HSM control of data migration Allow HSM Coordinator to control data migration (`lfs migrate`), which could allow copy tools to move data in parallel Ben Evans (Whamcloud) LU-6081 2.15
Enhanced Adaptive Compression in Lustre Introduce compression for the Lustre client and server Anna Fuchs Michael Kuhn (Universität Hamburg) LU-10026 2.15
Metadata Writeback Cache Overcome link latency for modifying metadata operations esp. noticeable in interactive type of workloads Oleg Drokin, Qian Yinjin (Whamcloud) LU-10938 2.16

Future Projects

Feature Feature Summary Point of Contact Tracker
RTDS(Real-Time Dynamic Striping) A policy based Lustre striping framework Li Xi (Whamcloud) LU-9809
OSP multiple modify requests to MDT Improve performance of cross-MDT modify operations. Grégoire Pichon (Atos) LU-6864
Directory Level Snapshot Directory level snapshot (DL-SNAP) is designed for user level file backups. DL-SNAP will be implemented by using copy-on-write mechanism on top of ldiskfs without modification of disk format. Kenichiro Sakai (Fujitsu) LU-11512
File Level Redundancy (FLR) - Phase 2 Immediate Write Mirroring Immediate mirrored writes from the client to multiple OSTs. Bobijam Xu (Whamcloud)

Potential Projects

These projects are potential areas of development that are looking for an interested party to work on or sponsor another developer to do. Many of them have more detailed descriptions, but it is worthwhile to contact the lustre-devel mailing list to discuss the project before starting implementation.

Feature Feature Summary Complexity Tracker
ioctl() number cleanups Clean up Linux IOC numbering to properly use "size" field so that mixed 32- and 64-bit kernel/userspace ioctls work correctly. Attention needs to be paid to maintaining userspace compatibility for a number of releases, so the old ioctl() numbers cannot simply be removed. 1 b=20731
Updated man pages Update the online manual pages for Lustre user tools and the lustreapi library. Split the existing lfs.1 and lctl.8 man pages into separate pages for each sub-command, describing options and providing usage examples. 2 LU-4315
Improve testing Efficiency Improve the performance, efficiency, and coverage of the acceptance-small.sh test scripts. As a basic step, printing the duration of each test script in the acceptance-small.sh test summary would tell us where the testing time is being spent. 3 b=23051
Error message improvements Review and improve the Lustre error messages to be more useful and follow the Lustre coding style (see Lustre_Coding_Style_Guidelines#Console_Error_Messages. A larger project is to change the core Lustre error message handling to generate better structured error messages so that they can be parsed/managed more easily. 4
Improve QOS Round-Robin object allocator Improve LOV QOS allocator to always do weighted round-robin allocation, instead of degrading into weighted random allocations once the OST free space becomes imbalanced. This evens out allocations continuously, avoids crazy/bad OST allocation imbalances when QOS becomes active, and allows adding weighting for things like current load, OST RAID rebuild, etc. 5 LU-9
ZFS ZIL Support Add support for the ZFS Intent Log (ZIL) to the Lustre osd-zfs 6 LU-4009
NFSv4 ACL Support Add handling of NFSv4 ACLs to Lustre 6 LU-13862
RPC Replay Signatures Allow MDS/OSS to determine if client can legitimately replay an RPC, by digitally signing it at processing time and verifying the signature at replay time. 6 b=18547
Simplified Interoperability Add functionality so servers can force clients quiesce all opertaions, drop all locks, and flush unreplied RPCs. This allows administrators to halt operations on clients without having to unmount the filesystem. This prevents unintended operations on the server (e.g. during maintenance), reduces recovery operations if a server is being cleanly shut down, and avoids RPC replay issues across major Lustre server version upgrades. Open file recovery should be done by generating new open RPCs on the client, rather than replaying RPCs stored in the replay list, which also allows removing the special-case handling of open RPCs that are already committed. 6 b=18496 LU-5703
Policy Engine Proposal New policy engine proposal for FLR, HSM, and others 6
Virtual Lustre Block Device Lustre object lloop driver previously exported a virtual block device to userspace for a file, bypassing the filesystem. Code partly worked but was removed from Lustre due to correctness issues and potential performance problems. It needs to be rewritten to work with newer kernels, by providing a new "backend" for the kernel loop driver and avoid data copies to avoid deadlocks under memory pressure. 6 LU-6585
Swap on Lustre Depends on the Lustre lloop block device. This needs to work properly under memory pressure, or it will be useless when swap is actually needed. 7 b=5498
Directory bulk operations Bulk metadata readdir/stat interface to speed up "ls -l" operations. Send back requested inode attributes for all directory entries as part of the extended dirent data. Integrate with any proposed API for this on the client. Needs Large Readdir RPCs to be efficient over the wire, since more data will be returned for every entry. 7 LU-14139
Small file IO aggregation Small file IO aggregation (multi-object RPCs), most likely for writes first, and possibly later for reads in conjunction with statahead. 7 b=944
Local object zero-copy IO Efficient data IO between a client and a local OST object; optimization to support local clients. Likely implemented as a fast-path connection between the OSC and the local OFD/OSD. Read cache should be kept on the OSD instead of at the client VFS level, so that the cache can be shared among all users of this OST. 9

Past Projects

Feature Feature Summary Point of Contact Tracker GA Release
Imperative Recovery Servers notify clients (via MGS) where a failed target has restarted to reduce waiting time during recovery. Jinshan Xiong (Intel) LU-19 2.2
LFSCK Phase 1 - OI Scrub Solution Architecture Check and repair Object Index on ldiskfs filesystems via OI Scrub to allow file-level backup/restore of MDT filesystems. Fan Yong (Intel) LU-957 2.3
LFSCK Phase 1.5 - FID-in-Dirent and LinkEA Consistency Enhance LFSCK to work with DNE filesystems, checking the Lustre FID stored in MDT directory entries and the parent LinkEA entries for hard links. Fan Yong (Intel) LU-1866 2.4
Shared Secret Key Encryption Allow shared-secret key encryption and authentication for Lustre network traffic without the use of an external keyserver Andrew Korty, Jeremy Filizetti (Indiana University) LU-3289 2.6
LFSCK Phase 2 - MDT-OST consistency Enhance LFSCK to check layout consistency between MDT file layouts and OST objects. Fan Yong (Intel) LU-1267 2.6
LFSCK Phase 3 - DNE consistency Enhance LFSCK to work with DNE filesystems, including remote directory entries, and OST orphan handling for multiple MDTs. Fan Yong (Intel) LU-2307 2.6
UID/GID Nodemap Allow mapping UID/GID and server-side configuration on a per-client/per-cluster basis Joshua Walgenbach, Kit Westneat (Indiana University) LU-4647 2.7
Dynamic LNET Configuration Introduces a user space script which will read routes from a config file and add those routes to LNET dynamically with the lctl utility. This allows the support of very large routing tables Amir Shehata (Intel) LU-2456 2.7
LFSCK Phase 4 - Performance Tuning Enhance LFSCK performance and efficiency. Fan Yong (Intel) LU-6361 2.8
Multiple metadata RPCs Support of multiple metadata modifications per client (in last_rcvd file) to improve the multi-threaded metadata performance of a single client Grégoire Pichon (Bull/Atos) LU-5319 2.8
DNE Phase IIb Asynchronous Commit of cross-MDT updates for improved performance. Remote rename and remote hard link functionality. Wang Di (Intel) LU-3534 2.8
Kerberos Revival Fix up existing Kerberos code so it is tested working again. Sébastien Buisson (Bull/Atos) LU-6356 2.8
Filesystem default OST pool Allow a default filesystem-wide default OST pool to be specified. Lai Siyao (Intel) LU-7660, LU-7335 2.9
UID/GID mapping Map UID/GID for remote client nodes to local UID/GID on the MDS and OSS. Allows a single Lustre filesystem to be shared across clients with administrative domains. Stephen Simms (Indiana University) LU-3291 2.9
Subdirectory Mounts Ability for client to mount subdirectories of a Lustre filesystem Wang Shilong (DDN) LU-28 2.9
Server side advise and hinting Add support new APIs and utilities for server/storage side advise of accessing file for server cache Li Xi (DDN) LU-4931 2.9
Large Bulk IO Increase the OST bulk IO maximum size to 16MB or larger for more efficient disk IO submission. Shuichi Ihara (DDN) LU-7990 2.9
Shared Key Crypto Allow node authentication and/or RPC encryption using symmetric shared key crypto with GSSAPI. Avoids complexity in configuring Kerberos across multiple domains. Stephen Simms (Indiana University) LU-3289 2.9
Multi-Rail LNet Use multiple LNet network interfaces concurrently to improve reliability and performance Amir Shehata (Intel), Olaf Weber (SGI) LU-7734 2.10
TBF policy enhancement An enhancement of NRS/TBF policy to support complex TBF policy with NID/JOBID expressions Li Xi (DDN) LU-5620 LU-7470 LU-8236 2.10
Simplified Userspace Snapshots Allow snapshots of ZFS targets to be mounted as a coherent filesystem Fan Yong (Intel) LU-8900 2.10
Layout Enhancement Add support for composite layouts on a single file, for File Level Replication, Data on MDT, PFL, etc. Jinshan Xiong, Niu Yawei (Intel) LU-3480 2.10
Progressive File Layouts Allow composite file layouts to be instantiated incrementally during file writes Jinshan Xiong, Niu Yawei (Intel) LU-8998 2.10
Project Quota Allow specifying a "project" or "subtree" identifier for files for accounting to a project, separate from UID/GID. Shuichi Ihara (DDN) LU-4017 2.10
NRS Delay policy Use NRS for fault injection. Intentionally delay request processing to simulate server load. Chris Horn (Cray) LU-6283 2.10
Patchless Server Remove Lustre kernel patches to allow Lustre servers to be more easily ported to new kernels, and to be built against vendor kernels without changing the vendor kernel RPMs. Oleg Drokin (Intel) LU-20 2.10
Asynchronous Lock ahead Allow user space to request LDLM extent locks in advance of need. Intended to optimize shared file IO. Patrick Farrell (Cray) LU-6179 2.11
Data on MDT Allow small files to be stored directly on the MDT for reduced RPC count and improved performance. Mikhail Pershin (Intel) LU-3285 2.11
File Level Redundancy (FLR) - Phase 1 Delayed Mirroring Files can be mirrored on multiple OSTs in the same namespace, with delayed resync of files using userspace tools. Jinshan Xiong (Intel) LU-3254 2.11
Lustre Audit with Changelogs Introduce Lustre Audit feature to be able to track all accesses to files residing on Lustre Sebastien Buisson (DDN) LU-9727 2.11
Lazy size on MDT Lazy size on MDT (LSOM) provides ability of "Lazy" file size from MDS rather than size from OSS today. Li Xi (DDN) LU-9538 2.12
Config save/edit/restore Ability to backup/edit/restore the client/MDS/OSS config llog files after a writeconf for config recovery if the config llog becomes corrupted, and to avoid losing filesystem tunable parameters (including all of the OST pool definitions) stored in the config llog if a writeconf is done. Ben Evans (Cray) LU-4939 2.12
Persistent Client Cache A framework to cache files in storage local to the Lustre client. Li Xi, Yingjin Qian (Whamcloud) LU-10092 2.13
Self Extending Layouts Avoid ENOSPC by leveraging extension spaces in layout Vitaly Fertman (Cray) LU-10070 2.13
Foreign Layouts Allow linking files/directories to arbitrary identifiers (e.g. DAOS datasets) Bruno Faccini (Intel) LU-11376 2.13
Data on MDT Phase 2 DNE+DoM optimization, migration, read-on-open, stat() and readdir() returning file data, glimpse-ahead, file create improvements. Mike Pershin (Whamcloud) LU-10176 2.13
File heat Add support new attribute "file heat" of objects to track "hot" files, for client and OFD Li Xi (Whamcloud) LU-10602 2.13
OST Pool Quotas Limit OST pool usage via quotas Sergey Chermencev (Cray) LU-11023 2.14
Client-side data encryption Encrypt files and directories (or possibly just filenames in directories) on the client before sending to the server. This avoids sending unencrypted data over the network, or ever having the data in plaintext on the server (in case of separate decryption from network plus encryption on disk). Sebastien Buisson (Whamcloud) LU-12275 2.14