Projects: Difference between revisions
Jump to navigation
Jump to search
m (→Potential Projects: fix URL display) |
m (Corrected the spelling of my last name.) |
||
Line 21: | Line 21: | ||
| [[Progressive File Layouts]] || Allow composite file layouts to be instantiated incrementally during file writes || Jinshan Xiong, Niu Yawei (Intel) || || 2016-12 | | [[Progressive File Layouts]] || Allow composite file layouts to be instantiated incrementally during file writes || Jinshan Xiong, Niu Yawei (Intel) || || 2016-12 | ||
|- | |- | ||
| [[Multi-Rail LNet]] || Use multiple LNet network interfaces concurrently to improve reliability and performance || Amir Shehata (Intel), Olaf | | [[Multi-Rail LNet]] || Use multiple LNet network interfaces concurrently to improve reliability and performance || Amir Shehata (Intel), Olaf Weber (SGI) || || 2016-11 | ||
|} | |} | ||
Revision as of 02:58, 6 November 2015
Current Projects
Feature | Feature Summary | Point of Contact | Tracker | Target Date (YYYY-MM) |
---|---|---|---|---|
UID/GID mapping | Map UID/GID for remote client nodes to local UID/GID on the MDS and OSS. Allows a single Lustre filesystem to be shared across clients with administrative domains. | Stephen Simms (Indiana University) | LU-3291 | 2015-11 |
Shared Key Crypto | Allow node authentication and/or RPC encryption using symmetric shared key crypto with GSSAPI. Avoids complexity in configuring Kerberos across multiple domains. | Stephen Simms (Indiana University) | LU-3289 | 2015-12 |
NRS Delay policy | Use NRS for fault injection. Intentionally delay request processing to simulate server load. | Chris Horn (Cray) | LU-6283 | 2015-05 |
Lock ahead | Allow user space to request LDLM extent locks in advance of need. Intended to optimize shared file IO. | Patrick Farrell (Cray) | LU-6179 | 2015-07 |
ZFS ZIL Support | Add support for the ZFS Intent Log (ZIL) to the Lustre osd-zfs | Alex Zhuravlev (Intel) | LU-4009 | 2016-01 |
Composite File Layouts | Add support for multiple layouts on a single file, for File Level Replication, Data on MDT, PFL, etc. | Jinshan Xiong, Niu Yawei (Intel) | LU-3480 | 2016-11 |
Progressive File Layouts | Allow composite file layouts to be instantiated incrementally during file writes | Jinshan Xiong, Niu Yawei (Intel) | 2016-12 | |
Multi-Rail LNet | Use multiple LNet network interfaces concurrently to improve reliability and performance | Amir Shehata (Intel), Olaf Weber (SGI) | 2016-11 |
Future Projects
Feature | Feature Summary | Point of Contact | Tracker |
---|---|---|---|
Patchless Server | Remove Lustre kernel patches to allow Lustre servers to be more easily ported to new kernels, and to be built against vendor kernels without changing the vendor kernel RPMs. | Andreas Dilger (Intel) | LU-20 |
T10 End-to-End Data Integrity | Add a path to pass T10 Protection Information from Lustre clients through to server disks to reduce CPU overhead of multiple checksum calculations. | ? (Seagate) | LU-2584 |
File Level Replication Phase 1 | Solution Architecture and HDL, Exclusive Open, RAID 1 Layout, Layout Modification Method, Read Only RAID 1 | Jinshan Xiong (Intel) | LU-3254 |
File Level Replication Phase 2 | Immediate asynchronous write from client. | Jinshan Xiong (Intel) | LU-3254 |
Data on MDT | Allow small files to be stored directly on the MDT for reduced RPC count and improved performance. | Mikhail Pershin (Intel) | . |
Quota for Projects | Allow specifying a "project" or "subtree" identifier for files for accounting to a project, separate from UID/GID. | Shuichi Ihara (DDN) | LU-4017 |
Large Bulk IO | Increase the OST bulk IO maximum size to 16MB or larger for more efficient disk IO submission. | Shuichi Ihara (DDN) | . |
Potential Projects
These projects are potential areas of development that are looking for an interested party to work on or sponsor another developer to do. Many of them have more detailed descriptions, but it is worthwhile to contact the lustre-devel mailing list
Feature | Feature Summary | Complexity | Tracker |
---|---|---|---|
ioctl() number cleanups | Clean up Linux IOC numbering to properly use "size" field so that mixed 32- and 64-bit kernel/userspace ioctls work correctly. Attention needs to be paid to maintaining userspace compatibility for a number of releases, so the old ioctl() numbers cannot simply be removed. | 1 | b=20731 |
Updated man pages | Update the online manual pages for Lustre user tools and the lustreapi library. Split the existing lfs.1 and lctl.8 man pages into separate pages for each sub-command, describing options and providing usage examples. | 2 | LU-4315 |
Improve testing Efficiency | Improve the performance, efficiency, and coverage of the acceptance-small.sh test scripts. As a basic step, printing the duration of each test script in the acceptance-small.sh test summary would tell us where the testing time is being spent. | 3 | b=23051 |
Config save/edit/restore | Need to be able to backup/edit/restore the client/MDS/OSS config llog files after a writeconf. One reason is for config recovery if the config llog becomes corrupted. Another reason is that all of the filesystem tunable parameters (including all of the OST pool definitions) are stored in the config llog and are lost if a writeconf is done. Being able to dump the config log to a plain text file, edit it, and then restore it would make administration considerably easier. | 3 | b=17094 |
Filesystem default OST pool | Allow a filesystem-wide default OST pool to be specified. Currently, it is possible to set the default stripe count, size, index on a filesystem with "lfs setstripe" on the filesystem root, but the OST pool name is ignored. There is no other mechanism to specify default the OST pool for all new files in the filesystem, if no pool is specified. This would be useful for WAN or other heterogeneous OST configurations. | 3 | LU-7335 |
Error message improvements | Review and improve the Lustre error messages to be more useful. A larger project is to change the core Lustre error message handling to generate better structured error messages so that they can be parsed/managed more easily. | 4 | |
Improve QOS Round-Robin object allocator | Improve LOV QOS allocator to always do weighted round-robin allocation, instead of degrading into weighted random allocations once the OST free space becomes imbalanced. This evens out allocations continuously, avoids crazy/bad OST allocation imbalances when QOS becomes active, and allows adding weighting for things like current load, OST RAID rebuild, etc. | 5 | LU-9 |
RPC Replay Signatures | Allow MDS/OSS to determine if client can legitimately replay an RPC, by digitally signing it at processing time and verifying the signature at replay time. | 6 | b=18547 |
Virtual Lustre Block Device | Lustre object lloop driver exports block device to userspace, bypassing filesystem. Code partly works and is currently part of Lustre, but has correctness issues and potential performance problems. It needs to be ported to newer kernels. | 6 | LU-6585 |
Swap on Lustre | Depends on the Lustre block device. Has problems when working under memory pressure, which makes it mostly useless until those problems are fixed. | 7 | b=5498 |
Directory readdir+ | Bulk metadata readdir/stat interface to speed up "ls -l" operations. Send back requested inode attributes for all directory entries as part of the extended dirent data. Integrate with any proposed API for this on the client. Needs Large Readdir RPCs to be efficient over the wire, since more data will be returned for every entry. | 7 | b=17845 |
Small file IO aggregation | Small file IO aggregation (multi-object RPCs), most likely for writes first, and possibly later for reads in conjunction with statahead. | 7 | b=944 |
Client-side data encryption | Encrypt files and directories (or possibly just filenames) on the client before sending to the server. This avoids sending unencrypted data over the network, or ever having the data in plaintext on the server (in case of separate decryption from network, encryption on disk). | 9 | b=5286 |
Local object zero-copy IO | Efficient data IO between a client and a local OST object; optimization to support local clients. Likely implemented as a fast-path connection between the OSC and the local OFD/OSD. Read cache should be kept on the OSD instead of at the client VFS level, so that the cache can be shared among all users of this OST. | 9 |
Past Projects
Feature | Feature Summary | Point of Contact | Tracker | Target Date (YYYY-MM) | Merge Date (YYYY-MM) |
---|---|---|---|---|---|
Dynamic LNET Config | Introduces a user space script which will read routes from a config file and add those routes to LNET dynamically with the lctl utility. This allows the support of very large routing tables | Amir Shehata (Intel) | LU-2456 | 2014-11 | 2014-11 |
LFSCK Phase 3 - DNE consistency | Enhance LFSCK to work with DNE filesystems, including remote directory entries, and OST orphan handling for multiple MDTs. | Fan Yong (Intel) | LU-2307 | 2014-11 | 2014-11 |
LFSCK Phase 4 - performance tuning | Enhance LFSCK performance and efficiency. | Fan Yong (Intel) | LU-6361 | 2015-05 | 2015-05 |
Multiple metadata RPCs | Support of multiple metadata modifications per client (in last_rcvd file) to improve the multi-threaded metadata performance of a single client | Grégoire Pichon (Bull/Atos) | LU-5319 | 2015-05 | 2015-05 |
DNE Phase IIb | Asynchronous Commit of cross-MDT updates for improved performance. Remote rename and remote hard link functionality. | Wang Di (Intel) | LU-3534 | 2015-05 | 2015-05 |
Kerberos Revival | Fix up existing Kerberos code so it is tested working again. | Sébastien Buisson (Bull/Atos) | LU-6356 | 2015-06 | 2015-06 |