Lustre 2.13.0 Changelog

Server support for kernels: 


 * RHEL 7.7 (3.10.0-1062.1.1.el7)

Client support for unpatched kernels: 


 * RHEL 7.7 (3.10.0-1062.1.1.el7)
 * SLES12 SP4 (4.12.14-95.29.1)
 * Ubuntu 18.04 (4.15.0-64)

Recommended e2fsprogs version: 


 * v1.45.2.wc1

Interoperability Support: 


 * Clients & Servers: Latest 2.12.X

Issues fixed between 2.12.0 and 2.13.0
 New Feature 
 * LU-1365: Implement ldiskfs LARGEDIR support for e2fsprogs
 * LU-9846: Overstriping - more than 1 stripe per OST per component
 * LU-10070: PFL self-extending file layout
 * LU-10092: PCC: Lustre Persistent Client Cache
 * LU-11297: Align LNet routing with Multi-Rail and LNet health
 * LU-11376: Special file/dir to represent DAOS Containers
 * LU-12624: striped directory allocate stripes by QoS
 * LU-12734: bash completion (for now: lctl set/get_param)

 Improvement 
 * LU-1538: cleanup test scripts
 * LU-2233: sanityn/40-46 takes ~800-900s
 * LU-7004: fix &quot;lctl set_param -P&quot; to allow deprecation of &quot;lctl conf_param&quot;
 * LU-8207: Add auto-stripe option to lfs_migrate
 * LU-8434: Rewrite test framework using Python
 * LU-8939: nodemap: remove deprecated lproc files
 * LU-8955: Send SELinux policy info to server
 * LU-9010: use static initialization for statically allocated objects when possible
 * LU-9625: remove &quot;lfs ls&quot; and &quot;lfs cp&quot; and references
 * LU-9629: lfs migrate does not work as a non-root user
 * LU-10258: lfs mirror command to open fd on component
 * LU-10885: enable flock by default
 * LU-11010: Remove call to ‘return’ after call to ‘skip’ in Lustre test suites
 * LU-11011: checksum type can not be selected permanently
 * LU-11022: FLR1.5: &quot;lfs mirror&quot; usability for Burst Buffer
 * LU-11041: enable build and test with kernel debug options
 * LU-11078: customise run_ior
 * LU-11089: Performance improvements for lu_object locking
 * LU-11213: DNE3: remote mkdir in ROOT/ by default
 * LU-11221: Do not hold pages locked for network IO on the server.
 * LU-11233: gcc8 support
 * LU-11285: don't stop on the first blocked lock in ldlm_reprocess_queue
 * LU-11296: page cache cleanup improvement on lock flush
 * LU-11329: add get_maintainer.pl script from kernel
 * LU-11355: Add Lustre trim support
 * LU-11367: integrate LSOM with lfs find
 * LU-11398: clean up ptlrpc completion callbacks
 * LU-11408: propagate grant shrink interval immediately
 * LU-11415: ksocklnd performance improvement on 40Gbps ethernet
 * LU-11456: MDT can skip sync if the object is synced yet
 * LU-11462: osd-zfs can skip object sync if it's not dirty
 * LU-11467: FLR1.5: add &quot;lfs mirror delete&quot; command
 * LU-11504: improve ha.sh to trigger lfsck after/during failover/failback
 * LU-11526: Support 64MB I/O RPC
 * LU-11555: warn when mounting ZFS pools without &quot;multihost&quot; enabled
 * LU-11606: use llapi_error for errors in liblustreapi files
 * LU-11616: Optimize handling statahead delay
 * LU-11673: Correct shell issues/mistakes in Lustre test suites
 * LU-11717: t-f:exit_status defect: test returns code 1 if log contains &quot;FAIL&quot;
 * LU-11736: Do not apply bulk IO tuning on MDT or MGT devices
 * LU-11739: Don't inherit default layout from root for new subdirectories
 * LU-11810: allow &quot;Fixes:&quot; tag in commit message signoff block
 * LU-11811: Provide an interface to set Lustre MDT into readonly mode
 * LU-11820: reduce conf-sanity test duration
 * LU-11830: Quiet compiler warnings on ARM
 * LU-11834: config instance truncation due to buffer size mismatch
 * LU-11838: Support linux kernel version 4.18
 * LU-11843: customise sanity-quota to work with project quotas set before the suite
 * LU-11845: Support encrypted ZFS volumes
 * LU-11852: osc cache waiters wakeups and skipping queue
 * LU-11876: Lustre dkms packages missing build dependency
 * LU-11885: reset quota setting upon test failed
 * LU-11902: Remove obsolete lustre-utils files and entries for DEB packages
 * LU-11913: mq-deadline should be in the list of allowed schedulers
 * LU-11914: Build error for l_getsepol.c due to missing openssl/evp.h
 * LU-11923: meta_bg conflicts with resize_inode if explicitly specified for a small filesystem
 * LU-11924: Combine cancellation of llog records at osp_sync_process_committed
 * LU-11963: Add nonrotational flag to obd_statfs
 * LU-11974: Improve llapi_layout_get_by_xattr
 * LU-11982: Incorrect text in lfs migrate help
 * LU-11999: DNE performance improvement
 * LU-12015: Update Changelog for ubuntu kernel version
 * LU-12019: Recognize Debian Kernel in autoconf and unify kernel module package directory
 * LU-12027: Allow &quot;lfs find&quot; atime/mtime/ctime with fine-grained values
 * LU-12033: Git ignore generated *.install server and client files
 * LU-12044: Translate HABEO_ macros to English
 * LU-12045: Ability to split test script execution into parts
 * LU-12090: Create an lfs command to remove files by FID
 * LU-12093: CAP_SYS_RESOURCE capability check overhead
 * LU-12103: Improve block allocation for large partitions
 * LU-12133: sanityn test_35 defect
 * LU-12137: update client to use iterate_shared
 * LU-12157: lfs_migrate -v output format is not well-formed
 * LU-12159: allow lfs getname to only print fsname
 * LU-12202: remove obsolete files from contrib
 * LU-12226: upgrade e2fsprogs to 1.45.2
 * LU-12236: Support more than the default root network namespace
 * LU-12270: Support linux kernel version 4.19
 * LU-12302: 0@lo status shows as &quot;down&quot;
 * LU-12313: Mark lustre_inode_cache as reclaimable
 * LU-12326: test-framework to recognize Bochs as VM
 * LU-12330: Support per-session JOBID without using the environment
 * LU-12331: hardened usercopy whitelisting
 * LU-12333: Add additional rpc flags to DEBUG_REQ_FLAGS
 * LU-12341: Make test-framework aware of kmemleak
 * LU-12349: print error to console when flock is diabled
 * LU-12368: concurrent statfs calls on the client should be blocked
 * LU-12430: remove misleading help message of &quot;Consult build/README.kernel-source for details&quot;
 * LU-12439: Convert &quot;Response timed out...&quot; in lnet_finalize_expired_responses to CDEBUG
 * LU-12440: Correct lnet_is_health_check when msg status is 0
 * LU-12492: Deprecate live/dead router check interval parameters (instead of removing them)
 * LU-12509: lockdep warning in ofd_precreate_objects
 * LU-12516: volatile file creation in .lustre/fid/
 * LU-12520: reserve lru pages for read in batch
 * LU-12527: lustreapi.h not c++ compatible
 * LU-12533: Improve readahead RPC issuance for large window sizes
 * LU-12535: lov_page_index surprisingly slow
 * LU-12545: Cleanup LPROC_LL_*
 * LU-12553: polling mode for changelog driver
 * LU-12562: sanity 256 to use /tmp for temporary file
 * LU-12578: obdfilter-survey w/echo-client does not gain from direct-io optimizations
 * LU-12621: o2iblnd QP sizing -EINVAL retries
 * LU-12635: gcc: -Werror=implicit-fallthrough
 * LU-12776: update lustre/ChangeLog to reflect primary kernels
 * LU-12789: Mellanox MOFED 4.7 support.
 * LU-12802: racer test stucks after test finishes
 * LU-12825: Build MOFED 4.7
 * LU-12845: Fix typos in ‘lctl pcc’ help message

 Bug 
 * LU-1732: enable wide striping by default
 * LU-1957: Test failure on test suite sanity, subtest test_180b
 * LU-4322: sanity test_101a: too many discarded pages
 * LU-4398: mdt_object_open_lock may not flush conflicting handles
 * LU-5604: Lots of FAIL_ID checking are lost
 * LU-6504: Wrong indentation in socklnd
 * LU-6836: sanity-quota test_4a: Passed grace time 12, 1436542665, 1436542679
 * LU-6951: sanity test_27m: @@@@@@ FAIL: OST0 was full but new created file still use it
 * LU-7159: sanity 224c always passes
 * LU-7239: mdd_attr_set synchronous when it need not be
 * LU-7558: niobuf.c:721:ptl_send_rpc LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT ...)
 * LU-7631: conf-sanity test_82a: getstripe -c wrong: found 2, expected 3
 * LU-7912: Stale comment in osc_page_transfer_add
 * LU-8047: optimize ll_md_blocking_ast for a failed lock
 * LU-8260: osd_ldiskfs fiemap code wrong address space usage
 * LU-8299: (vvp_io.c:922:vvp_io_fault_start) ASSERTION( cl_index(obj, offset) == fio-&gt;ft_index )
 * LU-8760: sanity-lfsck test 31g hung
 * LU-9193: Multiple hangs observed with many open/getattr
 * LU-9341: PFL: append should not instantiate full layout
 * LU-9706: conf-sanity test_53a: MDS soft lockup - CPU#0 stuck for 22s in osd_trans_commit_cb
 * LU-9792: &quot;loop&quot; devices testing with zfs is broken
 * LU-9835: Potential buffer overrun in jt_opt_threads
 * LU-9862: The patch that landed for LU-7890 leads to static checker errors
 * LU-9920: Use pagevec for marking pages dirty
 * LU-9964: &gt; 1 group lock on same file (group lock lifecycle/cbpending problem)
 * LU-9971: MR: ABA problem in lnet_discover_peer_locked
 * LU-10048: osd-ldiskfs to truncate outside of main transaction
 * LU-10094: sanity test_17f: 'ls' fails with &quot;ls: reading directory *: Input/output error&quot;
 * LU-10100: sanity test_27a: setstripe failed with &quot;error on ioctl 0x8008669a for '*' (3): Invalid argument&quot;
 * LU-10153: route via two different networks not supported
 * LU-10239: Lustre crash (client): The first extent to be fit in a RPC contains 17 chunks, which is over the limit 16.
 * LU-10384: Replace nids doesn't add failover nid and add_conn string to config
 * LU-10496: sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395
 * LU-10717: several conf-sanity tests failed: FAIL: start mgs failed
 * LU-10754: sanityn test 47b fails with 'create must fail'
 * LU-10815: sanity-scrub test_16: @@@@@@ FAIL: test_16 failed with 2
 * LU-10862: ldlm_pools_{count,shrink} DLMTRACE it too noisy
 * LU-10866: Add mount options for pacemaker script
 * LU-10883: llog backup files processing is not skipped because of string сomparison error
 * LU-10913: conf-sanity test_106: OST0000-osc-MDT0000: can't cancel record
 * LU-10929: skip sanity/315 if IO accounting is not supported
 * LU-10931: failed peer discovery still taking too long
 * LU-10949: LBUG lu_object.h:688:lu_object_get) ASSERTION( atomic_read(&amp;o-&gt;lo_header-&gt;loh_ref) &gt; 0 ) failed
 * LU-10953: sanity/300k fails on local small setups
 * LU-10966: `lfs check` documentation is misleading
 * LU-11090: sanity-lfsck test_4 failed: Oops: 0000 [#1] IP: qsd_config 0x110/0x350 [lquota]
 * LU-11111: crash doing LFSCK: orph_index_insert) ASSERTION( !(obj-&gt;mod_flags &amp; ORPHAN_OBJ)
 * LU-11157: sanity test_42e: invalid arithmetic operator (error token is &quot;.9&quot;)
 * LU-11161: sanity test_160g fails for DNE with mds2: User cl9 still registered
 * LU-11204: mdt_reint_unlink-&gt;lu_object_put crash
 * LU-11206: recovery-mds-scale test failover_ost fails with 'import is not in FULL state'
 * LU-11208: Interop 2.10.4&lt;-&gt;master sanity test_27G: 'testpool' is not empty
 * LU-11239: sanity-lfsck test 36a fails with 'Fail to split mirror'
 * LU-11243: Assertion and hang upon lod_add_device failure
 * LU-11251: replay-single test_70b test: ASSERTION( req_transno &lt; next_transno ) failed: LBUG
 * LU-11292: LNet: Run discovery protocols on routers
 * LU-11330: replay-single test_70d: Directory not empty rmdir fails
 * LU-11359: racer test 1 times out with client hung in dir_create.sh, ls, … and MDS in ldlm_completion_ast
 * LU-11401: Interop: sanity-flr test 203 needs server version check
 * LU-11402: Interop: sanity-quota test 60 needs a server version check
 * LU-11403: lov_io_init_empty Page fault on a file without stripes
 * LU-11409: only first 100 OSC can shrink grants
 * LU-11413: Large performance degradation in routing environment.
 * LU-11418: hung threads on MDT and MDT won't umount
 * LU-11426: 2/2 Olafs agree: changelog entries are emitted out of order
 * LU-11434: Interop 2.10.5&lt;-&gt;master conf-sanity test_109a: clear_conf failed
 * LU-11444: RPC resend may corrupt the data
 * LU-11453: sanity test 184a: Basic layout swap panics on Power8
 * LU-11485: MDS allows &quot;lfs setstripe&quot; to mark last mirror as stale
 * LU-11495: t-f defect: NAME=ncli llmount failed with FILESET set
 * LU-11496: recovery scale tests with IOR load are not functional
 * LU-11503: ha.sh defects: ha.sh: line 399: (1476438773 - start_time) / nr_loops: division by 0
 * LU-11537: Fix assertion: osd_trans_exec_op) ASSERTION( !ldiskfs_track_declares_assert )
 * LU-11542: Race between imp_state and imp_invalid
 * LU-11549: Unattached inodes after 3 min racer run.
 * LU-11566: sanity test_60aa: llog_print_cb) not enough space for print log records
 * LU-11568: Get rid of SLAB_DESTROY_BY_RCU
 * LU-11575: make debs fail on debian without systemd enabled
 * LU-11579: cl_file_inode_init) ASSERTION(inode-&gt;i_state &amp; (1 &lt;&lt; 3) ) failed:
 * LU-11584: kernel BUG at ldiskfs.h:1907!
 * LU-11585: recovery-double-scale test_pairwise_fail: Oops: BUG: unable to handle kernel NULL pointer dereference: qsd_lqe_read+0x284/0x570 [lquota]
 * LU-11586: sanity-pfl test_10: FAIL: /mnt/lustre/d10.sanity-pfl/file2 does not inherite root layout (subdiretory mounts)
 * LU-11591: MDS n003 crash, LustreError: 20558:0:(osp_sync.c:1258:osp_sync_thread) ASSERTION( thread-&gt;t_flags != SVC_RUNNING ) failed: 4632 changes, 16403 in progress, 0 in flight
 * LU-11594: sanity test_103a: FAIL: permissions failed
 * LU-11601: IR doesn't handle EAGAIN after initial connect when pinger_recov is 0
 * LU-11605: create_count stuck in 0 after changeing max_create_count to 0 and back 20 000
 * LU-11617: lockdep exposed a possible deadlock in chlg_open
 * LU-11620: BUG: sleeping function called from invalid context at mm/slub.c:940
 * LU-11625: ls -l returns &quot;ls: cannot access Invalid argument&quot;
 * LU-11626: mdc: obd might go away while referenced by code in mdc_changelog
 * LU-11634: sanityn/test_77 improvements
 * LU-11636: t-f test_mkdir does not support interop with non DNEII servers
 * LU-11647: niobuf.c:330:ptlrpc_register_bulk) ASSERTION( desc-&gt;bd_md_count == 0 ) failed:
 * LU-11658: The cl_cache may be uninitialized while osc is activating
 * LU-11661: sanityn: test_47g failed
 * LU-11665: sanity test 101c: FAIL: Small 4k read IO 240 !
 * LU-11669: add project in yml_test_group
 * LU-11670: Incorrect size when using lockahead
 * LU-11672: improving lru_max_age policy when lru resize is disabled
 * LU-11675: Don't allow new HSM requests during CDT_INIT
 * LU-11678: sanity-quota test 1 fails with 'user write success, but expect EDQUOT'
 * LU-11681: sanity test 65i fails with 'find /mnt/lustre failed'
 * LU-11690: LBUG with very wide striping: lod_ea_store_resize) ASSERTION( round &lt;= lov_mds_md_size(2000, (0x0BD30000 | 0x0BD0))
 * LU-11691: lfs getstripe buffer overflows with very large stripe counts
 * LU-11696: &quot;lfs getsom&quot; returns &quot;24&quot; (sizeof lustre_som_attr) to userspace
 * LU-11699: lod_device_free) ASSERTION( atomic_read(&amp;lu-&gt;ld_ref) == 0 ) failed: lu is ffff881a7e690000
 * LU-11700: GPF in osp_reserved_mb_low_seq_write
 * LU-11701: replay-single test_80a: FAIL: remote creation failed
 * LU-11702: ibc_rxs is created and freed with different size
 * LU-11712: Lustre build fails when CONFIG_BLK_DEV_INTEGRITY is unset
 * LU-11716: sanity-scrub test_1b test: BUG: unable to handle kernel NULL pointer dereference IP: [&lt;ffffffff816b68a9&gt;] _raw_read_lock+0x9/0x20
 * LU-11719: Refactor search_itree
 * LU-11720: SRPM unnecessarily tied to kernel version
 * LU-11721: &quot;lfs df -i&quot; prints used blocks percentage instead of inodes
 * LU-11723: kernel update [RHEL7.6 3.10.0-957.1.3.el7]
 * LU-11729: ARM: sanity test_810: BAD WRITE CHECKSUM with adler
 * LU-11737: LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set) ASSERTION( handle ) failed:
 * LU-11742: ERROR: RPATH is not allowed
 * LU-11743: lctl pool operations cannot be run on separate MGS system
 * LU-11749: sanity-sec test 23b fails with 'Should return gid=60010 or 60010 on client2'
 * LU-11750: lsvcgssd crash with latest krb5-libs
 * LU-11752: sanity test 271f fails with '1 READ RPC occured'
 * LU-11755: Remove redundant LASSERT in ns_is_{client,server}
 * LU-11756: kib_conn leak
 * LU-11757: lfs mkdir: Cannot allocate memory
 * LU-11758: remove assertion from osp_statfs
 * LU-11759: racer cleanup
 * LU-11760: formatted OST recognition change
 * LU-11761: blocked MDT mount and high cpu usage from lodXXXX_recYYYY threads
 * LU-11765: during failover test run, mdtest job fails, numerous stat failures 'No such file or directory'
 * LU-11768: sanity-quota test 6 fails with ‘LNet: Service thread pid &lt;pid&gt; was inactive for …’
 * LU-11770: preserve kernel API when T10-PI patches are applied
 * LU-11771: bad output in target_handle_reconnect: Recovery already passed deadline 71578:57
 * LU-11773: setstripe YAML API does not support lcme_flags
 * LU-11777: interop: sanityn test 102 fails with 'check_fhandle_syscalls failed'
 * LU-11778: interop: sanity test 317 fails with 'Expected Block 4096 got 10240 for f317.sanity'
 * LU-11779: interop: sanity-hsm test 255 fails with 'request on 0x200007935:0x268:0x0 is not SUCCEED on mds1'
 * LU-11780: interop: replay-single test 132a fails with Component #1 instantiation was not replayed
 * LU-11790: ext4: include terminating u32 in size of xattr entries when expanding inodes
 * LU-11796: Remove LASSERT(r0-&gt;lo_nr &lt;= lov_targets_nr(dev)) in maintenance branches
 * LU-11798: cur_grant goes to 0 and never increases with 2.8 client and 2.10 server
 * LU-11816: LNet Health: Correct timeout defaults
 * LU-11821: sizeof needs correct casting to compare with negatives
 * LU-11825: Remove LU-8964/pio feature &amp; supporting framework
 * LU-11827: Race between llog_cat_declare_add_rec and llog_cat_current_log
 * LU-11828: PFL crashes when invariant checking is enabled
 * LU-11835: DOM open resend doesn't return size
 * LU-11836: DOM read-open resend vs getattr deadlock
 * LU-11849: lfs find ! --projid doesn't work as expected
 * LU-11851: kernel BUG at ...ldiskfs/htree_lock.c:665
 * LU-11855: fix out-of-block vars
 * LU-11860: lnetctl fails when configuring numeric addresses
 * LU-11861: master can't build with lock debug enabled
 * LU-11862: lfsck avoid unexpected repairing
 * LU-11867: OST precreate object FID in LMA mismatch causes OST unusable
 * LU-11868: ZFS ea size limited to 32K
 * LU-11871: Missing man page lctl-nodemap-set-fileset
 * LU-11873: sanity test_801a: FAIL: (2) unexpected barrier status 'expired'
 * LU-11878: sanity test 103b: OOM because of too many bash processes: page allocation stalls for 18420ms
 * LU-11880: Allow Lustre build on Raspbian and other Debian derivatives
 * LU-11881: lctl pool_list: Value too large for defined data type (75)
 * LU-11883: Make deny_unknown visible on default nodemap
 * LU-11891: make --mdt-index usage consistent
 * LU-11892: Memory leak in MDT Coordinator
 * LU-11894: Check for asymmetrical route messages in LNet
 * LU-11897: ineffective memory allocation in ptlrpc
 * LU-11898: libcfs module parameter libcfs_debug_mb has no effect
 * LU-11899: Possible resource/memory leaks
 * LU-11905: Mismatched ENTRY/RETURN in mdc_intent_open_pack
 * LU-11907: sanity fails to cleanup d60g.sanity
 * LU-11911: lov_iocontrol: BUG: unable to handle kernel NULL pointer dereference at 0000000000000100
 * LU-11917: sanity test_258a: error, failed to take i_mutex, rc=0
 * LU-11919: cl_max_dirty_pages not zeroed on startup
 * LU-11920: memory leak in lod_lds_buf_get
 * LU-11922: mkfs.lustre in 1.44.3.wc1 causes corruption if 'metadata_csum' option enabled
 * LU-11925: Attributes not updated after open+append and write to archived,released file
 * LU-11926: Lost lease lock on migrate error
 * LU-11927: support for SLES12 SP4
 * LU-11930: sbi_flags printing broke
 * LU-11931: RDMA packets sent from client to MGS are timing out
 * LU-11933: sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete ASSERTION( dt_object_exists(dt) )
 * LU-11934: replay-single test_70c: Oom on client
 * LU-11943: many input/output error after soak running for couple of hours
 * LU-11944: WARNING: at ../fs/attr.c:212 notify_change doing tiny_writes
 * LU-11946: configure --enable-dist fails if libyaml-devel or zlib-devel are not installed
 * LU-11947: Regression caused by LU-11461
 * LU-11951: sanity: test_231a failure, idle disconnect &amp; import generation disagreement
 * LU-11956: conf-sanity test_32a failed with 1
 * LU-11960: Add missing libssl-dev DEB package
 * LU-11964: Heavy load and soft lockups on MDS with DOM
 * LU-11965: interop: sanity test 311 fails with 'create_count is zero'
 * LU-11967: MDS LBUG ASSERTION( o-&gt;opo_reserved == 0 ) failed
 * LU-11975: sanity test 60a hangs in llog_test_10
 * LU-11997: Crash in lustre_swab_fiemap
 * LU-12013: Crashes in sanity-lfsck test 13
 * LU-12014: check correct size in ll_dom_finish_open
 * LU-12017: Truncate vs setxattr deadlock with DoM
 * LU-12018: deadlock on OSS: quota reintegration vs memory release
 * LU-12020: LustreError: 32657:0:(mdc_lib.c:137:mdc_pack_name) ASSERTION( lu_name_is_valid_2(buf, cpy_len) ) failed:
 * LU-12021: Error message of mdt_attr_valid_xlate when 2.12 client to 2.10 server
 * LU-12025: Adding OST may cause EIO - delay activation of new OSTs on existing filesystem
 * LU-12030: rhel7.6 removed kmalloc-128 slab
 * LU-12034: env allocation in ptlrpc_set_wait causes slowdown on the client
 * LU-12037: Possible DNE issue leading to hung filesystem
 * LU-12039: Fix 'lfs mkdir' in sanity-selinux
 * LU-12040: File lost during recovery
 * LU-12042: sanity test_103a: FAIL: LU-974 umask test failed under remote dir
 * LU-12053: intermittent ping failures with MR/DD
 * LU-12055: lod: ignore root layout if parent's one includes pool
 * LU-12059: Update ZFS Version to 0.7.13
 * LU-12065: Client got evicted when lock callback timer expired on OSS
 * LU-12068: sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0
 * LU-12069: fix styling in mdt_reint_migrate_internal
 * LU-12072: Simplify the code for lov operating
 * LU-12075: mdt_reint_migrate_internal can break CoS logic
 * LU-12080: LNet Health: references left on eq
 * LU-12081: rename shouldn't PDO lock target parent if it's remote
 * LU-12094: NULL pointer dereference in lprocfs_import_seq_write
 * LU-12095: ocd_connect_flags are wrong during reconnect
 * LU-12098: changelog_deregister appears not to reliably clear all changelog entries
 * LU-12101: lctl push causes a crash
 * LU-12104: SEGV during &quot;lfs find [--mdt-count=[+,-]&lt;count&gt;, --mdt-hash=&lt;hashtype&gt;]&quot;
 * LU-12120: LustreError: 15069:0:(tgt_grant.c:561:tgt_grant_incoming) LBUG
 * LU-12127: Improve LFSCK orphan handling
 * LU-12131: Improve GSS/SSK handling in Lustre test suite
 * LU-12138: kernel update [SLES12 SP4 4.12.14-95.13.1]
 * LU-12139: kernel update [SLES12 SP3 4.4.176-94.88.1]
 * LU-12140: DOM: limitaion of data size
 * LU-12141: Wrong requires kernel version detected of lustre kmod built against target rhel7.6
 * LU-12147: statone utility doesn't form correct string with gcc8
 * LU-12151: metadata performance difference on root and non-root user
 * LU-12152: lnetctl export corrupts memory on routers
 * LU-12160: use-after-free in osd_object_delete
 * LU-12165: Enable quota enforcement crashed kernel
 * LU-12166: Tests if ZFS backend project supported is broken
 * LU-12168: obdfilter-survey SHORT msg
 * LU-12169: Assertion on mdc_pack_name
 * LU-12172: kernel update [RHEL7.6 3.10.0-957.10.1.el7]
 * LU-12178: MDS deadlock with 2.12.0 (quotas?)
 * LU-12195: sanity/43 and sanityn/14 fail on local setups
 * LU-12199: md's are not detached from uncommitted messages that have health check performed on them
 * LU-12203: sanity-lfsck test_36[b,c] defect: tests are to be skipped for FILESET
 * LU-12206: error handling on mdt_init0
 * LU-12208: formatall defect: LU-8323 breaks interop with mixed old clients installed
 * LU-12212: Often requests timeouts during dbench run
 * LU-12218: ASSERTION( mbits == (req-&gt;rq_mbits &amp; PTLRPC_BULK_OPS_MASK) ) failes if OBD_CONNECT_BULK_MBITS isn't used.
 * LU-12219: obdfilter-survey: line 509: size * 1024 / (actual_rsz * thr): division by 0 (error token is &quot;)&quot;)
 * LU-12221: LBUG: (statahead.c:477:ll_sai_put) ASSERTION( !sa_has_callback(sai) )
 * LU-12225: jobid_get_from_cache stalled on spin_lock(&amp;pidmap-&gt;jp_lock)
 * LU-12227: Lustre init script does not check if ZFS devices are already mounted
 * LU-12229: ost-pools test 1j fails with 'test_1j returned 1'
 * LU-12242: kernel update [RHEL7.6 3.10.0-957.12.1.el7]
 * LU-12247: sanityn test 34 fails with 'test_34 returned 1'
 * LU-12248: sanity test_418: df and lfs df inodes output mismatch
 * LU-12253: lfs: double free or corruption (!prev)
 * LU-12261: sanity test 43b fails with 'expected error, got success'
 * LU-12263: build breakage with 2.12.53 tag
 * LU-12264: The lnet peer discovery queue (lnet_peer.lp_dc_pendq) is susceptible to concurrent manipulation
 * LU-12266: Fix up non-directory creation in SGID directories
 * LU-12267: sanity test_103a: some ACL tests fail with SELinux enforced
 * LU-12276: Lustre client compilation failing on CentOS 7.6 + MOFED 4.6
 * LU-12279: client got evicted due to network issue.
 * LU-12282: Pass IB_OPTIONS to debian rules doesn't work
 * LU-12296: ll_dom_lock_cancel should zero kms attribute
 * LU-12299: avoid panic for too large cpu partitions
 * LU-12306: kernel update [RHEL7.6 3.10.0-957.12.2.el7]
 * LU-12309: write fails with -EOPNOTSUPP with zfs 0.8.0-rc5 using 0.7.x pool
 * LU-12314: Add Missing Description to sanity test 258a
 * LU-12323: save_stack_trace_tsk is not exported on arm architectures
 * LU-12324: zero transno on transaction commit
 * LU-12328: FLR mirroring on 2.12.1-1 not usable if OST is down
 * LU-12335: mb_prealloc_table table read/write code is racy
 * LU-12342: lsvcgss config file overwritten on update
 * LU-12343: osc_lock.c:687:osc_ldlm_weigh_ast) LBUG
 * LU-12345: backport - ext4: optimize ext4_find_delalloc_range in nodelalloc mode
 * LU-12350: sanity-flr test_33: file content error: expected: ost1, actual: ost2
 * LU-12352: libcfs crashes with certain cpu_npartitions values
 * LU-12361: cp --sparse doesn't work correctly in
 * LU-12367: readahead page count calculation is incorrect for unaligned reads
 * LU-12374: client went down w/ panic during lustre_rmmod
 * LU-12375: lnet should start after opa service
 * LU-12381: o2iblnd uses wrong IB interface
 * LU-12382: Deadloop on tiny write
 * LU-12383: lfs project inhert difference between ldiskfs and zfs
 * LU-12385: 2.12.2: IOR read testing reliably inducing LNetError: Timed out RDMA with ...@o2ib1 (6): c: 0, oc: 0, rc: 63
 * LU-12387: l_tunedisk does not properly handle multipath devices
 * LU-12392: gcc8 can't build lfs.c
 * LU-12394: lctl get_param llite.lfs*.extents_stats only displays first two buckets
 * LU-12395: Failed dependencies while installing lustre-client-tests and lustre-iokit rpms on RHEL 8.0 client
 * LU-12396: lfs df output is treated as binary data on RHEL 8.0 client
 * LU-12399: sanity test_420: FAIL: incorrect perms on /mnt/lustre/d420.sanity/testdir/testfile
 * LU-12401: Various issues with checksum calculation for Kerberos and SSK
 * LU-12402: LNet Health: lnet_finalize recurssion
 * LU-12411: Hang on lnetctl route del
 * LU-12412: mdt: task umount:19248 blocked for more than 120 seconds
 * LU-12420: llog_reader crashes when reading llog that is not inited
 * LU-12422: LNet MR routing: update route state in case discovery is off
 * LU-12423: LNet MR routing: only push updates if discovery is enabled
 * LU-12424: LNet MR routing: possible loop when discovery is off
 * LU-12427: LNet MR routing: output a warning if discovery is off
 * LU-12428: sanity-sec: test_13 nodemap_del failed with 1
 * LU-12431: Close race on cp_state changes
 * LU-12436: Memory allocation failure error dropped
 * LU-12438: Use kernel_read/write instead of vfs_read/write for PCC
 * LU-12441: Response tracker is not detached on router ping reply
 * LU-12443: sanity test_271f: @@@@@@ FAIL: expect 1 READ RPC, occured
 * LU-12446: Variable STRIPE_BYTES Undefined under sanityn.sh/16b
 * LU-12447: Fedora28 Lustre client build failure
 * LU-12455: readcache max file size show/store inconsistent
 * LU-12456: kernel update [RHEL8.0 4.18.0-80.4.2.el8_0]
 * LU-12457: support for RHEL 7.7
 * LU-12458: kernel update [RHEL7.6 3.10.0-957.21.3.el7]
 * LU-12462: osc_cache_writeback_range: ASSERTION( ext-&gt;oe_start &gt;= start &amp;&amp; ext-&gt;oe_end &lt;= end ) failed
 * LU-12472: Kerberos: sanity-krb5.sh needs some fixes
 * LU-12473: pool_list by path incorrectly prints path as pool prefix
 * LU-12474: racer can't be found on Ubuntu 18.04
 * LU-12478: RHEL8 packaging
 * LU-12479: gcc8 warnings
 * LU-12481: conf-sanity test_61: setfattr: /mnt/lustre/f61.conf-sanity: Argument list too long
 * LU-12482: sanity-hsm uses nonexisting variable
 * LU-12483: sanity test 60h improper mds version check.
 * LU-12484: typo in is_project_quota_supported for zfs case
 * LU-12485: (osd_handler.c:2146:osd_object_release) ASSERTION( !(o-&gt;oo_destroyed == 0 &amp;&amp; o-&gt;oo_inode &amp;&amp; o-&gt;oo_inode-&gt;i_nlink == 0) ) faile
 * LU-12489: Tests marked slow that are not slow (anymore?)
 * LU-12491: large-scale: crash in lu_env_remove RIP: memcmp+0x9/0x50
 * LU-12494: kernel update [SLES12 SP4 4.12.14-95.19.1]
 * LU-12495: sanity test_413b: subdirs shouldn't be evenly distributed
 * LU-12501: lfs df can never return
 * LU-12510: mds server hangs cv_wait_common
 * LU-12513: Address YAML limitations
 * LU-12523: Add jobid to rpctrace debug messages
 * LU-12524: mdc_close matched open debug msg causes memory fragmentation
 * LU-12536: Processes stuck in unkillable sleep waiting on IO during Lustre re-export of NFS testing
 * LU-12537: recovery-mds-scale failover_mds: BUG: unable to handle kernel NULL pointer dereference at (null) IP: lnet_attach_rsp_tracker.isra.32+0xad/0x200 [lnet]
 * LU-12538: bad spinlock magic in sanity test 413b
 * LU-12539: Pass --with-o2ib option along when building deb packages
 * LU-12540: overstripe landing broke a MKFS options handling
 * LU-12558: kernel update [SLES12 SP3 4.4.180-94.100.1]
 * LU-12559: Reconnecting from idle exposes import in IMP_NEW state, resulting in EIO
 * LU-12560: acceptance small doesn't use full path for finding test-groups
 * LU-12565: Use bit locking in obd_echo
 * LU-12566: GPF when umounting client
 * LU-12568: LNetError: 28086:0:(lib-move.c:2862:lnet_detach_rsp_tracker) ASSERTION( rspt-&gt;rspt_cpt == cpt ) failed
 * LU-12569: IBLND_CREDITS_HIGHWATER does not check connection queue depth
 * LU-12570: sanity test 134a crash with SSK in use
 * LU-12575: lbuild: build MOFED on el8
 * LU-12579: runtests: test_1: @@@@@@ FAIL: Space not all freed: now 309264kB, was 307632kB.
 * LU-12581: Slab object &quot;Freechain corrupt&quot; on client
 * LU-12586: truncate to start of uninstantiated PFL component silently fails
 * LU-12587: DIO fallback to Buffer IO unexpectedly
 * LU-12589: sanity test_102a: setfattr: /mnt/lustre/f102a.sanity: Numerical result out of range
 * LU-12590: Lustre lustre_msg_hdr_size_v2 bug
 * LU-12593: update_log corruption
 * LU-12595: Attempt to add route using non-local gateway should return EHOSTUNREACH, not EINVAL
 * LU-12600: Lustre tgt_brw_write bug
 * LU-12602: Lustre mdt_getxattr_pack_reply bug
 * LU-12603: Lustre ldlm_request_cancel bug
 * LU-12604: Lustre mdt_file_secctx_unpack bug
 * LU-12605: Lustre target_handle_connect bug
 * LU-12608: kernel update [RHEL7.6 3.10.0-957.27.2.el7]
 * LU-12611: LNet Routing: don't stop adding routes for any error
 * LU-12612: Lustre osd_bufs_get bug
 * LU-12613: Lustre lustre_msg_string bug
 * LU-12614: Lustre ldlm_cancel_hpreq_check bug
 * LU-12615: Lustre mdt_object_remote bug
 * LU-12616: MDS node crashed LustreError: 23042:0:(mdt_handler.c:5135:mdt_init0) ASSERTION( info != ((void *)0) )
 * LU-12622: Another batch of test fixes for SSK
 * LU-12625: sometimes configure can't detect timespec64 with gcc8
 * LU-12626: Attempt to create existing net returns EINVAL
 * LU-12627: ASSERTION( info-&gt;fti_attr.la_valid == 0 )
 * LU-12639: sanity test 317 uses uninitialized variable $facet
 * LU-12650: get_root_path filesystem name compare error that leads to fid2path fail
 * LU-12657: sanity/103 and sanityn/25 fail with 4.18 kernel
 * LU-12660: kernel update [SLES12 SP4 4.12.14-95.29.1]
 * LU-12671: replay-vbr accesses non-existent mdd.sync_permission setting
 * LU-12672: recovery-small test 66 does not work correctly
 * LU-12674: osp should handle -EINPROGRESS on llog objects
 * LU-12675: LBUG &quot;(lu_object.c:1196:lu_device_fini) ASSERTION( atomic_read(&amp;d-&gt;ld_ref) == 0 ) failed: Refcount is 1&quot; running sanity.sh on latest master
 * LU-12681: Data corruption - due incorrect KMS with SEL files
 * LU-12685: ll_statahead_threda has incorrect error handling code
 * LU-12686: ln_mt_waitq use by lnet monitor thread is completely ineffective
 * LU-12690: error handling of ll_och_fill
 * LU-12691: obd_max_recoverable_clients is not atomic
 * LU-12694: make &quot;lfs quota&quot; display correct group quota information
 * LU-12703: sanity test_421a: FAIL: rmfid with fsname failed
 * LU-12705: build fails on power9 client
 * LU-12707: mds-survey test 1 crashes with a NULL pointer dereference in echo_object_free
 * LU-12712: sanity-pfl tests triggering “not SEL magic on SEL file”
 * LU-12719: crash in lustre_find_lwp_by_index
 * LU-12724: kernel update [RHEL7.7 3.10.0-1062.1.1.el7]
 * LU-12739: Race with discovery thread completion and message queueing
 * LU-12745: Lustre fails to compile against zfs dkms 0.8+
 * LU-12751: error is missing in few places in sanity.sh
 * LU-12753: sanity test_300a: FAIL: 1:stripe_count is 1, expect 2
 * LU-12755: CPU soft lockup on mkfs.lustre
 * LU-12758: sanity-quota test 65 fails with 'usr quota not match'
 * LU-12760: sanity test_161d incorrect use of stack_trap
 * LU-12763: Use alternate ping processing for non-mr peers
 * LU-12764: ubuntu16 client build failure
 * LU-12769: replay-dual test 0b hangs in client mount
 * LU-12773: sanity test 805 uses do_node instead of do_facet
 * LU-12777: conf-sanity test 103 fails with ‘set mdt quota type failed’
 * LU-12783: interop: sanity test 27I fails with “multiop failed”
 * LU-12784: interop: sanity test 102ha fails with “saving trusted.big on /mnt/lustre/f102ha.sanity failed”
 * LU-12790: LustreError: 96800:0:(jobid.c:103:jobid_get_from_environ) jobid: get 'PBS_JOBID' failed: rc = -75
 * LU-12795: lp_utils is duplicated and not used.
 * LU-12799: NULL pointer dereference IP: ptlrpc_disconnect_and_idle_import
 * LU-12800: mdd_is_parent goes into infinite loop
 * LU-12803: lustre in staging interferes with 2.12.2 on Ubuntu 18.04
 * LU-12824: Unable to add single Infiniband interface to multiple o2ib LNets
 * LU-12838: ptlrpc watchdog ratelimiting is broken
 * LU-12842: lctl llog_print doesn't accept snapshot names with leading number
 * LU-12844: error: ‘strncpy’ specified bound depends on the length of the source argument
 * LU-12859: Both &quot;flock&quot; and &quot;localflock&quot; appear in /proc/mounts
 * LU-12893: lnet: peer ni selection broken
 * LU-12903: lctl-pcc man page does not exist
 * LU-12932: parallel-scale test rr_alloc fails with ‘failed while setting qos_threshold_rr &amp; creat_count’
 * LU-12946: Multipath path flapping issue
 * LU-13023: PCC: The size of the PCC copy is not truncated to 0 when attach
 * LU-13030: pcc: auto_attach doesn't work after client cache cleared

 Task 
 * LU-10171: define pct(a,b) once
 * LU-11394: Updates for ZFS 0.8.0, no separate SPL
 * LU-11683: Update ZFS Version to 0.7.12
 * LU-11689: lfs project should proceed all dirs if hit errors
 * LU-12026: verify that MDS stores atime/mtime/ctime during LSOM update
 * LU-12561: Remove RHEL6/SLES11 support from master for 2.13

 Technical task 
 * LU-10777: DoM performance is bad with FIO write
 * LU-10894: Compilebench results are lower than expected
 * LU-11298: LNet: Router peer instead of Router Peer NI
 * LU-11299: LNet: Router pinger
 * LU-11300: LNet: Router Aliveness and Health
 * LU-11378: LNet: modify router selection to be MR aware
 * LU-11421: DoM: manual migration OST-MDT, MDT-MDT
 * LU-11470: LNet Health: Add drop all rule
 * LU-11475: LNet discovery: allow deleting primary_nid in router case
 * LU-11477: handle health for both incoming and outgoing messages
 * LU-11478: LNet: discovery sequence numbers could be misleading
 * LU-11551: LNet: Do not allow deletion fo rouer interface
 * LU-11641: LNet Router: handle discovery off case
 * LU-11649: lnet router: handle health off
 * LU-11664: LNet Router: push updates when router interfaces change
 * LU-12074: Update lustre tests
 * LU-12163: Fix cpt confusion
 * LU-12164: LNet: code cleanup
 * LU-12200: LNet: MR Routing use peer_timeout
 * LU-12201: LNet: detach response tracker
 * LU-12237: LNet Health: Initialize recovery ping mdh
 * LU-12249: LNet Health: list corruption
 * LU-12254: lnet: correct discovery event queue release
 * LU-12339: LNet Health: selecting loopback interface for sending
 * LU-12344: LNet Health: tweak health errors
 * LU-12526: Auto attach for PCC during IO

 Question/Request 
 * LU-9581: recovery-small tests 52 60 remove from ALWAYS_EXCEPT
 * LU-12298: Cannot Start Lustre 2.12.1 Server