File:LUG2019-Cross Tier Unified Namespace-Chaarawi.pdf

   This presentation will give an update on the cross-tier unified namespace concept that was introduced at LAD’18.

   This functionality allows a userspace distributed object store like DAOS to be integrated with a Lustre filesystem under a single unified namespace. The open-source DAOS object store will be used as the baseline example, but the suggested approach and associated Lustre changes are effectively agnostic to the object store and are designed to work with any storage tier relying on a URI to identify a collection of objects (i.e. storage containers in the DAOS case). The presentation will also cover:
   – how the cross-tier consistency could be maintained by storing a back reference to the Lustre object (i.e. FID) in the container attribute.
   – improvements to the software stack presented at the LAD. The new model allows native Lustre files and directories to be accessed directly without involving FUSE.

   The Lustre changes required to support this integration will then be presented in further details. DAOS containers will be represented in the Lustre namespace through files and directories with special – aka foreign – LOV and LMV EAs (see LU-11376 for further details). Beyond the unified namespace, the foreign LOV and LMV formats can be reused to implement future features like the Lustre Client Container Image (CCI) or advanced HSM functionality.

   Finally, dataset migrations between the object store and the Lustre tier will be considered with a discussion on the different use cases (i.e. I/O middleware-level vs DAOS-level copy) and tools that can potentially be leveraged.