Lustre Upstreaming to Linux Kernel
Lustre client code targetted for upstream kernel submission is currently being managed as part of the 'master' Lustre development branch, and submitted patches should follow the normal Lustre patch submission process. Once all issues in the Lustre code that are known to be blocking upstream submission have been addressed, then the client will be resubmitted to the upstream kernel.
Proposed Transition to an Upstream-first Development Model
For upstreaming to work, the normal development flow has to generate patches to mainline Linux - while still supporting the distro kernels that most people use to run Lustre. This transition can occur in stages. This section provides a high-level overview. This won't be without challenges - but the majority of the transition could happen without interrupting feature work or normal development.
[I] Separate the kernel code, compatibility code, and userspace code
The Lustre tree should be reorganized to have a clear separation of concerns:
fs/lustre/ net/lnet/ net/libcfs/ lustre_compat/ tests/ utils/
The functional components of libcfs/ would stay in that directory and the compatibility components would live in lustre_compat/. Centralizing the compatibility code makes it easier to maintain and update and allows us to start removing the compatibility code from the modules themselves. lustre_compat/ could still be compiled into libcfs.ko, if we want to avoid creating even more modules.
[II] Get fs/ and net/ to compile on a mainline kernel
Once the compatibility code is isolated, we must get fs/ and net/ to compile on a mainline kernel - without any configuration or lustre_compat/ layer.
This would validated by adding build validation to each patch submitted to Gerrit. The kernel version would be pinned in the contrib/lbuild/ scripts and we'd periodically update it and fix any new build failures.
Once this is achieved, we'll have a native Linux client/server that can be run on older distributions via a compatibility layer.
[III] Move fs/ and net/ to a separate kernel tree
Transition to maintaining fs/ and net/ as a series on patches on top of a mainline kernel release. At this point, we'll generating patches to mainline Linux while retaining the ability to support older distro kernels via lustre_compat/. Similar to the previous step, we periodically rebase our Lustre patch series - fixing lustre_compat/ as needed.
This is the only step that requires a change the Lustre development workflow - patches would have to be split and sent to two different repos. We can delay this step until we have some confidence that Lustre has a path to be accepted to mainline.
[IV] Submit the patch series for inclusion
Once we are comfortable with the above process, we can submit the initial patches to add Lustre support to the kernel. Our normal development flow will generate a batch of patches to be submitted during each merge window. After the merge window, we can focus on testing and making sure that our backport to older distro kernels it still working.
FAQ
- Who will actually run the Lustre code in mainline Linux?
- Everyone. Releases for older distros will be a combination of the upstream Lustre combined with lustre_compat/ and whatever stuff the kernel won't allow (like GPUDirect).
- What does a Lustre release look like?
- We can generate a tarball by combining an upstream Lustre release from mainline along with lustre_compat/ and the userspace stuff. Vendors and third-parties can base their versions of Lustre on those tarballs. Every time a new kernel releases - a new Lustre release tarball will be created. LTS releases can center around the LTS kernel releases.
- How will we validate that fs/ and net/ build on mainline?
- It would probably be easiest to create a minimalist mainline kernel build in Jenkins. This would allow us to reuse most of the existing lbuild scripting. The build would be non-enforced at first. Testing would remain on distro kernels, since most people use those.
- Does anyone else have a similar model? Does this even work?
- AMD GPU seems to have a similar approach, at least. We should talk to other developers working in a model similar to this.
This is still a high level sketch, but I think this is a feasible path to upstreaming Lustre. We need to define a clear roadmap with tangible milestones to have a hope of upstreaming working.
But it's important that we don't disrupt developers established workflows. We don't want to complicate contributing to Lustre and we don't want to discourage people from contributing their changes upstream.
Past Talks and Coverage
This section covers previous talks about the Lustre upstreaming process.
Lustre Administrators and Developers (LAD)
LAD 2022: Native Linux client Status - James Simmons
LAD 2020: Linux Lustre Client State - James Simmons
Lustre User Group (LUG)
LUG 2024: Native Linux client Status - James Simmons
LUG 2019: Lustre - A view from the outside - Neil Brown