File:LUG2019-Migratefs Overlay Filesystem-Thiell.pdf

I’ll introduce migratefs, an open source overlay file system that we developed in-house, to ease data migration between file systems. When maintaining access to existing data is required, the lifecycling of data storage systems usually presents two main challenges: trying to minimize the amount of old and unused data that is transferred to the new system, as well as minimizing disruption to users’ existing workflows. migratefs addresses both needs in a novel and innovative way: it combines the old and the new scratch file systems in a single and unified view, and transparently migrates modified data when it’s accessed.

This unique approach allowed us to put our new scratch file system into production within our regular cluster maintenance schedule, without the traditional need of an extended downtime that is usually required to copy all the existing data to the new system. It has also been completely transparent to our users, which can continue to access their existing data and create new files in the new system without having to modify a single line of code, saving them time and hassle.

We released migratefs as an open-source tool, to make it available to the wider community, and plan to continue improving it by making it suitable for additional file system migration use cases.