Robinhood Policy Engine

From Lustre Wiki
Revision as of 13:08, 26 May 2026 by Xyzisinus (talk | contribs)
Jump to navigation Jump to search

Introduction

The Robinhood Policy Engine is an open-source tool that assists in the management of large file systems. It keeps an updated copy of the file system metadata in a MySQL database which allows the querying of metadata from the database rather than from the file system directly, saving time and resources.

Overview

Robinhood Policy Engine is a versatile tool for managing the contents of large file systems. Originally developed at CEA for HPC environments, it is designed to perform all its tasks in parallel, making it particularly well-suited for filesystems with millions of entries and petabytes of data.

Robinhood maintains a replica of filesystem metadata in a MySQL/MariaDB database. This database can be queried at will, enabling administrators to get an overall view of filesystem contents without the overhead of directly scanning the filesystem. It provides enhanced find and du clones (rbh-find and rbh-du) that query the database instead of the filesystem, offering significantly faster results.

Main features:

  • Policy Engine: Schedule actions on filesystem entries according to admin-defined criteria, based on entry attributes (age, size, path, owner, Lustre-specific attributes, etc.).
  • User/group usage accounting: Including file size profiling and top consumers reporting.
  • Fast find and du clones: rbh-find and rbh-du query the metadata database instead of the filesystem for near-instant results.
  • Customizable alerts: Trigger notifications on filesystem entries meeting specified conditions.
  • Lustre-aware: Aware of Lustre OSTs, pools, and projects; can list/purge files per OST or pool and read MDT changelogs for near-real-time database updates.
  • Lustre/HSM integration: Schedule archive, release, and remove actions through Lustre's Hierarchical Storage Management framework.
  • Web GUI: Provides a graphical overview of filesystem contents and usage.

Robinhood supports any POSIX filesystem, but implements advanced features specifically for Lustre filesystems. It is distributed under the CeCILL-C license (LGPL-compatible).

Example: Using Robinhood to Archive Lustre Files with HSM

A common use case for Robinhood is managing Lustre/HSM (Hierarchical Storage Management) to automatically archive files from Lustre to a backend storage system (e.g., tape or object storage). Below is a short example of how this works.

First, include the built-in Lustre/HSM policy definitions in your Robinhood configuration file: 

%include "includes/lhsm.inc"

This defines three policies: lhsm_archive, lhsm_release, and lhsm_remove. Then define an archive policy to copy files to the backend after they have been stable for one hour: 

 lhsm_archive_rules {
     # Don't archive empty files or temp files
     ignore { size == 0 }
     ignore { name == "*.tmp" or name == "*.log" }

     rule default {
         condition { last_mod > 1h }
     }
 }

 lhsm_archive_trigger {
     trigger_on = scheduled;
     check_interval = 15min;
 }

To archive all eligible files immediately from the command line: 

robinhood --run=lhsm_archive --target=all

To release archived files when an OST reaches 95% capacity (freeing space back to 93%): 

 lhsm_release_trigger {
     trigger_on = ost_usage;
     high_threshold_pct = 95%;
     low_threshold_pct = 93%;
     check_interval = 5min;
 }

You can then check HSM status across the filesystem using: 

rbh-report --status-info lhsm

For a full tutorial, see Lustre/HSM with Robinhood v3 Tutorial.

Papers

Presentations

Robinhood User Group (RUG) Events

Conference Presentations

Retrieved from "http://wiki.lustre.org/index.php?title=Robinhood_Policy_Engine&oldid=5544"