Robinhood Policy Engine: Difference between revisions

From Lustre Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
=== Introduction ===
=== Introduction ===
The Robinhood Policy Engine is an open-source tool that assists in the management of large file systems. It keeps an updated copy of the file system metadata in a MySQL database which allows the querying of metadata from the database rather than from the file system directly, saving time and resources.
 
The Robinhood Policy Engine is an open-source tool that assists in the management of large file systems.
It keeps an updated copy of the file system metadata in a MySQL database.  It understands Lustre file's `hsm state`.  Based
on the state, it can perform actions specified by a policy, such as "archive all modified files to the HSM storage".
Those capablities make it an integral component in many Lustre HSM solutions.


* [https://github.com/cea-hpc/robinhood/ Robinhood Github]
* [https://github.com/cea-hpc/robinhood/ Robinhood Github]
* [https://github.com/cea-hpc/robinhood/wiki Robinhood Wiki]
* [https://github.com/cea-hpc/robinhood/wiki Robinhood Wiki]
* [https://github.com/cea-hpc/robinhood/wiki/Documentation Online Documentation]
* [https://github.com/cea-hpc/robinhood/wiki/Documentation Online Documentation]
* Mailing Lists
 
** [https://lists.sourceforge.net/lists/listinfo/robinhood-news robinhood-news]: announcement of new releases.
Mailing Lists
** [https://lists.sourceforge.net/lists/listinfo/robinhood-support robinhood-support]: user discussions and support.
* [https://lists.sourceforge.net/lists/listinfo/robinhood-news robinhood-news]: announcement of new releases.
** [https://lists.sourceforge.net/lists/listinfo/robinhood-devel robinhood-devel]: design and coding discussions.
* [https://lists.sourceforge.net/lists/listinfo/robinhood-support robinhood-support]: user discussions and support.
* [https://lists.sourceforge.net/lists/listinfo/robinhood-devel robinhood-devel]: design and coding discussions.


=== Overview ===
=== Overview ===


Robinhood Policy Engine is a versatile tool for managing the contents of large file systems. Originally developed at [https://www-hpc.cea.fr/ CEA] for HPC environments, it is designed to perform all its tasks in parallel, making it particularly well-suited for filesystems with millions of entries and petabytes of data.
Robinhood supports any POSIX file system, but implements advanced features specifically for Lustre file systems. It is distributed under the [http://www.cecill.info/licences/Licence_CeCILL-C_V1-en.html CeCILL-C] license (LGPL-compatible).
 
Robinhood maintains a replica of filesystem metadata in a MySQL/MariaDB database. This database can be queried at will, enabling administrators to get an overall view of filesystem contents without the overhead of directly scanning the filesystem. It provides enhanced ''find'' and ''du'' clones ('''rbh-find''' and '''rbh-du''') that query the database instead of the filesystem, offering significantly faster results.


'''Main features:'''
Robinhood maintains a replica of file system metadata in a MySQL/MariaDB database, which can provide an overall view of the file system
* '''Policy Engine:''' Schedule actions on filesystem entries according to admin-defined criteria, based on entry attributes (age, size, path, owner, Lustre-specific attributes, etc.).
without the overhead of directly scanning the file system.
* '''User/group usage accounting:''' Including file size profiling and top consumers reporting.
With its knowledge of the file system, it provides the following main functionalities.
* '''Fast ''find'' and ''du'' clones:''' '''rbh-find''' and '''rbh-du''' query the metadata database instead of the filesystem for near-instant results.
* '''Customizable alerts:''' Trigger notifications on filesystem entries meeting specified conditions.
* '''Lustre-aware:''' Aware of Lustre OSTs, pools, and projects; can list/purge files per OST or pool and read MDT changelogs for near-real-time database updates.
* '''Lustre/HSM integration:''' Schedule archive, release, and remove actions through Lustre's Hierarchical Storage Management framework.
* '''Web GUI:''' Provides a graphical overview of filesystem contents and usage.


Robinhood supports any POSIX filesystem, but implements advanced features specifically for [https://www.lustre.org/ Lustre] filesystems. It is distributed under the [http://www.cecill.info/licences/Licence_CeCILL-C_V1-en.html CeCILL-C] license (LGPL-compatible).
* '''Policy Engine:''' Schedule actions on file system entries according to admin-defined criteria, based on entry attributes (age, size, path, owner, Lustre-specific attributes, etc.).
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#policy-concepts Policy Concepts]
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#policy-rules Policy Rules]
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#actions Actions]
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#triggers Triggers]
* '''User/group usage accounting:''' File size profiling and top consumers reporting.  For example, Fast ''find'' and ''du'' clones: '''rbh-find''' and '''rbh-du''' query the metadata database instead of the file system for near-instant results.
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#rbh-report rbh-report]
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#rbh-find rbh-find]
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#rbh-du rbh-du]
* '''Customizable alerts:''' Trigger notifications on file system entries meeting specified conditions.
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#alerter-status-manager-actions Alerter Plugin]
** [https://github.com/cea-hpc/robinhood/wiki/How-to-setup-quota-alerts How to Setup Quota Alerts]
* '''Lustre-aware:''' Aware of Lustre OSTs and MDTs; can read MDT changelogs for near-real-time database updates.
** [https://github.com/cea-hpc/robinhood/wiki/v3_lustre_tuto Tutorial: Lustre File System Configuration]
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#enabling-lustre-v2-changelogs Lustre Changelogs]
* '''Lustre/HSM integration:''' Schedule file archive, release, and remove actions through Lustre's Hierarchical Storage Management, HSM, framework.
** [https://github.com/cea-hpc/robinhood/wiki/v3_lhsm_tuto Tutorial: Lustre/HSM with Robinhood v3]
** [https://github.com/cea-hpc/robinhood/wiki/robinhood_v3_admin_doc#lhsm-status-manager-actions lhsm Plugin]


=== Example: Using Robinhood to Archive Lustre Files with HSM ===
=== Example: Using Robinhood to Archive Lustre Files with HSM ===


A common use case for Robinhood is managing Lustre/HSM (Hierarchical Storage Management) to automatically archive files from Lustre to a backend storage system (e.g., tape or object storage). Below is a short example of how this works.
A common use case for Robinhood is managing Lustre/HSM to automatically archive files from Lustre to a backend storage system, typically slower but
 
low-cost, such as Azure's blob storage or AWS' S3. Below is an example of how this works.
First, include the built-in Lustre/HSM policy definitions in your Robinhood configuration file:


<nowiki>%include "includes/lhsm.inc"</nowiki>
First, include a policy file in your Robinhood configuration file:


This defines three policies: '''lhsm_archive''', '''lhsm_release''', and '''lhsm_remove'''. Then define an archive policy to copy files to the backend after they have been stable for one hour:
<nowiki>%include "includes/lhsm_archive.inc"</nowiki>


<nowiki>
The content of `lhsm_archive.inc` policy file can be
lhsm_archive_rules {
<pre>
    # Don't archive empty files or temp files
lhsm_archive_rules {
    ignore { size == 0 }
    # Don't archive empty files or .tmp or .log files
    ignore { name == "*.tmp" or name == "*.log" }
    ignore { size == 0 or name == "*.tmp" or name == "*.log" }
    rule default {
        # Last modification to the file should be at least 1 hour ago
        condition { last_mod > 1h }
    }
}
lhsm_archive_trigger {
    trigger_on = scheduled;
    # Schedule archive checks/actions every 15 minutes
    check_interval = 15min;
}
</pre>


    rule default {
To activate the above policy:
        condition { last_mod > 1h }
    }
}
 
lhsm_archive_trigger {
    trigger_on = scheduled;
    check_interval = 15min;
}
</nowiki>
 
To archive all eligible files immediately from the command line:


  robinhood --run=lhsm_archive --target=all
  robinhood --run=lhsm_archive --target=all


To release archived files when an OST reaches 95% capacity (freeing space back to 93%):
You can then check HSM status across the file system using:
 
<nowiki>
lhsm_release_trigger {
    trigger_on = ost_usage;
    high_threshold_pct = 95%;
    low_threshold_pct = 93%;
    check_interval = 5min;
}
</nowiki>
 
You can then check HSM status across the filesystem using:
 
rbh-report --status-info lhsm
 
For a full tutorial, see [https://github.com/cea-hpc/robinhood/wiki/v3_lhsm_tuto Lustre/HSM with Robinhood v3 Tutorial].
 
== Papers ==
* [https://arxiv.org/pdf/1505.01448.pdf Taking back control of HPC file systems with Robinhood Policy Engine] — Thomas Leibovici (CEA), International Workshop on the Lustre Ecosystem, 2015. ([http://lustre.ornl.gov/ecosystem-2015/documents/LustreEco2015-Leibovici.pdf Presentation slides])
* [https://cug.org/proceedings/cug2014_proceedings/includes/files/pap157.pdf Using Robinhood to Purge Data from Lustre File Systems] — Tina M. Declerck (LBNL), Cray User Group, 2014.


== Presentations ==
rbh-report --status-info lhsm_archive


=== Robinhood User Group (RUG) Events ===
== Papers and Presentations ==
* [https://github.com/cea-hpc/robinhood/wiki/RUG2017#agenda--slide-decks Robinhood User Group 2017] — October 3, 2017, Paris ([http://robinhood.sourceforge.net/rug17/rug17.tgz Download all slides])
** [http://robinhood.sourceforge.net/rug17/RUG17-ThomasLeibovici-ProjectUpdate.pdf Project Update] (Thomas Leibovici, CEA)
** [http://robinhood.sourceforge.net/rug17/RUG17-Cray-CorySpitz-FastScanning.pdf Fast Lustre file system scanning] (Cory Spitz, Cray)
** [http://robinhood.sourceforge.net/rug17/RUG17-Seagate-RobinhoodOptimizations.pdf Improving Robinhood performance for large scale deployments] (Nathan Rutman, Cray)
** [http://robinhood.sourceforge.net/rug17/RUG17-GSI-SiteStatus.pdf Site status: GSI] (Gabriele Iannetti, GSI)
** [http://robinhood.sourceforge.net/rug17/RUG17-UnivCambridge-SiteStatus.pdf Site status: University of Cambridge] (Matthew Raso-Barnett, University of Cambridge)
* [https://github.com/cea-hpc/robinhood/wiki/RUG2016#agenda--slide-decks Robinhood User Group 2016] — September 19, 2016, Paris ([http://robinhood.sourceforge.net/rug16/rug16.tgz Download all slides])
** [http://robinhood.sourceforge.net/rug16/RUG16_project_update.pdf Project Update] (Thomas Leibovici, CEA)
** [http://robinhood.sourceforge.net/rug16/RUG16_DKRZ.pdf Experience with Robinhood 2.5.5 on Lustre with DNE] (Carsten Beyer, DKRZ)
** [http://robinhood.sourceforge.net/rug16/RUG16_checker.pdf Robinhood v3 new use case: integrity check] (Dominique Martinet, CEA)
** [http://robinhood.sourceforge.net/rug16/RUG16_Cray_DTacchella.pdf Add HSM to existing Lustre FS] (Davide Tacchella, Cray)
** [http://robinhood.sourceforge.net/rug16/RUG16_rbh3_lhsm.pdf Focus: Lustre/HSM and Robinhood v3] (Henri Doreau, CEA)
** [http://robinhood.sourceforge.net/rug16/RUG16_Stanford_Sherlock.pdf Using Robinhood v3 for Lustre/Cloud storage seamless integration] (Stephane Thiell, Stanford)
** [http://robinhood.sourceforge.net/rug16/RUG16-Dilger_Lustre-HSM.pdf Lustre/HSM proposals] (Andreas Dilger)
* [https://github.com/cea-hpc/robinhood/wiki/RUG2015#agenda Robinhood User Group 2015] — September 21, 2015, Paris
** [http://robinhood.sourceforge.net/rug15/rug_rbh_project.pdf Project status] (Thomas Leibovici, CEA)
** [http://robinhood.sourceforge.net/rug15/Robinhood_and_Lustre-HSM_a_return_of_experience.pdf Robinhood and Lustre-HSM: a return of experience] (Bruno Faccini, Intel)
** [http://robinhood.sourceforge.net/rug15/CSCS_Site_Status.pdf CSCS site status] (Carmelo Ponti, CSCS)
** [http://robinhood.sourceforge.net/rug15/s+c_robinhood_technical_computing.pdf RobinHood in Technical Computing Environments] (Daniel Kobras, Science+Computing)
** [http://robinhood.sourceforge.net/rug15/rug_robinhood_v3.pdf Robinhood v3] (Thomas Leibovici, CEA)
** [http://robinhood.sourceforge.net/rug15/Atos-RUG2015-210915-V3.0_robinhood_benchmarks.pdf Benchmarking Robinhood ingest rate on various architectures] (Bruno Travouillon, Atos)
** [http://robinhood.sourceforge.net/rug15/rbh_roadmap.pdf Short-term and mid-term roadmap for Robinhood] (Henri Doreau, CEA)


=== Conference Presentations ===
* [https://github.com/cea-hpc/robinhood/wiki/Papers-and-presentations Robinhood User Group (RUG) event papers and presentations]
* [http://cdn.opensfs.org/wp-content/uploads/2017/06/Thur9.5-FaberColin-28LUG2920Seagate20Robinhood20Optimizations2028629.pdf Improving overall Robinhood performance for use on large scale deployments] — Colin Faber (Seagate), LUG 2016
* Improving overall Robinhood performance for use on large scale deployments (LUG 2016) [https://www.youtube.com/watch?v=O4quji8OnVA Video] [http://cdn.opensfs.org/wp-content/uploads/2017/06/Thur9.5-FaberColin-28LUG2920Seagate20Robinhood20Optimizations2028629.pdf Slides]
* Robinhood v3 and Beyond (LAD 2015)
* Data Life Cycle Monitoring using RobinHood at Scale (LAD 2015) [https://www.youtube.com/watch?v=W4-YGQlq9QI Video] [https://www.eofs.eu/wp-content/uploads/2024/02/17_gabriele_paciucci_lad15-rbh-04.pdf Slides]
** [https://www.youtube.com/watch?v=W1H4lzUold4 Video]
* Robinhood v3 and Beyond (LAD 2015) [https://www.youtube.com/watch?v=W1H4lzUold4 Video] [https://www.eofs.eu/wp-content/uploads/2024/02/18_thomas_leibovici_lad_rbh_v3.pdf Slides]
** [http://www.eofs.eu/_media/events/lad15/18_thomas_leibovici_lad_rbh_v3.pdf Slides]
* Taking back control of HPC file systems with Robinhood Policy Engine, International Workshop on the Lustre Ecosystem, 2015 [https://arxiv.org/pdf/1505.01448.pdf Slides]
* Data Life Cycle Monitoring using RobinHood at Scale (LAD 2015)
* Using Robinhood to Purge Data from Lustre File Systems Cray User Group, 2014 [https://cug.org/proceedings/cug2014_proceedings/includes/files/pap157.pdf Slides]
** [https://www.youtube.com/watch?v=W4-YGQlq9QI Video]
* Robinhood Policy Engine (LUG 2013) [https://www.youtube.com/watch?v=b9qyeDyTF64 Video] [http://cdn.opensfs.org/wp-content/uploads/2013/04/lug13-robinhood.pdf Slides]
** [http://www.eofs.eu/fileadmin/lad2015/slides/17_Gabriele_Paciucci_LAD15-RBH-04.pdf Slides]
* ROBINHOOD Policy Engine (LUG 2013)
** [https://www.youtube.com/watch?v=b9qyeDyTF64 Video]
** [http://cdn.opensfs.org/wp-content/uploads/2013/04/lug13-robinhood.pdf Slides]
* [http://docplayer.net/6659180-Lustre-usage-monitoring-what-the-are-users-doing-with-my-filesystem.html What the &#%@ are users doing with my filesystem?] — Kilian Cavalotti (CEA), LAD 2013


[[Category:Monitoring]]
[[Category:Monitoring]]
[[Category:Tools]]
[[Category:Tools]]

Latest revision as of 06:25, 28 May 2026

Introduction

The Robinhood Policy Engine is an open-source tool that assists in the management of large file systems. It keeps an updated copy of the file system metadata in a MySQL database. It understands Lustre file's `hsm state`. Based on the state, it can perform actions specified by a policy, such as "archive all modified files to the HSM storage". Those capablities make it an integral component in many Lustre HSM solutions.

Mailing Lists

Overview

Robinhood supports any POSIX file system, but implements advanced features specifically for Lustre file systems. It is distributed under the CeCILL-C license (LGPL-compatible).

Robinhood maintains a replica of file system metadata in a MySQL/MariaDB database, which can provide an overall view of the file system without the overhead of directly scanning the file system. With its knowledge of the file system, it provides the following main functionalities.

Example: Using Robinhood to Archive Lustre Files with HSM

A common use case for Robinhood is managing Lustre/HSM to automatically archive files from Lustre to a backend storage system, typically slower but low-cost, such as Azure's blob storage or AWS' S3. Below is an example of how this works.

First, include a policy file in your Robinhood configuration file:

%include "includes/lhsm_archive.inc"

The content of `lhsm_archive.inc` policy file can be 

lhsm_archive_rules {
    # Don't archive empty files or .tmp or .log files
    ignore { size == 0 or name == "*.tmp" or name == "*.log" }
    rule default {
        # Last modification to the file should be at least 1 hour ago
        condition { last_mod > 1h }
    }
}
lhsm_archive_trigger {
    trigger_on = scheduled;
    # Schedule archive checks/actions every 15 minutes
    check_interval = 15min;
}

To activate the above policy: 

robinhood --run=lhsm_archive --target=all

You can then check HSM status across the file system using: 

rbh-report --status-info lhsm_archive

Papers and Presentations

Retrieved from "http://wiki.lustre.org/index.php?title=Robinhood_Policy_Engine&oldid=5547"