Lustre Quota Troubleshooting: Difference between revisions

From Lustre Wiki
Jump to navigation Jump to search
Line 10: Line 10:
If you have improvements, corrections, or more information to share on this topic please contribute to this page.
If you have improvements, corrections, or more information to share on this topic please contribute to this page.


== Quota Troubleshooting Tools ==
== Status Verification ==


=== du ===
To verify the status of space accounting and quota, run the following on every Lustre node that serves an MDT or OST and has the target mounted


If you know that a user or group has all their data below a directory you can simply use 'du -hks <directory>' and compare the output with 'lfs quota -u <user> <directory>' or 'lfs quota -g <group> <directory>'. Of course, small differences are normal in case there is I/O activity while the commands are running. Note that du is intelligent enough to not count sizes many times if files are hard linked.
lctl get_param osd-*.fsname-*.quota_slave_*.info | sed "/^osd/c\----------------------------------------"
 
=== check_quotas.pl ===
 
The perl script check_quotas.pl can be used to scan complete file systems, execute a stat for each file, add up the capacity for each user or group, and compare the output with output of 'lfs quota'. In order to not count sizes of hard linked files many times, the tool stores all inode numbers internally and therefore needs a lot of main memory. You can download the tool here: https://www.scc.kit.edu/scc/sw/lustre_tools/check_quotas.tgz
 
=== compare_user_group_ost_quotas.pl ===
 
In case user and group quotas are activated you can add up all user and group quotas reported by the OSS for each OST. (On the OSS you can get this information with 'lctl get_param osd-ldiskfs.<ost_name>.quota_slave.acct_user' and
'lctl get_param osd-ldiskfs.<ost_name>.quota_slave.acct_group'.) The result should be pretty much identical but our investigation showed that this is not true, especially for older file systems. However, we do not yet understand the reason for this difference. The perl script to easily do this comparison is called compare_user_group_ost_quotas.pl. It can be downloaded here: https://www.scc.kit.edu/scc/sw/lustre_tools/compare_user_group_ost_quotas.tgz
 
=== stat_ost_dev.pl, ost_obj_pattern_2_fid.pl and files_2_ostids.pl ===
 
If you are using ldiskfs, Lustre quotas are computed from quotas of the ldiskfs (etx4) level. However, the bug of LU-4345 (for reference see above) could have caused wrong UIDs or GIDs on the ldiskfs level which means that quotas are forged. The UIDs and GIDs on the ldiskfs level are not visible to users but an administrator can use debugfs to display them while the file system is in production. The challenge is to do this in an efficient way. The perl script stat_ost_dev.pl can be used to pipe many stat commands to same debugfs call and the perl script ost_obj_pattern_2_fid.pl can be used to filter the output in a flexible way. The two scripts can be used for many investigations, for example to find all OST object IDs which belong to a user or group. These object IDs can be compared with object IDs which belong to the user's files on the MDS (as displayed by 'lfs getstripe'). There is another perl script called files_2_ostids.pl which collects this information. For more details please check Roland Laifer's talk at LAD'15. The 3 perl scripts can be downloaded here: https://www.scc.kit.edu/scc/sw/lustre_tools/compare_ost_object_ids.tgz


Closely inspect the values for '''quota enabled:''' and '''space acct:'''. The values can be '''none''' or any combination of the letters '''u''', '''g''' and '''p''', representing user, group and project. The default value for '''quota enabled:''' is '''none''' and the default value for '''space acct:''' is '''ugp'''


== Repairing wrong Quotas ==
== Repairing wrong Quotas ==

Revision as of 13:02, 10 January 2025

Introduction

This document is based on and tested with Lustre 2.15 and LDISKFS. If your Lustre version is older or you are using ZFS, the content may not be applicable.

Space accounting and project ID tracking are enabled by default on a LDISKFS file system. However, quota restrictions are NOT enabled by default. User, Group and Project quota restrictions must be manually enabled on all targets, as well as configuring data/block and/or metadata/inode limits.

Adding to This Guide

If you have improvements, corrections, or more information to share on this topic please contribute to this page.

Status Verification

To verify the status of space accounting and quota, run the following on every Lustre node that serves an MDT or OST and has the target mounted 

lctl get_param osd-*.fsname-*.quota_slave_*.info | sed "/^osd/c\----------------------------------------"

Closely inspect the values for quota enabled: and space acct:. The values can be none or any combination of the letters u, g and p, representing user, group and project. The default value for quota enabled: is none and the default value for space acct: is ugp

Repairing wrong Quotas

Lustre quotas are recomputed if you turn them off and on. However, if UIDs and GIDs on the ldiskfs level are wrong (see above) this does not repair wrong quotas. Since manual repair is hardly possible, the only viable solution is to wait until Lustre version 2.6 or higher can be installed and use the online file system check (option layout of LFSCK) to repair wrong quotas. Note that we have not yet tried this solution.


References and Links

Retrieved from "http://wiki.lustre.org/index.php?title=Lustre_Quota_Troubleshooting&oldid=5062"