Lustre Quota Troubleshooting
In the past, Lustre quotas frequently caused problems. One reason are numerous implementation changes, e.g. due to adaptions to underlying file systems.
Recently, the bug explained in https://jira.hpdd.intel.com/browse/LU-4345 ( fix for b2_5 branch at http://review.whamcloud.com/11435 ) caused quotas to be forged. If you have been running file systems with version Lustre 2.4 your quotas are probably still (slightly) incorrect.
During the first part of his talk at LAD'15 Roland Laifer explained tools for wrong quota investigation. Below you can find links to the tools and further information.
Adding to This Guide
If you have improvements, corrections, or more information to share on this topic please contribute to this page. Ideally this would become a community resource.
Quota Troubleshooting Tools
If you know that a user or group has all their data below a directory you can simply use 'du -hks <directory>' and compare the output with 'lfs quota -u <user> <directory>' or 'lfs quota -g <group> <directory>'. Of course, small differences are normal in case there is I/O activity while the commands are running. Note that du is intelligent enough to not count sizes many times if files are hard linked.
The perl script check_quotas.pl can be used to scan complete file systems, execute a stat for each file, add up the capacity for each user or group, and compare the output with output of 'lfs quota'. In order to not count sizes of hard linked files many times, the tool stores all inode numbers internally and therefore needs a lot of main memory. You can download the tool here: https://www.scc.kit.edu/scc/sw/lustre_tools/check_quotas.tgz
In case user and group quotas are activated you can add up all user and group quotas reported by the OSS for each OST. (On the OSS you can get this information with 'lctl get_param osd-ldiskfs.<ost_name>.quota_slave.acct_user' and 'lctl get_param osd-ldiskfs.<ost_name>.quota_slave.acct_group'.) The result should be pretty much identical but our investigation showed that this is not true, especially for older file systems. However, we do not yet understand the reason for this difference. The perl script to easily do this comparison is called compare_user_group_ost_quotas.pl. It can be downloaded here: https://www.scc.kit.edu/scc/sw/lustre_tools/compare_user_group_ost_quotas.tgz
stat_ost_dev.pl, ost_obj_pattern_2_fid.pl and files_2_ostids.pl
If you are using ldiskfs, Lustre quotas are computed from quotas of the ldiskfs (etx4) level. However, the bug of LU-4345 (for reference see above) could have caused wrong UIDs or GIDs on the ldiskfs level which means that quotas are forged. The UIDs and GIDs on the ldiskfs level are not visible to users but an administrator can use debugfs to display them while the file system is in production. The challenge is to do this in an efficient way. The perl script stat_ost_dev.pl can be used to pipe many stat commands to same debugfs call and the perl script ost_obj_pattern_2_fid.pl can be used to filter the output in a flexible way. The two scripts can be used for many investigations, for example to find all OST object IDs which belong to a user or group. These object IDs can be compared with object IDs which belong to the user's files on the MDS (as displayed by 'lfs getstripe'). There is another perl script called files_2_ostids.pl which collects this information. For more details please check Roland Laifer's talk at LAD'15. The 3 perl scripts can be downloaded here: https://www.scc.kit.edu/scc/sw/lustre_tools/compare_ost_object_ids.tgz
Repairing wrong Quotas
Lustre quotas are recomputed if you turn them off and on. However, if UIDs and GIDs on the ldiskfs level are wrong (see above) this does not repair wrong quotas. Since manual repair is hardly possible, the only viable solution is to wait until Lustre version 2.6 or higher can be installed and use the online file system check (option layout of LFSCK) to repair wrong quotas. Note that we have not yet tried this solution.
References and Links
- Roland Laifer, "Lustre tools for ldiskfs investigation and lightweight I/O statistics", LAD2015. http://www.eofs.eu/fileadmin/lad2015/slides/13_Roland_Laifer_kit_20150922.pdf