OST Pool Quotas

Description
OST Pool Quotas feature gives an ability to limit user's(group's/project's) disk usage at OST pool.

Each Quota Pool(QP) duplicates content of OST pool. Thus QP could be tuned with standard lctl pool_new/add/remove/erase commands. All QPs are subset of a global pool that includes all OSTs and MDTs(DOM case).

See more details in OST Pool Quotas HLD.

DOM and MDT pools
From Quota Master point of view "data" MDTs are regular members together with OSTs. But until we haven't a mechanism to group MDTs in pools, Quota Pools support only OSTs.

New options at lfs quota/setquota
2 new options were introduced to support QP at lfs quota and setquota:
 * -o "pool_name" is new option to be used with lfs setquota to set quota for user/qroup/project per quota pool.
 * Example: lfs setquota -u quota_usr -o "qpool1" -B10M /mnt/lustre
 * --pool "pool_name" is an option to show quota for user/group/project per quota pool.
 * Example: lfs quota -u quota_usr --pool "qpool1" /mnt/lustre

Hard limit example
Let's imagine you haven't any OST pool(if you want to setup already existed pool, go to 3): Note, you need to setup wide-system hard limit before setting QP limit. If you don't need to limit user at all OSTs and MDTs at system, only per pool, I would recommend to set some unrealistic big hard limit. Without that OSTs don't know that quota for quota_usr is enabled.
 * 1) lctl pool_new qpool1 /mnt/lustre
 * 2) lctl pool_add qpool1 OST000 /mnt/lustre
 * 3) lfs setquota -u quota_usr -B100T /mnt/lustre /* it is a limit for global pool. QPs don't work properly without that */
 * 4) lfs setquota -u quota_usr -o "qpool1" -B100M /mnt/lustre

Soft limit example
Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes   quota   limit   grace   files   quota   limit   grace /mnt/lustre  14336*  10240 107374182400     26s       1       0       0       -
 * 1) lctl pool_new qpool1 /mnt/lustre
 * 2) lctl pool_add qpool1 OST000 /mnt/lustre
 * 3) lfs setquota -u quota_usr -B100T /mnt/lustre /* notify OSTs that to enforce quota for quota_usr */
 * 4) lfs setquota -t -u --block-grace 30 -o qpool1 /mnt/lustre /* set block grace 30 s for all users at qpool1 */
 * 5) lfs setquota -u quota_usr -b10M -o qpool1 /mnt/lustre /* soft limit 10M for quota_usr */
 * 6) /runas -u quota_usr dd if=/dev/urandom of=/mnt/lustre/dir/f1 bs=1M count=15
 * 7) [root@dhcppc3 tests]# lfs quota -u quota_usr --pool qpool1 /mnt/lustre

Docs
OST Pool Quotas HLD

OST Pool Quota Test Plan

Tickets
LU-11023