OST Pool Quotas Test Report - Revision history

Sergey: /* Cluster that was used for performance, stress and failover testing */

2020-10-30T08:51:55Z

Cluster that was used for performance, stress and failover testing

← Older revision		Revision as of 01:51, 30 October 2020
Line 61:		Line 61:

	Each client had about 62GB of total memory.		Each client had about 62GB of total memory.

	max_dirty_mb was set to 10MB at each client:		max_dirty_mb was set to 10MB at each client:
	lctl set_param osc.*.max_dirty_mb=10		lctl set_param osc.*.max_dirty_mb=10

Sergey: /* Cluster that was used for performance, stress and failover testing */

2020-10-30T08:51:09Z

Cluster that was used for performance, stress and failover testing

← Older revision		Revision as of 01:51, 30 October 2020
Line 61:		Line 61:

	Each client had about 62GB of total memory.		Each client had about 62GB of total memory.
			max_dirty_mb was set to 10MB at each client:
			lctl set_param osc.*.max_dirty_mb=10
			10MB was chosen to avoid storing too much data in a client cache. For example, if some user makes IO load in a system with 4OSTs from 5 clients with max_dirty_mb set to 100, clients may store in a cache 2GB(100MB per each OST at 5 clients). As data from the cache may be written over quota, max_diry_mb shouldn't be high to avoid unexpected EDQUOT. 1MB was refused to don't clear caches too often, so 10MB was finally picked as a medium value.

	== Performance testing ==		== Performance testing ==
	As a baseline was used d0452cf - the patch before "LU-11023 quota: quota pools for OSTs"(09f9fb32).		As a baseline was used d0452cf - the patch before "LU-11023 quota: quota pools for OSTs"(09f9fb32).

Adilger: /* Performance testing */ expand DIO, BIO

2020-10-29T18:39:51Z

Performance testing: expand DIO, BIO

← Older revision		Revision as of 11:39, 29 October 2020
Line 78:		Line 78:
	{\| class="wikitable"		{\| class="wikitable"
	\|- style="background-color:#efefef;"		\|- style="background-color:#efefef;"
	! style="font-weight:bold;" \| cslmo17 HDD IOR Throughput (~~DIO~~)		! style="font-weight:bold;" \| cslmo17 HDD IOR Throughput (Direct IO)
	! d0452cf		! d0452cf
	! dd35e17		! dd35e17
Line 95:		Line 95:
	{\| class="wikitable"		{\| class="wikitable"
	\|- style="background-color:#efefef;"		\|- style="background-color:#efefef;"
	! style="font-weight:bold;" \| cslmo17 HDD IOR Throughput (~~BIO~~)		! style="font-weight:bold;" \| cslmo17 HDD IOR Throughput (Buffered IO)
	! d0452cf		! d0452cf
	! dd35e17		! dd35e17

Sergey: /* Performance testing */

2020-10-26T19:01:27Z

Performance testing

@@ Line 92: / Line 92: @@
 | 10694.86 MiB/sec
 | -0.07%
 |}

Sergey: /* Performance testing */

2020-10-26T18:54:08Z

Performance testing

@@ Line 75: / Line 75: @@
   Each user: block hard limit 1T to enforce quota.
   Each user: block hard limit 200G per each pool.
 == Stress testing ==

Sergey: /* Performance testing */

2020-10-26T18:15:38Z

Performance testing

← Older revision		Revision as of 11:15, 26 October 2020
Line 65:		Line 65:

	It was compared with 09f9fb32 + 8704d14c(LU-13677 quota: qunit sorting doesn't work). 8704d14c was applied directly at 09f9fb32 as this patch solves a problem introduced in the main pool quotas patch.		It was compared with 09f9fb32 + 8704d14c(LU-13677 quota: qunit sorting doesn't work). 8704d14c was applied directly at 09f9fb32 as this patch solves a problem introduced in the main pool quotas patch.

			1 user from 5 clients made load with mdtest and ior. Pools configuration:
			global0: OST0000, OST0001
			global1: OST0002, OST0003

			max_dirty_mb was set to 10MB at each client
			lctl set_param osc.*.max_dirty_mb=10
			Quota limits:
			Each user: block hard limit 1T to enforce quota.
			Each user: block hard limit 200G per each pool.

	== Stress testing ==		== Stress testing ==

Sergey: /* Performance testing */

2020-10-26T18:10:26Z

Performance testing

← Older revision		Revision as of 11:10, 26 October 2020
Line 62:		Line 62:
	Each client had about 62GB of total memory.		Each client had about 62GB of total memory.
	== Performance testing ==		== Performance testing ==
	in ~~progress~~		As a baseline was used d0452cf - the patch before "LU-11023 quota: quota pools for OSTs"(09f9fb32).

			It was compared with 09f9fb32 + 8704d14c(LU-13677 quota: qunit sorting doesn't work). 8704d14c was applied directly at 09f9fb32 as this patch solves a problem introduced in the main pool quotas patch.

	== Stress testing ==		== Stress testing ==
	5 users from 5 clients make load with mdtest and ior.		5 users from 5 clients make load with mdtest and ior.

Sergey: /* Interoperability testing */

2020-10-15T17:53:52Z

Interoperability testing

← Older revision		Revision as of 10:53, 15 October 2020
Line 177:		Line 177:
	Test-Parameters: clientversion=2.10.8 testlist=sanity-quota clientdistro=el7.6		Test-Parameters: clientversion=2.10.8 testlist=sanity-quota clientdistro=el7.6
	Test-Parameters: serverversion=2.10.8 testlist=sanity-quota serverdistro=el7.6		Test-Parameters: serverversion=2.10.8 testlist=sanity-quota serverdistro=el7.6

			If server doesn't support PQ, newly added sanity-quota tests(1b,1c,1d,1e,1f,1g,3b,3c,67,68,69,71a,71b,72) are skipped.
			Only sanity-quota 70 is started even if server doesn't support QP. It checks that "old" server works correctly with a client that support QP.

	Below are links to results:		Below are links to results:

Sergey: /* Failover testing */

2020-10-15T12:48:03Z

Failover testing

← Older revision		Revision as of 05:48, 15 October 2020
Line 126:		Line 126:
	Except periodical failover/failback of servers. Depending on test "victim node" was only MDS, MDS and OSS or 2 OSS.		Except periodical failover/failback of servers. Depending on test "victim node" was only MDS, MDS and OSS or 2 OSS.

	During failover testing was faced [https://jira.whamcloud.com/browse/LU-14033 LU-14033]. It is FS corruption caused by IO failure at ldiskfs layer.		During failover testing was faced [https://jira.whamcloud.com/browse/LU-14033 LU-14033]. It is a FS corruption caused by IO failure at ldiskfs layer.
	Now there is no arguments that Pool Quotas may be connected with the issue. Furthermore, LU-14033 was also faced in a test where IOR and mdtest made load from user without quota limits(ha-ost-ior-mdtest).		Now there is no arguments that Pool Quotas may be connected with the issue. Furthermore, LU-14033 was also faced in a test where IOR and mdtest made load from a user without quota limits(ha-ost-ior-mdtest).

	=== ha-mds-ior-QP ===		=== ha-mds-ior-QP ===

Sergey: /* Interoperability testing */

2020-10-15T12:46:02Z

Interoperability testing

← Older revision		Revision as of 05:46, 15 October 2020
Line 172:		Line 172:

	== Interoperability testing ==		== Interoperability testing ==
	To check interoperability ~~were used~~ following test parameters:		To check interoperability was submitted https://review.whamcloud.com/#/c/40175/ with following test parameters:
	Test-Parameters: clientversion=2.12.3 testlist=sanity-quota		Test-Parameters: clientversion=2.12.3 testlist=sanity-quota clientdistro=el7.6
	Test-Parameters: serverversion=2.12.3 testlist=sanity-quota		Test-Parameters: serverversion=2.12.3 testlist=sanity-quota serverdistro=el7.6
	Test-Parameters: clientversion=2.10.8 testlist=sanity-quota clientdistro=el7.6		Test-Parameters: clientversion=2.10.8 testlist=sanity-quota clientdistro=el7.6
	Test-Parameters: serverversion=2.10.8 testlist=sanity-quota serverdistro=el7.6		Test-Parameters: serverversion=2.10.8 testlist=sanity-quota serverdistro=el7.6

	~~These parameters last were used at patchset 39 (https://review.whamcloud.com/#/c/39469/39).~~
	Below are links to results:		Below are links to results:
			server 2.13.56.7, client 2.10.8 - https://testing.whamcloud.com/test_sets/016a537c-cf20-4f57-a7b9-e13c7624438f
			server 2.13.56.7, client 2.12.3 - https://testing.whamcloud.com/test_sets/32dc544b-df98-4df0-9669-b87130166af4
			server 2.12.3, client 2.13.56.7 - https://testing.whamcloud.com/test_sets/15326174-13f4-4970-873c-dc830e1ea1d1
			server 2.10.8, client 2.13.56.7 - https://testing.whamcloud.com/test_sets/0b1c0133-2040-41f4-bc35-68a035aa60bd

			Similar testing was also performed during preparing patch for landing, see patch set 39(https://review.whamcloud.com/#/c/39469/39).

			No new issues was found during testing.

	== Compatibility testing ==		== Compatibility testing ==