Lustre Monitoring and Statistics Guide - Revision history

Adilger: /* Jobstats: Finding jobs doing I/O over watermark */ add lljobstats and glljobstats

2023-12-21T08:27:43Z

Jobstats: Finding jobs doing I/O over watermark: add lljobstats and glljobstats

← Older revision		Revision as of 01:27, 21 December 2023
Line 246:		Line 246:
	==== Jobstats: Finding jobs doing I/O over watermark ====		==== Jobstats: Finding jobs doing I/O over watermark ====

	The ~~```~~lljobstat~~```~~ script can be used to collect and filter current output created by jobstats from the local Lustre targets. It ~~is useful to check for~~ jobs doing the most RPCs on the current server. This is included in Lustre 2.16.0 and later, but is a standalone Python script and could be copied to older installations as needed.		The <code>lljobstat</code> script can be used to collect and filter current output created by jobstats from the local Lustre targets. It shows the jobs doing the most RPCs on the current server, and is useful for quickly debugging the source of a high IO/RPC load on a server, similar to <code>top</code>. This is included in Lustre 2.16.0 and later, but is a standalone Python script and could be copied to older installations as needed.

			A more advanced version of <code>lljobstat</code> is <code>glljobstat</code>, the [https://github.com/DDNeu/global-lustre-jobstats Global Lustre Jobstats utility]. This allows monitoring the job stats from all of the servers, but needs a more complex configuration to run.

	==== Jobstats: A lightweight solution to provide I/O statistics to users ====		==== Jobstats: A lightweight solution to provide I/O statistics to users ====

Adilger: /* Jobstats: Finding jobs doing I/O over watermark */

2023-12-21T08:21:11Z

Jobstats: Finding jobs doing I/O over watermark

← Older revision		Revision as of 01:21, 21 December 2023
Line 246:		Line 246:
	==== Jobstats: Finding jobs doing I/O over watermark ====		==== Jobstats: Finding jobs doing I/O over watermark ====

	The ~~Perl~~ script ~~show_high_jobstats.pl~~ can be used to collect and filter current output created by jobstats from Lustre ~~servers~~. It is useful to check for jobs doing ~~I/O over a high watermark~~. This ~~tool was shortly mentioned during Roland Laifer's talk at LAD'15, see references~~. ~~You can download it here:~~		The ```lljobstat``` script can be used to collect and filter current output created by jobstats from the local Lustre targets. It is useful to check for jobs doing the most RPCs on the current server. This is included in Lustre 2.16.0 and later, but is a standalone Python script and could be copied to older installations as needed.
	~~https://www~~.~~scc.kit.edu/scc/sw/lustre_tools/show_high_jobstats~~.~~tgz~~

	==== Jobstats: A lightweight solution to provide I/O statistics to users ====		==== Jobstats: A lightweight solution to provide I/O statistics to users ====

Sknolin at 15:03, 6 October 2022

2022-10-06T15:03:35Z

← Older revision		Revision as of 08:03, 6 October 2022
Line 231:		Line 231:

	Unsure if the source for that plugin is available.		Unsure if the source for that plugin is available.

			==== Prometheus Exporters ====

			* https://github.com/HewlettPackard/lustre_exporter - this appears inactive (10/2022)
			* https://github.com/GSI-HPC/lustre_exporter - branch of HP exporter, more recent changes
			* https://github.com/whamcloud/lustrefs-exporter - reinventing things with Rust instead of Go?

	==== A Note about Jobstats ====		==== A Note about Jobstats ====

Sknolin: No need for my name on this any more I think, as people have added some things.

2022-10-06T14:59:50Z

No need for my name on this any more I think, as people have added some things.

← Older revision		Revision as of 07:59, 6 October 2022
Line 1:		Line 1:
	== Introduction ==		== Introduction ==

	~~This guide is by Scott Nolin (scott.nolin@ssec.wisc.edu), of the University of Wisconsin Space Science and Engineering Center.~~

	There are a variety of useful statistics and counters available on Lustre servers and clients. This is an attempt to detail some of these statistics and methods for collecting and working with them.		There are a variety of useful statistics and counters available on Lustre servers and clients. This is an attempt to detail some of these statistics and methods for collecting and working with them.

Jgrund at 01:34, 19 August 2019

2019-08-19T01:34:31Z

← Older revision		Revision as of 18:34, 18 August 2019
Line 157:		Line 157:
	*LMT - provides 'top' style monitoring of server nodes, and historical data via mysql. https://github.com/chaos/lmt		*LMT - provides 'top' style monitoring of server nodes, and historical data via mysql. https://github.com/chaos/lmt
	*lltop and xltop - monitoring with batch scheduler integration. Newer Lustre versions with jobstats likely provide similar data very conveniently, but these are still very good for examples of working with monitoring data. https://github.com/jhammond/lltop https://github.com/jhammond/xltop		*lltop and xltop - monitoring with batch scheduler integration. Newer Lustre versions with jobstats likely provide similar data very conveniently, but these are still very good for examples of working with monitoring data. https://github.com/jhammond/lltop https://github.com/jhammond/xltop
	Integrated Manager for Lustre - With version 4.0, IML is FOSS software. This can be installed in a monitoring only mode. https://github.com/whamcloud/integrated-manager-for-lustre		*Integrated Manager for Lustre - With version 4.0, IML is FOSS software. This can be installed in a monitoring only mode. https://github.com/whamcloud/integrated-manager-for-lustre

	=== Build it Yourself ===		=== Build it Yourself ===

Jgrund at 00:05, 19 August 2019

2019-08-19T00:05:22Z

← Older revision		Revision as of 17:05, 18 August 2019
Line 157:		Line 157:
	*LMT - provides 'top' style monitoring of server nodes, and historical data via mysql. https://github.com/chaos/lmt		*LMT - provides 'top' style monitoring of server nodes, and historical data via mysql. https://github.com/chaos/lmt
	*lltop and xltop - monitoring with batch scheduler integration. Newer Lustre versions with jobstats likely provide similar data very conveniently, but these are still very good for examples of working with monitoring data. https://github.com/jhammond/lltop https://github.com/jhammond/xltop		*lltop and xltop - monitoring with batch scheduler integration. Newer Lustre versions with jobstats likely provide similar data very conveniently, but these are still very good for examples of working with monitoring data. https://github.com/jhammond/lltop https://github.com/jhammond/xltop
	~~Intel~~ Manager for Lustre - With version 4.0, IML is FOSS software. This can be installed in a monitoring only mode. https://github.com/~~intel~~-~~hpdd/Online~~-~~Help~~		Integrated Manager for Lustre - With version 4.0, IML is FOSS software. This can be installed in a monitoring only mode. https://github.com/whamcloud/integrated-manager-for-lustre

	=== Build it Yourself ===		=== Build it Yourself ===

Sknolin: note that metrics work in 2.10

2018-11-15T23:44:55Z

note that metrics work in 2.10

← Older revision		Revision as of 16:44, 15 November 2018
Line 15:		Line 15:
	== Lustre Versions ==		== Lustre Versions ==

	This information is based on working ~~primarily~~ with Lustre 2.4 and 2.5.		This information was originally based on working with Lustre 2.4 and 2.5. The same metrics are available in 2.10.

	== Reading /proc vs lctl ==		== Reading /proc vs lctl ==

JEThornsberry: Documented actual format for job_stats data

2018-10-01T21:36:44Z

Documented actual format for job_stats data

← Older revision		Revision as of 14:36, 1 October 2018
Line 67:		Line 67:
	=== Jobstats ===		=== Jobstats ===

	Jobstats are slightly more complex multi-line records. They ~~look~~ like JSON, except for the (-) blocks for each job. Each OST or MDT also has an entry for each jobid (or procname_uid perhaps), and then the data.		Jobstats are slightly more complex multi-line records. They are formatted in YAML, which looks like JSON, except for the (-) blocks for each job. Each OST or MDT also has an entry for each jobid (or procname_uid perhaps), and then the data.

	Example:		Example:

Sknolin: /* Open Source Monitoring Packages */

2017-11-06T19:01:59Z

Open Source Monitoring Packages

← Older revision		Revision as of 12:01, 6 November 2017
Line 157:		Line 157:
	*LMT - provides 'top' style monitoring of server nodes, and historical data via mysql. https://github.com/chaos/lmt		*LMT - provides 'top' style monitoring of server nodes, and historical data via mysql. https://github.com/chaos/lmt
	*lltop and xltop - monitoring with batch scheduler integration. Newer Lustre versions with jobstats likely provide similar data very conveniently, but these are still very good for examples of working with monitoring data. https://github.com/jhammond/lltop https://github.com/jhammond/xltop		*lltop and xltop - monitoring with batch scheduler integration. Newer Lustre versions with jobstats likely provide similar data very conveniently, but these are still very good for examples of working with monitoring data. https://github.com/jhammond/lltop https://github.com/jhammond/xltop
			Intel Manager for Lustre - With version 4.0, IML is FOSS software. This can be installed in a monitoring only mode. https://github.com/intel-hpdd/Online-Help

	=== Build it Yourself ===		=== Build it Yourself ===

Sknolin: /* check_mk and Graphite */

2017-01-31T16:45:45Z

check_mk and Graphite

← Older revision		Revision as of 09:45, 31 January 2017
Line 213:		Line 213:

	* Graphite - http://graphite.readthedocs.org/en/latest/		* Graphite - http://graphite.readthedocs.org/en/latest/
	* Lustrestats.pm - perl module to parse different types of lustre stats, used by lustrestats scripts		* http://www.ssec.wisc.edu/~scottn/files/Lustrestats.pm - perl module to parse different types of lustre stats, used by lustrestats scripts
	* OMD - check_mk, nagios, pnp4nagios		* OMD - check_mk, nagios, pnp4nagios
	* check_mk local scripts - these are called via check_mk, at whatever rate is desired. http://www.ssec.wisc.edu/~scottn/files/lustre_stats_mds.cmk http://www.ssec.wisc.edu/~scottn/files/lustre_stats_oss.cmk		* check_mk local scripts - these are called via check_mk, at whatever rate is desired. http://www.ssec.wisc.edu/~scottn/files/lustre_stats_mds.cmk http://www.ssec.wisc.edu/~scottn/files/lustre_stats_oss.cmk