http://wiki.lustre.org/index.php?title=Lustre_Tuning&feed=atom&action=historyLustre Tuning - Revision history2024-03-29T15:35:51ZRevision history for this page on the wikiMediaWiki 1.39.3http://wiki.lustre.org/index.php?title=Lustre_Tuning&diff=3688&oldid=prevAdilger: /* Number of Inodes for OST */2019-10-03T01:26:10Z<p><span dir="auto"><span class="autocomment">Number of Inodes for OST</span></span></p>
<table style="background-color: #fff; color: #202122;" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 18:26, 2 October 2019</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l101">Line 101:</td>
<td colspan="2" class="diff-lineno">Line 101:</td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>You can specify the number of inodes on the OST file systems via the ''-N<num_inodes>'' option to ''--mkfsoptions''. Alternately, if you know the average file size, then you can also specify the OST inode ratio for the OST file systems via ''-i <average_file_size / (number_of_stripes * 2)>''. For example, if the average file size is 80MB and there are by default 4 stripes per file then ''--mkfsoptions='-i 10485760''' would be appropriate (10MB per OST inode).</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>You can specify the number of inodes on the OST file systems via the ''-N<num_inodes>'' option to ''--mkfsoptions''. Alternately, if you know the average file size, then you can also specify the OST inode ratio for the OST file systems via ''-i <average_file_size / (number_of_stripes * 2)>''. For example, if the average file size is 80MB and there are by default 4 stripes per file then ''--mkfsoptions='-i 10485760''' would be appropriate (10MB per OST inode).</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;"></del></div></td><td colspan="2" class="diff-side-added"></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">[[Category: NeedsReview]]</del></div></td><td colspan="2" class="diff-side-added"></td></tr>
</table>Adilgerhttp://wiki.lustre.org/index.php?title=Lustre_Tuning&diff=3687&oldid=prevAdilger: /* Number of Inodes for OST */2019-10-03T01:25:56Z<p><span dir="auto"><span class="autocomment">Number of Inodes for OST</span></span></p>
<table style="background-color: #fff; color: #202122;" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 18:25, 2 October 2019</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l96">Line 96:</td>
<td colspan="2" class="diff-lineno">Line 96:</td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===Number of Inodes for OST===</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===Number of Inodes for OST===</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>For OST file systems, it is normally advantageous to take local file system usage into account. Try and minimize the number of inodes created on each OST, while keeping enough margin for potential variance in future usage. This helps in reducing the format and ''e2fsck'' time, and makes more space available for data. The current default is to create one inode per <del style="font-weight: bold; text-decoration: none;">16KB </del>of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>For OST file systems, it is normally advantageous to take local file system usage into account. Try and minimize the number of inodes created on each OST, while keeping enough margin for potential variance in future usage. This helps in reducing the format and ''e2fsck'' time, and makes more space available for data. The current default is to create one inode per <ins style="font-weight: bold; text-decoration: none;">64KB </ins>of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least <ins style="font-weight: bold; text-decoration: none;">twice as many inodes as expected to avoid running out of inodes before space</ins>:</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div> num_ost_inodes = <del style="font-weight: bold; text-decoration: none;">4 </del>* <num_mds_inodes> * <default_stripe_count> / <number_osts></div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div> num_ost_inodes = <ins style="font-weight: bold; text-decoration: none;">2 </ins>* <num_mds_inodes> * <default_stripe_count> / <number_osts></div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>You can specify the number of inodes on the OST file systems via the ''-N<num_inodes>'' option to ''--mkfsoptions''. Alternately, if you know the average file size, then you can also specify the OST inode <del style="font-weight: bold; text-decoration: none;">count </del>for the OST file systems via ''-i <average_file_size / (number_of_stripes * <del style="font-weight: bold; text-decoration: none;">4</del>)>''. <del style="font-weight: bold; text-decoration: none;">(</del>For example, if the average file size is <del style="font-weight: bold; text-decoration: none;">16MB </del>and there are by default 4 stripes per file then ''--mkfsoptions='-i <del style="font-weight: bold; text-decoration: none;">1048576</del>''' would be appropriate).</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>You can specify the number of inodes on the OST file systems via the ''-N<num_inodes>'' option to ''--mkfsoptions''. Alternately, if you know the average file size, then you can also specify the OST inode <ins style="font-weight: bold; text-decoration: none;">ratio </ins>for the OST file systems via ''-i <average_file_size / (number_of_stripes * <ins style="font-weight: bold; text-decoration: none;">2</ins>)>''. <ins style="font-weight: bold; text-decoration: none;"> </ins>For example, if the average file size is <ins style="font-weight: bold; text-decoration: none;">80MB </ins>and there are by default 4 stripes per file then ''--mkfsoptions='-i <ins style="font-weight: bold; text-decoration: none;">10485760</ins>''' would be appropriate <ins style="font-weight: bold; text-decoration: none;">(10MB per OST inode</ins>).</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category: NeedsReview]]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category: NeedsReview]]</div></td></tr>
<!-- diff cache key wiki_lustre_org:diff::1.12:old-3686:rev-3687 -->
</table>Adilgerhttp://wiki.lustre.org/index.php?title=Lustre_Tuning&diff=3686&oldid=prevAdilger: update for current defaults2019-10-03T00:48:41Z<p>update for current defaults</p>
<table style="background-color: #fff; color: #202122;" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 17:48, 2 October 2019</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1">Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">{| class='wikitable'</del></div></td><td colspan="2" class="diff-side-added"></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">|- </del></div></td><td colspan="2" class="diff-side-added"></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">!Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date.</del></div></td><td colspan="2" class="diff-side-added"></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">|}</del></div></td><td colspan="2" class="diff-side-added"></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;"><small>''(Updated: Feb 2010)''</small></del></div></td><td colspan="2" class="diff-side-added"></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>__TOC__</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>__TOC__</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Many options in <del style="font-weight: bold; text-decoration: none;">Lustre™ </del>are set by means of kernel module parameters. These parameters are contained in the ''modprobe.conf'' file (On SuSE, this may be ''modprobe.conf.local'').</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Many options in <ins style="font-weight: bold; text-decoration: none;">Lustre </ins>are set by means of kernel module parameters. These parameters are contained in the ''modprobe.conf'' file (On SuSE, this may be ''modprobe.conf.local'').</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==OSS Service Thread Count==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==OSS Service Thread Count==</div></td></tr>
<tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l13">Line 13:</td>
<td colspan="2" class="diff-lineno">Line 8:</td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div> options ost oss_num_threads={N}</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div> options ost oss_num_threads={N}</div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"> options ost oss_max_threads={N}</ins></div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>An OSS <del style="font-weight: bold; text-decoration: none;">can have </del>a maximum of 512 service threads and a minimum of 2 service</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>An OSS <ins style="font-weight: bold; text-decoration: none;">defaults to </ins>a maximum of 512 service threads and a minimum of 2 service</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>threads. The number of service threads is a function of how much RAM and how</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>threads. The number of service threads is a function of how much RAM and how</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>many CPUs are on each OSS node (1 thread / <del style="font-weight: bold; text-decoration: none;">128MB </del>* <del style="font-weight: bold; text-decoration: none;">num_cpus</del>). If the load on the</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>many CPUs are on each OSS node (1 thread / <ins style="font-weight: bold; text-decoration: none;">256MB </ins>* <ins style="font-weight: bold; text-decoration: none;">NUMA regions</ins>). If the load on the</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>OSS node is high, new service threads will be started in order to process more</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>OSS node is high, new service threads will be started in order to process more</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>requests concurrently, up to 4x the initial number of threads (subject to the</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>requests concurrently, up to 4x the initial number of threads (subject to the</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>maximum of 512). For <del style="font-weight: bold; text-decoration: none;">a 2GB 2</del>-<del style="font-weight: bold; text-decoration: none;">CPU system</del>, <del style="font-weight: bold; text-decoration: none;">the default thread count </del>is 32 <del style="font-weight: bold; text-decoration: none;">and the</del></div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>maximum of 512). <ins style="font-weight: bold; text-decoration: none;"> </ins>For <ins style="font-weight: bold; text-decoration: none;">disk</ins>-<ins style="font-weight: bold; text-decoration: none;">based OSTs</ins>, <ins style="font-weight: bold; text-decoration: none;">a good starting point </ins>is 32 <ins style="font-weight: bold; text-decoration: none;">threads per OST</ins>.</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">maximum thread count is 128</del>.</div></td><td colspan="2" class="diff-side-added"></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Increasing the size of the thread pool may help when:</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Increasing the size of the thread pool may help when:</div></td></tr>
<tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l55">Line 55:</td>
<td colspan="2" class="diff-lineno">Line 50:</td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div> options mds mds_num_threads={N}</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div> options mds mds_num_threads={N}</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">At this time, no testing has been done as to what the optimal number of MDS threads are. </del>The default number varies based on the server size up to a maximum of <del style="font-weight: bold; text-decoration: none;">32</del>. The maximum number of threads (''MDS_MAX_THREADS'') is <del style="font-weight: bold; text-decoration: none;">512</del>.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>The default number varies based on the server size up to a maximum of <ins style="font-weight: bold; text-decoration: none;">64</ins>. The maximum number of threads (''MDS_MAX_THREADS'') is <ins style="font-weight: bold; text-decoration: none;">1024</ins>.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>'''''Note:'''''</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>'''''Note:'''''</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>The OSS and MDS will automatically start new service threads dynamically in response to server loading <del style="font-weight: bold; text-decoration: none;">within a factor of 4. The default is calculated the same way as before</del>.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>The OSS and MDS will automatically start new service threads dynamically in response to server loading.</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Setting the ''<del style="font-weight: bold; text-decoration: none;">_mu_threads</del>'' module parameter disables the automatic thread creation behavior.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Setting the ''<ins style="font-weight: bold; text-decoration: none;">*_num_threads</ins>'' module parameter disables the automatic thread creation behavior.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==LNET Tunables==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==LNET Tunables==</div></td></tr>
<tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l78">Line 78:</td>
<td colspan="2" class="diff-lineno">Line 73:</td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==Planning for Inodes==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==Planning for Inodes==</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Every time you create a file on a Lustre file system, it consumes one inode on the MDS and one inode for each OST object that the file is striped over (normally it is based on the default stripe count option ''-c'', but this may change on a per-file basis). In ''<del style="font-weight: bold; text-decoration: none;">ext3/</del>ldiskfs'' file systems, inodes are pre-allocated, so creating a new file does not consume any of the free blocks. However, this also means that the format-time options should be conservative as it is not possible to increase the number of inodes after the file system is formatted. But it is possible to add OSTs with additional space and inodes to the file system.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Every time you create a file on a Lustre file system, it consumes one inode on the MDS and one inode for each OST object that the file is striped over (normally it is based on the default stripe count option ''-c'', but this may change on a per-file basis). In ''ldiskfs'' file systems, inodes are pre-allocated <ins style="font-weight: bold; text-decoration: none;">at format time</ins>, so creating a new file does not consume any of the free blocks. However, this also means that the format-time options should be conservative as it is not possible to increase the number of inodes after the file system is formatted. But it is possible to add OSTs with additional space and inodes to the file system.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>To be on the safe side, plan for <del style="font-weight: bold; text-decoration: none;">4KB </del>per inode on the MDS. This is the default. For the OST, the amount of space taken by each object depends entirely upon the usage pattern of the users/applications running on the system. Lustre, by necessity, defaults to a very conservative estimate for the object size (<del style="font-weight: bold; text-decoration: none;">16KB </del>per object). You can almost always increase this for file system installations. Many Lustre file systems have average file sizes over 1MB per object.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>To be on the safe side, plan for <ins style="font-weight: bold; text-decoration: none;">2.5KB </ins>per inode on the MDS. This is the default. For the OST, the amount of space taken by each object depends entirely upon the usage pattern of the users/applications running on the system. Lustre, by necessity, defaults to a very conservative estimate for the object size (<ins style="font-weight: bold; text-decoration: none;">64KB </ins>per object <ins style="font-weight: bold; text-decoration: none;">for OSTs below 10GiB in size, up to 1MB per object for OSTs over 16TiB in size</ins>). You can almost always increase this for file system installations. Many Lustre file systems have average file sizes over 1MB per object<ins style="font-weight: bold; text-decoration: none;">, and this can easily be calculated from existing filesystems by looking at the output of ''df'' and ''df -i'' dividing the total KB used by the total number of inodes used</ins>.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Sizing the MDT ==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Sizing the MDT ==</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>When calculating the MDS size, the only important factor is the average size of files to be stored in the file system. If the average file size is, for example, 5MB and you have 100TB of usable OST space, then you need at least ''100TB * 1024GB/TB * 1024MB/GB / 5MB/inode = 20 million inodes''. We recommend that you have twice the minimum, that is, 40 million inodes in this example. At the default <del style="font-weight: bold; text-decoration: none;">4KB </del>per inode, this works out to only <del style="font-weight: bold; text-decoration: none;">160GB </del>of space for the <del style="font-weight: bold; text-decoration: none;">MDS</del>.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>When calculating the MDS size, the only important factor is the average size of files to be stored in the file system. If the average file size is, for example, 5MB and you have 100TB of usable OST space, then you need at least ''100TB * 1024GB/TB * 1024MB/GB / 5MB/inode = 20 million inodes''. We recommend that you have twice the minimum, that is, 40 million inodes in this example. At the default <ins style="font-weight: bold; text-decoration: none;">2.5KiB of MDT space </ins>per inode, this works out to only <ins style="font-weight: bold; text-decoration: none;">96GB </ins>of space for the <ins style="font-weight: bold; text-decoration: none;">MDT</ins>.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Conversely, if you have a very small average file size, for example 4KB, Lustre is <del style="font-weight: bold; text-decoration: none;">not very </del>efficient. <del style="font-weight: bold; text-decoration: none;">This is because you consume as much space on </del>the <del style="font-weight: bold; text-decoration: none;">MDS as you are consuming on </del>the <del style="font-weight: bold; text-decoration: none;">OSTs</del>. This <del style="font-weight: bold; text-decoration: none;">is not a very common configuration for Lustre</del>.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Conversely, if you have a very small average file size, for example 4KB, Lustre is <ins style="font-weight: bold; text-decoration: none;">most </ins>efficient <ins style="font-weight: bold; text-decoration: none;">if storing the small files directly on the MDT using the [[Data_on_MDT]] feature</ins>. <ins style="font-weight: bold; text-decoration: none;"> In this case, the MDT(s) should be large enough to hold the data of </ins>the <ins style="font-weight: bold; text-decoration: none;">small files entirely, in addition to </ins>the <ins style="font-weight: bold; text-decoration: none;">normal inode data</ins>. <ins style="font-weight: bold; text-decoration: none;"> </ins>This <ins style="font-weight: bold; text-decoration: none;">would mean about 6.5KiB of MDT space per inode</ins>.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==Overriding Default Formatting Options==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==Overriding Default Formatting Options==</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>To override the default formatting options for any of the Lustre backing filesystems, use the ''--mkfsoptions='backing fs options''' argument to ''mkfs.lustre'' to pass formatting options to the backing ''mkfs''. For all options to format backing <del style="font-weight: bold; text-decoration: none;">''ext3'' and </del>''ldiskfs'' filesystems, see the ''mke2fs(8)'' man page; this section only discusses some Lustre-specific options.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>To override the default formatting options for any of the Lustre backing filesystems, use the ''--mkfsoptions='backing fs options''' argument to ''mkfs.lustre'' to pass formatting options to the backing ''mkfs''. For all options to format backing ''ldiskfs'' filesystems, see the ''mke2fs(8)'' man page; this section only discusses some Lustre-specific options<ins style="font-weight: bold; text-decoration: none;">. For ''ZFS'' filesystems, see the ''zpool(8)'' man page</ins>.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>===Number of Inodes for MDS===</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>===Number of Inodes for <ins style="font-weight: bold; text-decoration: none;">ldiskfs </ins>MDS===</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>To override the inode ratio, use the option ''-i <bytes per inode>'' <del style="font-weight: bold; text-decoration: none;">(for </del>instance, ''--mkfsoptions='-i 4096''' <del style="font-weight: bold; text-decoration: none;">to create </del>one inode per 4096 bytes of <del style="font-weight: bold; text-decoration: none;">file system </del>space<del style="font-weight: bold; text-decoration: none;">)</del>. Alternately, if you are specifying some absolute number of inodes, use the ''-N<number of inodes>'' option. You should not specify the ''-i'' option with an inode ratio below one inode per <del style="font-weight: bold; text-decoration: none;">1024 </del>bytes in order to avoid <del style="font-weight: bold; text-decoration: none;">unintentional mistakes</del>. Instead, use the ''-N'' option.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>To override the <ins style="font-weight: bold; text-decoration: none;">bytes-per-</ins>inode ratio<ins style="font-weight: bold; text-decoration: none;">, which affects the number of inodes created for a given MDT size</ins>, use the option ''-i <bytes per inode>''<ins style="font-weight: bold; text-decoration: none;">. For </ins>instance, ''--mkfsoptions='-i 4096''' <ins style="font-weight: bold; text-decoration: none;">creates </ins>one inode per 4096 bytes of <ins style="font-weight: bold; text-decoration: none;">MDT filesystem </ins>space. Alternately, if you are specifying some absolute number of inodes, use the ''-N<number of inodes>'' option. <ins style="font-weight: bold; text-decoration: none;"> </ins>You should not specify the ''-i'' option with an inode ratio below one inode per <ins style="font-weight: bold; text-decoration: none;">1536 </ins>bytes in order to avoid <ins style="font-weight: bold; text-decoration: none;">problems running out of space on the MDT before all of the inodes are allocated</ins>. Instead, use the ''-N'' option.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>A 2TB MDS by default will have <del style="font-weight: bold; text-decoration: none;">512M </del>inodes. <del style="font-weight: bold; text-decoration: none;">The largest currently-supported file system size is 8TB, which </del>would hold 2B inodes. With an MDS inode ratio of <del style="font-weight: bold; text-decoration: none;">1024 </del>bytes per inode, a <del style="font-weight: bold; text-decoration: none;">2TB </del>MDS would hold 2B inodes, and a <del style="font-weight: bold; text-decoration: none;">4TB </del>MDS would hold 4B inodes, which is the maximum number of inodes currently supported by <del style="font-weight: bold; text-decoration: none;">ext3</del>.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>A 2TB MDS by default will have <ins style="font-weight: bold; text-decoration: none;">400M </ins>inodes. <ins style="font-weight: bold; text-decoration: none;">A 10TB MDT </ins>would hold 2B inodes. With an MDS inode ratio of <ins style="font-weight: bold; text-decoration: none;">1536 </ins>bytes per inode, a <ins style="font-weight: bold; text-decoration: none;">3TB </ins>MDS would hold 2B inodes, and a <ins style="font-weight: bold; text-decoration: none;">6TB </ins>MDS would hold 4B inodes, which is the maximum number of inodes currently supported by <ins style="font-weight: bold; text-decoration: none;">ldiskfs</ins>.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===Inode Size for MDS===</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===Inode Size for MDS===</div></td></tr>
<tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Lustre uses "large" inodes on the backing file systems in order to efficiently store Lustre metadata with each file. On the MDS, each inode is at least <del style="font-weight: bold; text-decoration: none;">512 </del>bytes in size by default, while on the OST each inode is <del style="font-weight: bold; text-decoration: none;">256 </del>bytes in size. Lustre (or more specifically the backing ''<del style="font-weight: bold; text-decoration: none;">ext3</del>'' file system), also needs sufficient space left for other metadata like the journal (up to <del style="font-weight: bold; text-decoration: none;">400MB</del>), bitmaps and directories. There are also a few regular files that Lustre uses to maintain cluster consistency.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Lustre uses "large" inodes on the backing file systems in order to efficiently store Lustre metadata with each file. On the MDS, each inode is at least <ins style="font-weight: bold; text-decoration: none;">1024 </ins>bytes in size by default, while on the OST each inode is <ins style="font-weight: bold; text-decoration: none;">512 </ins>bytes in size. Lustre (or more specifically the backing ''<ins style="font-weight: bold; text-decoration: none;">ldiskfs</ins>'' file system), also needs sufficient space left for other metadata like the journal (up to <ins style="font-weight: bold; text-decoration: none;">4GB</ins>), bitmaps and directories. There are also a few regular files that Lustre uses to maintain cluster consistency.</div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>To specify a larger inode size, use the ''-I <inodesize>'' option. We do NOT recommend specifying a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>To specify a larger inode size, use the ''-I <inodesize>'' option. We do NOT recommend specifying a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.</div></td></tr>
</table>Adilgerhttp://wiki.lustre.org/index.php?title=Lustre_Tuning&diff=1963&oldid=prevKenRawlings: initial page creation as part of accelerated wiki migration2016-10-13T14:22:06Z<p>initial page creation as part of accelerated wiki migration</p>
<p><b>New page</b></p><div>{| class='wikitable'<br />
|- <br />
!Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date.<br />
|}<br />
<br />
<small>''(Updated: Feb 2010)''</small><br />
__TOC__<br />
Many options in Lustre™ are set by means of kernel module parameters. These parameters are contained in the ''modprobe.conf'' file (On SuSE, this may be ''modprobe.conf.local'').<br />
<br />
==OSS Service Thread Count==<br />
The ''oss_num_threads'' parameter allows the number of OST service threads to be<br />
specified at module load time on the OSS nodes:<br />
<br />
options ost oss_num_threads={N}<br />
<br />
An OSS can have a maximum of 512 service threads and a minimum of 2 service<br />
threads. The number of service threads is a function of how much RAM and how<br />
many CPUs are on each OSS node (1 thread / 128MB * num_cpus). If the load on the<br />
OSS node is high, new service threads will be started in order to process more<br />
requests concurrently, up to 4x the initial number of threads (subject to the<br />
maximum of 512). For a 2GB 2-CPU system, the default thread count is 32 and the<br />
maximum thread count is 128.<br />
<br />
Increasing the size of the thread pool may help when:<br />
* Several OSTs are exported from a single OSS<br />
* Back-end storage is running synchronously<br />
* I/O completions take excessive time due to slow storage<br />
<br />
Decreasing the size of the thread pool may help if:<br />
* The clients are overwhelming the storage capacity<br />
* There are lots of "slow I/O" or similar messages<br />
<br />
Increasing the number of I/O threads allows the kernel and storage to<br />
aggregate many writes together for more efficient disk I/O. The OSS thread pool is<br />
shared—each thread allocates approximately 1.5 MB (maximum RPC size + 0.5 MB)<br />
for internal I/O buffers.<br />
<br />
It is very important to consider memory consumption when increasing the thread<br />
pool size. Drives are only able to sustain a certain amount of parallel I/O activity<br />
before performance is degraded due to the high number of seeks and the OST<br />
threads just waiting for I/O. In this situation, it may be advisable to decrease the<br />
load by decreasing the number of OST threads.<br />
<br />
Determining the optimum number of OST threads is a process of trial and error. You<br />
may want to start with a number of OST threads equal to the number of actual disk<br />
spindles on the node. If you use RAID, subtract any dead spindles not used for<br />
actual data (e.g., 1 of N of spindles for RAID5, 2 of N spindles for RAID6), and<br />
monitor the performance of clients during usual workloads. If performance is degraded,<br />
increase the thread count and see how that works until performance is degraded again<br />
or you reach satisfactory performance.<br />
<br />
==MDS Service Thread Count==<br />
There is a similar parameter for the number of MDS service threads:<br />
<br />
options mds mds_num_threads={N}<br />
<br />
At this time, no testing has been done as to what the optimal number of MDS threads are. The default number varies based on the server size up to a maximum of 32. The maximum number of threads (''MDS_MAX_THREADS'') is 512.<br />
<br />
'''''Note:'''''<br />
The OSS and MDS will automatically start new service threads dynamically in response to server loading within a factor of 4. The default is calculated the same way as before.<br />
Setting the ''_mu_threads'' module parameter disables the automatic thread creation behavior.<br />
<br />
==LNET Tunables==<br />
''Transmit and receive buffer size:''<br />
With Lustre release 1.4.7 and later, ''ksocklnd'' now has separate parameters for the transmit and receive buffers.<br />
<br />
options ksocklnd tx_buffer_size=0 rx_buffer_size=0<br />
<br />
If these parameters are left at the default (0), the system automatically tunes the transmit and receive buffer size. In almost every case, the defaults produce the best performance. Do not attempt to tune this unless you are a network expert.<br />
<br />
''irq_affinity:''<br />
By default, this parameter is on. In the normal case on an SMP system, we would like our network traffic to remain local to a single CPU. This helps to keep the processor cache warm and minimizes the impact of context switches. This is especially helpful when an SMP system has more than one network interface and ideal when the number of interfaces equals the number of CPUs.<br />
<br />
If you have an SMP platform with a single fast interface such as 10GB Ethernet and more than two CPUs, you may see performance improve by turning this parameter off, as always test to compare the impact.<br />
<br />
=Options for Formatting MDS and OST=<br />
The backing file systems on the MDS and OSTs are independent of each other, so the formatting parameters for them should not be same. The size of the MDS backing file system depends solely on how many inodes you want in the total Lustre file system. It is not related to the size of the aggregate OST space.<br />
<br />
==Planning for Inodes==<br />
Every time you create a file on a Lustre file system, it consumes one inode on the MDS and one inode for each OST object that the file is striped over (normally it is based on the default stripe count option ''-c'', but this may change on a per-file basis). In ''ext3/ldiskfs'' file systems, inodes are pre-allocated, so creating a new file does not consume any of the free blocks. However, this also means that the format-time options should be conservative as it is not possible to increase the number of inodes after the file system is formatted. But it is possible to add OSTs with additional space and inodes to the file system.<br />
<br />
To be on the safe side, plan for 4KB per inode on the MDS. This is the default. For the OST, the amount of space taken by each object depends entirely upon the usage pattern of the users/applications running on the system. Lustre, by necessity, defaults to a very conservative estimate for the object size (16KB per object). You can almost always increase this for file system installations. Many Lustre file systems have average file sizes over 1MB per object.<br />
<br />
== Sizing the MDT ==<br />
When calculating the MDS size, the only important factor is the average size of files to be stored in the file system. If the average file size is, for example, 5MB and you have 100TB of usable OST space, then you need at least ''100TB * 1024GB/TB * 1024MB/GB / 5MB/inode = 20 million inodes''. We recommend that you have twice the minimum, that is, 40 million inodes in this example. At the default 4KB per inode, this works out to only 160GB of space for the MDS.<br />
<br />
Conversely, if you have a very small average file size, for example 4KB, Lustre is not very efficient. This is because you consume as much space on the MDS as you are consuming on the OSTs. This is not a very common configuration for Lustre.<br />
<br />
==Overriding Default Formatting Options==<br />
To override the default formatting options for any of the Lustre backing filesystems, use the ''--mkfsoptions='backing fs options''' argument to ''mkfs.lustre'' to pass formatting options to the backing ''mkfs''. For all options to format backing ''ext3'' and ''ldiskfs'' filesystems, see the ''mke2fs(8)'' man page; this section only discusses some Lustre-specific options.<br />
<br />
===Number of Inodes for MDS===<br />
To override the inode ratio, use the option ''-i <bytes per inode>'' (for instance, ''--mkfsoptions='-i 4096''' to create one inode per 4096 bytes of file system space). Alternately, if you are specifying some absolute number of inodes, use the ''-N<number of inodes>'' option. You should not specify the ''-i'' option with an inode ratio below one inode per 1024 bytes in order to avoid unintentional mistakes. Instead, use the ''-N'' option.<br />
<br />
A 2TB MDS by default will have 512M inodes. The largest currently-supported file system size is 8TB, which would hold 2B inodes. With an MDS inode ratio of 1024 bytes per inode, a 2TB MDS would hold 2B inodes, and a 4TB MDS would hold 4B inodes, which is the maximum number of inodes currently supported by ext3.<br />
<br />
===Inode Size for MDS===<br />
Lustre uses "large" inodes on the backing file systems in order to efficiently store Lustre metadata with each file. On the MDS, each inode is at least 512 bytes in size by default, while on the OST each inode is 256 bytes in size. Lustre (or more specifically the backing ''ext3'' file system), also needs sufficient space left for other metadata like the journal (up to 400MB), bitmaps and directories. There are also a few regular files that Lustre uses to maintain cluster consistency.<br />
<br />
To specify a larger inode size, use the ''-I <inodesize>'' option. We do NOT recommend specifying a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.<br />
<br />
===Number of Inodes for OST===<br />
For OST file systems, it is normally advantageous to take local file system usage into account. Try and minimize the number of inodes created on each OST, while keeping enough margin for potential variance in future usage. This helps in reducing the format and ''e2fsck'' time, and makes more space available for data. The current default is to create one inode per 16KB of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:<br />
<br />
num_ost_inodes = 4 * <num_mds_inodes> * <default_stripe_count> / <number_osts><br />
<br />
You can specify the number of inodes on the OST file systems via the ''-N<num_inodes>'' option to ''--mkfsoptions''. Alternately, if you know the average file size, then you can also specify the OST inode count for the OST file systems via ''-i <average_file_size / (number_of_stripes * 4)>''. (For example, if the average file size is 16MB and there are by default 4 stripes per file then ''--mkfsoptions='-i 1048576''' would be appropriate).<br />
<br />
[[Category: NeedsReview]]</div>KenRawlings