http://wiki.lustre.org/index.php?action=history&feed=atom&title=Enhanced_Adaptive_Compression_in_Lustre 2026-11-12T17:57:15Z Revision history for this page on the wiki MediaWiki 1.39.7 http://wiki.lustre.org/index.php?title=Enhanced_Adaptive_Compression_in_Lustre&diff=5299&oldid=prev 2025-09-25T21:41:26Z

<p><span dir="auto"><span class="autocomment">Project Links: </span> fix URL</span></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 13:41, 25 September 2025</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l260">Line 260:</td> <td colspan="2" class="diff-lineno">Line 260:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://wr.informatik.uni-hamburg.de/research/projects/ipcc-l/start Uni Hamburg Scientific Computing Group]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://wr.informatik.uni-hamburg.de/research/projects/ipcc-l/start Uni Hamburg Scientific Computing Group]</div></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>   * [https://jira.whamcloud.com/browse/LU-10026 <del style="font-weight: bold; text-decoration: none;">Jira</del>]</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>   * [https://jira.whamcloud.com/browse/LU-10026 <ins style="font-weight: bold; text-decoration: none;">LU-10026 Client-side data compression</ins>]</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:Layout]]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:Layout]]</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:Architecture]]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>[[Category:Architecture]]</div></td></tr> <!-- diff cache key wiki_lustre_org:diff::1.12:old-5298:rev-5299 --> </table>

Adilger http://wiki.lustre.org/index.php?title=Enhanced_Adaptive_Compression_in_Lustre&diff=5298&oldid=prev 2025-09-25T21:40:40Z

<p>initial page creation</p> <a href="http://wiki.lustre.org/index.php?title=Enhanced_Adaptive_Compression_in_Lustre&amp;diff=5298&amp;oldid=3497">Show changes</a>

Adilger http://wiki.lustre.org/index.php?title=Enhanced_Adaptive_Compression_in_Lustre&diff=3497&oldid=prev 2019-03-07T10:43:34Z

<p></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 02:43, 7 March 2019</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l21">Line 21:</td> <td colspan="2" class="diff-lineno">Line 21:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://wr.informatik.uni-hamburg.de/research/projects/ipcc-l/start Scientific Computing Group]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://wr.informatik.uni-hamburg.de/research/projects/ipcc-l/start Scientific Computing Group]</div></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>   * [https://jira.whamcloud.com/browse/LU-10026]</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>   * [https://jira.whamcloud.com/browse/LU-10026 <ins style="font-weight: bold; text-decoration: none;">Jira</ins>]</div></td></tr> <!-- diff cache key wiki_lustre_org:diff::1.12:old-3496:rev-3497 --> </table>

Fuchs http://wiki.lustre.org/index.php?title=Enhanced_Adaptive_Compression_in_Lustre&diff=3496&oldid=prev 2019-03-07T10:43:01Z

<p></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 02:43, 7 March 2019</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l20">Line 20:</td> <td colspan="2" class="diff-lineno">Line 20:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===== Project Links =====</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===== Project Links =====</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">  * [https://software.intel.com/articles/intel-parallel-computing-center-at-university-of-hamburg-scientific-computing Intel]</del></div></td><td colspan="2" class="diff-side-added"></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://wr.informatik.uni-hamburg.de/research/projects/ipcc-l/start Scientific Computing Group]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://wr.informatik.uni-hamburg.de/research/projects/ipcc-l/start Scientific Computing Group]</div></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>   * [https://jira.<del style="font-weight: bold; text-decoration: none;">hpdd.intel</del>.com/browse/<del style="font-weight: bold; text-decoration: none;">LU-10026 </del>LU-10026]</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>   * [https://jira.<ins style="font-weight: bold; text-decoration: none;">whamcloud</ins>.com/browse/LU-10026]</div></td></tr> <!-- diff cache key wiki_lustre_org:diff::1.12:old-3090:rev-3496 --> </table>

Fuchs http://wiki.lustre.org/index.php?title=Enhanced_Adaptive_Compression_in_Lustre&diff=3090&oldid=prev 2018-03-06T17:47:35Z

<p></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 09:47, 6 March 2018</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l22">Line 22:</td> <td colspan="2" class="diff-lineno">Line 22:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://software.intel.com/articles/intel-parallel-computing-center-at-university-of-hamburg-scientific-computing Intel]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://software.intel.com/articles/intel-parallel-computing-center-at-university-of-hamburg-scientific-computing Intel]</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://wr.informatik.uni-hamburg.de/research/projects/ipcc-l/start Scientific Computing Group]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://wr.informatik.uni-hamburg.de/research/projects/ipcc-l/start Scientific Computing Group]</div></td></tr> <tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">  * [https://jira.hpdd.intel.com/browse/LU-10026 LU-10026]</ins></div></td></tr> <!-- diff cache key wiki_lustre_org:diff::1.12:old-2683:rev-3090 --> </table>

Nrutman http://wiki.lustre.org/index.php?title=Enhanced_Adaptive_Compression_in_Lustre&diff=2683&oldid=prev 2017-08-21T19:11:53Z

<p><span dir="auto"><span class="autocomment">Funding</span></span></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 11:11, 21 August 2017</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l16">Line 16:</td> <td colspan="2" class="diff-lineno">Line 16:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===== Funding =====</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===== Funding =====</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Intel Parallel Computing Center for Lustre “Enhanced Adaptive Compression in Lustre”</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Intel Parallel Computing Center for Lustre <ins style="font-weight: bold; text-decoration: none;">Universität Hamburg </ins>“Enhanced Adaptive Compression in Lustre”</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===== Project Links =====</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>===== Project Links =====</div></td></tr> <!-- diff cache key wiki_lustre_org:diff::1.12:old-1940:rev-2683 --> </table>

Fuchs http://wiki.lustre.org/index.php?title=Enhanced_Adaptive_Compression_in_Lustre&diff=1940&oldid=prev 2016-09-22T12:07:24Z

<p></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 04:07, 22 September 2016</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l21">Line 21:</td> <td colspan="2" class="diff-lineno">Line 21:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://software.intel.com/articles/intel-parallel-computing-center-at-university-of-hamburg-scientific-computing Intel]</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>   * [https://software.intel.com/articles/intel-parallel-computing-center-at-university-of-hamburg-scientific-computing Intel]</div></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>   * [<del style="font-weight: bold; text-decoration: none;">http</del>://wr.informatik.uni-hamburg.de/research/projects/ipcc-l <del style="font-weight: bold; text-decoration: none;">WR</del>]</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>   * [<ins style="font-weight: bold; text-decoration: none;">https</ins>://wr.informatik.uni-hamburg.de/research/projects/ipcc-l<ins style="font-weight: bold; text-decoration: none;">/start Scientific Computing Group</ins>]</div></td></tr> <!-- diff cache key wiki_lustre_org:diff::1.12:old-1807:rev-1940 --> </table>

Fuchs http://wiki.lustre.org/index.php?title=Enhanced_Adaptive_Compression_in_Lustre&diff=1807&oldid=prev 2016-06-29T14:01:58Z

<p>ipcc compression page</p> <p><b>New page</b></p><div>===== General Information =====<br /> <br /> Due to the increasing gap between computational speed, network speed and storage capacity, it has become necessary to investigate data reduction techniques.<br /> Storage systems have become a significant part of the total cost of ownership due to the increased amount of storage devices, their associated acquisition cost and energy consumption.<br /> <br /> Ultimately, we are aiming for compression support in Lustre at multiple levels:<br /> <br /> - Client-side compression allows using the available network and storage capacity more efficiently,<br /> - Client hints empower applications to provide information useful for compression and<br /> - Adaptive compression makes it possible to choose appropriate settings depending on performance metrics and projected benefits.<br /> <br /> Compression will be completely transparent to the applications because it will be performed by the client and/or server on their behalf.<br /> However, it will be possible for users to tune Lustre&#039;s behavior to obtain the best performance/compression/etc.<br /> When using client-side compression, the single stream performance bottleneck will directly benefit from the compression.<br /> <br /> ===== Funding =====<br /> <br /> Intel Parallel Computing Center for Lustre “Enhanced Adaptive Compression in Lustre”<br /> <br /> ===== Project Links =====<br /> <br /> * [https://software.intel.com/articles/intel-parallel-computing-center-at-university-of-hamburg-scientific-computing Intel]<br /> * [http://wr.informatik.uni-hamburg.de/research/projects/ipcc-l WR]</div>

Fuchs