forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
73 lines
8.4 KiB
HTML
73 lines
8.4 KiB
HTML
<a name="mrs_01_1688"></a><a name="mrs_01_1688"></a>
|
|
|
|
<h1 class="topictitle1">Improving Read Performance Using Client Metadata Cache</h1>
|
|
<div id="body1595904097004"><div class="section" id="mrs_01_1688__sc681d92b08594c59aaee44cbf13a08e3"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1688__a43fb0a987f5c4296aefe8189ed2bf423">Improve the HDFS read performance by using the client to cache the metadata for block locations.</p>
|
|
<div class="note" id="mrs_01_1688__n6ff0735c7d8e4079b9b15643d1de8aba"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1688__a036268bd21b048119f6795d4331ab425">This function is recommended only for reading files that are not modified frequently. Because the data modification done on the server side by some other client is invisible to the cache client, which may cause the metadata obtained from the cache to be outdated.</p>
|
|
<p id="mrs_01_1688__p1520492017202">This section applies to MRS 3.<em id="mrs_01_1688__i4586154017527">x</em> or later.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1688__sfc9a344b9e964a9c9be32aa544b65fe5"><h4 class="sectiontitle">Procedure</h4><p id="mrs_01_1688__a2471556f8ea0443387c3488b9e9888b3"><strong id="mrs_01_1688__b885594295213">Navigation path for setting parameters:</strong></p>
|
|
<p id="mrs_01_1688__ae2bcd657ca8046679656a82f7e91b2a1">On FusionInsight Manager, choose <span id="mrs_01_1688__text79112367811"><strong id="mrs_01_1688__b1234434495213">Cluster</strong> > </span><em id="mrs_01_1688__i734504412529">Name of the desired cluster</em> > <strong id="mrs_01_1688__b834515447529">Services</strong> > <strong id="mrs_01_1688__b5346544205214">HDFS</strong> > <strong id="mrs_01_1688__b2346184495218">Configurations</strong>, select <span class="parmvalue" id="mrs_01_1688__parmvalue33466441522"><b>All Configurations</b></span>, and enter the parameter name in the search box.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1688__ta3729e896db44b35b9ed497c389bf514" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter configuration</caption><thead align="left"><tr id="mrs_01_1688__rcb217d3f809b4439977b1e36093690ae"><th align="left" class="cellrowborder" valign="top" width="27.51%" id="mcps1.3.2.4.2.4.1.1"><p id="mrs_01_1688__a64e320d5eace46d19b1119fbf7b54545">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="59.45%" id="mcps1.3.2.4.2.4.1.2"><p id="mrs_01_1688__a51cfb2486f204f868788f7b9ee69d5d9">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="13.04%" id="mcps1.3.2.4.2.4.1.3"><p id="mrs_01_1688__ab710d4350ef248c0bfb75fdc502c2d1a">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1688__r06606778e5d94b5a975a68840ba496e0"><td class="cellrowborder" valign="top" width="27.51%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_1688__a2426c82108544400af0f2f6c2f481dde">dfs.client.metadata.cache.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.45%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_1688__a73d8daac0718460bbc3d9530578a42dc">Enables or disables the client to cache the metadata for block locations. Set this parameter to <span class="parmvalue" id="mrs_01_1688__parmvalue58231321105314"><b>true</b></span> and use it along with the <span class="parmname" id="mrs_01_1688__parmname5828132155311"><b>dfs.client.metadata.cache.pattern</b></span> parameter to enable the cache.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="13.04%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_1688__a7445851a46d64ac2afe66a53cee5cd97">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1688__r4cf84db7801c4fbba62da49775e5705d"><td class="cellrowborder" valign="top" width="27.51%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_1688__a26953ae06802443ea93553466ea0cef1">dfs.client.metadata.cache.pattern</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.45%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_1688__ad3e98f366c4a4e8e9b3976108c9f53a3">Indicates the regular expression pattern of the path of the file to be cached. Only the metadata for block locations of these files is cached until the metadata expires. This parameter is valid only when <span class="parmname" id="mrs_01_1688__parmname15522126165317"><b>dfs.client.metadata.cache.enabled</b></span> is set to <span class="parmvalue" id="mrs_01_1688__parmvalue1552292645317"><b>true</b></span>.</p>
|
|
<p id="mrs_01_1688__a35abf68754a349e1b2fd26acbf90c423">Example: <span class="filepath" id="mrs_01_1688__filepath1936183610536"><b>/test.*</b></span> indicates that all files whose paths start with <strong id="mrs_01_1688__b15366133615534">/test</strong> are read.</p>
|
|
<div class="note" id="mrs_01_1688__n90bb6614cbbb42e0a4948f0ba552853a"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="mrs_01_1688__u08220e1eab904339b8a830efa390a356"><li id="mrs_01_1688__l047f5af3aeb9406d9abda11f491184e3">To ensure consistency, configure a specific mode to cache only files that are not frequently modified by other clients.</li></ul>
|
|
<ul id="mrs_01_1688__ufd39a3d1469a4893bb1e6c715aa449ac"><li id="mrs_01_1688__la04b5236a0c14b3ea0bdfb54cc1db0c8">The regular expression pattern verifies only the path of the URI, but not the schema and authority in the case of the Fully Qualified path.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="13.04%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_1688__p20755533249">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1688__r6b5d560a21c24540ad9c439fd6cdbe2d"><td class="cellrowborder" valign="top" width="27.51%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_1688__a0e4eca6a7c5b43b4a21a46abed3fc750">dfs.client.metadata.cache.expiry.sec</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.45%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_1688__a6ef95dccb8424b2ebae62ee11c8f85bc">Indicates the duration for caching metadata. The cache entry becomes invalid after its caching time exceeds this duration. Even metadata that is frequently used during the caching process can become invalid.</p>
|
|
<p id="mrs_01_1688__a8da04b1eae88443993c4037511869a0c">Time suffixes <strong id="mrs_01_1688__b194924585319">s</strong>/<strong id="mrs_01_1688__b1854204555316">m</strong>/<strong id="mrs_01_1688__b355845105311">h</strong> can be used to indicate second, minute, and hour, respectively.</p>
|
|
<div class="note" id="mrs_01_1688__neee2ba73a1f94283bd6c234dae2ec085"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1688__a0caf1656808b48fb8d6538d9e061ad78">If this parameter is set to <span class="parmvalue" id="mrs_01_1688__parmvalue10521446185316"><b>0s</b></span>, the cache function is disabled.</p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="13.04%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_1688__a69d6f15c00f74af49a80445dc7c0f663">60s</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1688__r69c1069200ae44f8a3aa3aae611b9535"><td class="cellrowborder" valign="top" width="27.51%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_1688__affa346519ffb45f396694aaa9f08b329">dfs.client.metadata.cache.max.entries</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.45%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_1688__acc3693f367b6431fbc86670d8746bcee">Indicates the maximum number of non-expired data items that can be cached at a time.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="13.04%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_1688__afb24a42ddc424198b0efb30d59482b9a">65536</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_1688__n03fb443136e6441698042999a0d11407"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1688__a274eee6b2d554cfcbcab6fb4b8f34311">Call <em id="mrs_01_1688__i5148145045319">DFSClient#clearLocatedBlockCache()</em> to completely clear the client cache before it expires.</p>
|
|
<p id="mrs_01_1688__a2aa06e72d4e74efc90d61abb8ff14ba2">The sample usage is as follows:</p>
|
|
<pre class="screen" id="mrs_01_1688__s17cfe52b69c545cd85ef841a65a9fa6d"> FileSystem fs = FileSystem.get(conf);
|
|
DistributedFileSystem dfs = (DistributedFileSystem) fs;
|
|
DFSClient dfsClient = dfs.getClient();
|
|
<strong id="mrs_01_1688__a5dc9526253384dd6bd307accde513e5a">dfsClient.clearLocatedBlockCache();</strong></pre>
|
|
</div></div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0829.html">HDFS Performance Tuning</a></div>
|
|
</div>
|
|
</div>
|
|
|