forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
45 lines
4.5 KiB
HTML
45 lines
4.5 KiB
HTML
<a name="mrs_01_0847"></a><a name="mrs_01_0847"></a>
|
|
|
|
<h1 class="topictitle1">AM Optimization for Big Tasks</h1>
|
|
<div id="body1590130630211"><div class="section" id="mrs_01_0847__sad1e4f844e2f4636a8fcaa459f68b7bf"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0847__ab6177157edc14464b299ed2107d91d2a">A big job containing 100,000 Map tasks fails. It is found that the failure is triggered by the slow response of ApplicationMaster (AM).</p>
|
|
</div>
|
|
<p id="mrs_01_0847__a3075ee5941f6452d9a36e18b7870c511">When the number of tasks increases, the number of objects managed by the AM increases, which requires much more memory for management. The default memory heap for AM is 1 GB.</p>
|
|
<div class="section" id="mrs_01_0847__s3db30605cd8c4e038d7c2fece830cf20"><h4 class="sectiontitle">Procedure</h4><p id="mrs_01_0847__a259be26b03e04b20bd7330fa11029199">You can improve the AM performance by setting the following parameters.</p>
|
|
<p id="mrs_01_0847__a961742942d4d470485a8230b4ac5a0b5">Navigation path for setting parameters:</p>
|
|
<p id="mrs_01_0847__p9481953124">Adjust the following parameters in the <strong id="mrs_01_0847__b1517554614380">mapred-site.xml</strong> configuration file on the client to adjust the following parameters: The <strong id="mrs_01_0847__b16884305395">mapred-site.xml</strong> configuration file is in the <strong id="mrs_01_0847__b794352120394">conf</strong> directory of the client installation path, for example, <strong id="mrs_01_0847__b1985720392">/opt/</strong><strong id="mrs_01_0847__b169812711399"></strong><strong id="mrs_01_0847__b20983715395">client/Yarn/config</strong>.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0847__t81f211b2f40b4e79b7b48ec1a58d8f66" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_0847__rda63c82ae8cf472a89a34e8bb38b6143"><th align="left" class="cellrowborder" valign="top" width="25.31%" id="mcps1.3.3.5.1.4.1.1"><p id="mrs_01_0847__adf61c38a2f8d4d59b9945f47c37b78db"><strong id="mrs_01_0847__b1084912018101">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="28.26%" id="mcps1.3.3.5.1.4.1.2"><p id="mrs_01_0847__a017c25f5bf4a480182b8d292aebc1349"><strong id="mrs_01_0847__b196904113102">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="46.43%" id="mcps1.3.3.5.1.4.1.3"><p id="mrs_01_0847__aa56d98ac3d524dd19c2966380c1afe19"><strong id="mrs_01_0847__b104861925103">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0847__rf232fd61ec434e30a60b170c660c9511"><td class="cellrowborder" valign="top" width="25.31%" headers="mcps1.3.3.5.1.4.1.1 "><p id="mrs_01_0847__a240308eb9c584ab99d992a3bdea2a901">yarn.app.mapreduce.am.resource.mb</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.26%" headers="mcps1.3.3.5.1.4.1.2 "><p id="mrs_01_0847__aa2eba3898a614bedb3d8c976f454fd6b">This parameter must be greater than the heap size specified by <strong id="mrs_01_0847__b16558553181016">yarn.app.mapreduce.am.command-opts</strong>. Unit: MB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.43%" headers="mcps1.3.3.5.1.4.1.3 "><p id="mrs_01_0847__ae29298e871fb49688edc73f9eb111457">1536</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0847__rd035be4617ae4d759a07ee9e3c09e3cb"><td class="cellrowborder" valign="top" width="25.31%" headers="mcps1.3.3.5.1.4.1.1 "><p id="mrs_01_0847__a071848657e2b45a6935c9a9bdb116751">yarn.app.mapreduce.am.command-opts</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.26%" headers="mcps1.3.3.5.1.4.1.2 "><p id="mrs_01_0847__a488e4822d7334a1a8e5a421a4f47d91b">Indicates the JVM startup parameters loaded to MapReduce ApplicationMaster.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.43%" headers="mcps1.3.3.5.1.4.1.3 "><p id="mrs_01_0847__p546561016178">For versions earlier than MRS 3.x: -Xmx1024m -XX:CMSFullGCsBeforeCompaction=1 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -verbose:gc</p>
|
|
<p id="mrs_01_0847__p840312176179">MRS 3.x or later: -Xmx1024m -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -verbose:gc -Djava.security.krb5.conf=${KRB5_CONFIG} -Dhadoop.home.dir=<em id="mrs_01_0847__i797567113511">${BIGDATA_HOME}</em>/FusionInsight_HD_<em id="mrs_01_0847__i049416228339">xxx</em>/install/FusionInsight-Hadoop-<em id="mrs_01_0847__i462913318338">xxx</em>/hadoop</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0843.html">MapReduce Performance Tuning</a></div>
|
|
</div>
|
|
</div>
|
|
|