forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
60 lines
9.9 KiB
HTML
60 lines
9.9 KiB
HTML
<a name="mrs_01_0839"></a><a name="mrs_01_0839"></a>
|
|
|
|
<h1 class="topictitle1">Configuring the Distributed Cache</h1>
|
|
<div id="body1590130629896"><div class="section" id="mrs_01_0839__s63b8d4d1a1784f7abf0fb575603ca962"><h4 class="sectiontitle">Scenarios</h4><div class="note" id="mrs_01_0839__note1286611300323"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0839__p1486614300321">This section applies to MRS 3.<em id="mrs_01_0839__i1216817511275">x</em> or later.</p>
|
|
</div></div>
|
|
<p id="mrs_01_0839__a343db8381ada4681a42990449ae2c510">Distributed caching is useful in the following scenarios:</p>
|
|
<p id="mrs_01_0839__a35755ed6f4dc4569a22ec8380175eaf0"><strong id="mrs_01_0839__adb8436b5e67f40e2ac04e1efd775d2ba">Rolling Upgrade</strong></p>
|
|
<p id="mrs_01_0839__a24a0072efb1e43b29a6fe4b37ea57f11">During the upgrade, applications must keep the text content (JAR file or configuration file) unchanged. The content is not based on Yarn of the current version, but on the version when it is submitted. This is a challenging issue. Generally, applications (such as MapReduce, Hive, and Tez) need to be installed locally. Libraries need to be installed on all cluster servers (clients and servers). When a rolling upgrade or downgrade starts in the cluster, the version of the locally installed library changes during application running. During the rolling upgrade, only a few NodeManagers are upgraded first. These NodeManagers obtain the software of the latest version. This leads to inconsistent behavior and can result in run-time errors.</p>
|
|
<p id="mrs_01_0839__aefe3495364894007b22d2ff9d45a181e"><strong id="mrs_01_0839__a780e8f198b524afcae41a6a426bb443d">Co-existence of Multiple Yarn Versions</strong></p>
|
|
<p id="mrs_01_0839__a3397ed5d588e4f96ae35b1758262937a">Cluster administrators may run tasks that use multiple versions of Yarn and Hadoop JARs in a cluster. However, this task is difficult to be implemented because the JARs have been localized and have only one version.</p>
|
|
<p id="mrs_01_0839__a6ca8d5286f254a91867530ed8dcc3355">The MapReduce application framework can be deployed through the distributed cache and does not depend on the static version copied during installation. Therefore, you can store multiple versions of Hadoop in HDFS and configure the <span class="filepath" id="mrs_01_0839__f44cde1ce0b5b4da1898f0b25af2560b9"><b>mapred-site.xml</b></span> file to specify the default version used by the task. You can run different versions of MapReduce by setting proper configuration attributes without using the versions deployed in the cluster.</p>
|
|
<div class="fignone" id="mrs_01_0839__ff6b35503342f4e29b9fc759caf375c43"><a name="mrs_01_0839__ff6b35503342f4e29b9fc759caf375c43"></a><a name="ff6b35503342f4e29b9fc759caf375c43"></a><span class="figcap"><b>Figure 1 </b>Clusters with NodeManagers and Applications of multiple versions</span><br><span><img id="mrs_01_0839__id3aaaa9ef58e4363bef53f28f13cc01f" src="en-us_image_0000001348770401.png"></span></div>
|
|
</div>
|
|
<p id="mrs_01_0839__abd5240fdd160483381f3f89685df297d">As shown in <a href="#mrs_01_0839__ff6b35503342f4e29b9fc759caf375c43">Figure 1</a>, the application can use Hadoop JARs in HDFS instead of the local version. Therefore, during the rolling upgrade, even if NodeManager has been upgraded, the application can still run Hadoop of the earlier version.</p>
|
|
<div class="section" id="mrs_01_0839__sd225fc31316c448e8429fc422bfda814"><h4 class="sectiontitle">Configuration Description</h4><ol id="mrs_01_0839__od60edfba9f13477693ed5a6bea722b7f"><li id="mrs_01_0839__l6b6eb9cc36cf42668d521a1cf15a13d4"><span>Save the MapReduce <strong id="mrs_01_0839__b28730217351930">.tar</strong> package of the specified version to a directory that can be accessed by applications in HDFS, as shown in the following command.</span><p><p id="mrs_01_0839__ad761ff8df73d4fcb88bd0b9e6d1f1159"><i><span class="varname" id="mrs_01_0839__v507b4fe120424b0183f458ddb28343c1">$HADOOP_HOME</span></i><strong id="mrs_01_0839__b37891743145911">/bin/hdfs dfs -put</strong> <i><span class="varname" id="mrs_01_0839__vaeaea216664244bab8f6558c6243ec14">hadoop-x.tar.gz</span></i> <i><span class="varname" id="mrs_01_0839__vab6d8a0a13f0404c8a7bb69ee99950c2">/mapred/framework/</span></i></p>
|
|
</p></li><li id="mrs_01_0839__l544aecfb485b4cba980fd8d015ea3b8e"><span>Set parameters in the <span class="filepath" id="mrs_01_0839__facd50f970ac8406bad3d857573758c6f"><b>mapred-site.xml</b></span> file based on <a href="#mrs_01_0839__tb0ddb31c45b94f19967571acd6bfc4e3">Table 1</a>.</span><p>
|
|
<div class="tablenoborder"><a name="mrs_01_0839__tb0ddb31c45b94f19967571acd6bfc4e3"></a><a name="tb0ddb31c45b94f19967571acd6bfc4e3"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0839__tb0ddb31c45b94f19967571acd6bfc4e3" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Distributed cache parameters</caption><thead align="left"><tr id="mrs_01_0839__r8e0872dcc3184635b57419637041186e"><th align="left" class="cellrowborder" valign="top" width="18.3%" id="mcps1.3.3.2.2.2.1.2.4.1.1"><p id="mrs_01_0839__adc719298cd5744ad90cfb9cc66ce19a3"><strong id="mrs_01_0839__ae3859414cf9649f6a73fe10135ae1f4f">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="69.64%" id="mcps1.3.3.2.2.2.1.2.4.1.2"><p id="mrs_01_0839__ae7576dfcd8ff4c1e937cc221c1030cfa"><strong id="mrs_01_0839__ae41de99cce014167b10c1476b7131740">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="12.06%" id="mcps1.3.3.2.2.2.1.2.4.1.3"><p id="mrs_01_0839__a2a4d2b77767f4400acb1d05799f5288e"><strong id="mrs_01_0839__ad38c116c37bf4e378b8f5ef9c66b731a">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0839__r782abf7d7c67454098661cdd92fac66b"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.3.2.2.2.1.2.4.1.1 "><p id="mrs_01_0839__ac0ac3eaa6c9346d69aded8b11567ae61">mapreduce.application.framework.path</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="69.64%" headers="mcps1.3.3.2.2.2.1.2.4.1.2 "><p id="mrs_01_0839__a8848dc8170d34491a90f5f7502a2b1de">Indicates the URL directing to the archive location.</p>
|
|
<div class="note" id="mrs_01_0839__ne1a318d5526147259f1d049cdc885523"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_0839__a3b8f329800244e35aca9085ae4f2bbd2">This property can also create an alias for the archive if the URL fragment identity name is specified as follows. In this example, the alias is set to <strong id="mrs_01_0839__b139310262051930">mr-framework</strong>.</p>
|
|
<pre class="screen" id="mrs_01_0839__s41c990e092814b61ae85d8079a547ee2"><property> <name>mapreduce.application.framework.path</name> <value>hdfs:/mapred/framework/hadoop-x.tar.gz#mr-framework</value> </property></pre>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.06%" headers="mcps1.3.3.2.2.2.1.2.4.1.3 "><p id="mrs_01_0839__a334d1a5cb56540dd821eee8dcf9735b9">NA</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0839__r6c33cf9b55274a3493d6a60be4e35b62"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.3.2.2.2.1.2.4.1.1 "><p id="mrs_01_0839__a664dc2bb257742b78a9ce7f86edacc5b">mapreduce.application.classpath</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="69.64%" headers="mcps1.3.3.2.2.2.1.2.4.1.2 "><p id="mrs_01_0839__ab80823cf200d47d288269fa698af9791">Indicates the parameter property, which contains the MapReduce JARs in the class directory.</p>
|
|
<div class="note" id="mrs_01_0839__n4ec530a6d536433281a62ecd421970ec"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_0839__a3c51130269494d07b68b7b44bac6fbb4">For example, the alias <strong id="mrs_01_0839__b41031696151930">mr-framework</strong> used in the framework path is used to match the directory.</p>
|
|
<pre class="screen" id="mrs_01_0839__s43c2cc7b1b8a4e5db896ab02e094178b"><property> <name>mapreduce.application.classpath</name> <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/etc/hadoop/conf/secure</value></property></pre>
|
|
<p id="mrs_01_0839__a35ebecb0065a4254a05c54316633d6c6"></p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.06%" headers="mcps1.3.3.2.2.2.1.2.4.1.3 "><p id="mrs_01_0839__a359d7203b01347f788c32890a832274d">N/A</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_0839__a01e0eab1ff534838bfdf1d64f4215c29">You can upload MapReduce tarballs of multiple versions to HDFS. Different <span class="filepath" id="mrs_01_0839__f3858e63cf9ea4f0fa3fc4727aa4709d3"><b>mapred-site.xml</b></span> files indicate different locations. After that, you can run tasks for a specific <span class="filepath" id="mrs_01_0839__fff2704780fda4174be8a4b1f117a6546"><b>mapred-site.xml</b></span> file. The following is an example of running an MapReduce task for the MapReduce tarball of the <em id="mrs_01_0839__i4842044751930">x</em> version:</p>
|
|
<p id="mrs_01_0839__a6041e8a829544c7baa15b4bef1f4fb76"><b><span class="cmdname" id="mrs_01_0839__cmdname174341857171110">hadoop jar</span></b> <i><span class="varname" id="mrs_01_0839__vfb9c7430870b45bf915c40fbb48da6f0">share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar</span></i> <i><span class="varname" id="mrs_01_0839__v5c35ab0154814ea18ca52bd42b9c614e">pi</span></i> <b><span class="cmdname" id="mrs_01_0839__cmdname5269213126">-conf</span></b> <b><span class="cmdname" id="mrs_01_0839__cmdname2363191114121">etc/hadoop-x/mapred-site.xml</span></b> <i><span class="varname" id="mrs_01_0839__v14b4bd22a5ba44479963163a4eca4f55">10 10</span></i></p>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0834.html">Using MapReduce</a></div>
|
|
</div>
|
|
</div>
|
|
|