forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
87 lines
11 KiB
HTML
87 lines
11 KiB
HTML
<a name="mrs_01_1684"></a><a name="mrs_01_1684"></a>
|
|
|
|
<h1 class="topictitle1">Performing Concurrent Operations on HDFS Files</h1>
|
|
<div id="body1595904096999"><div class="section" id="mrs_01_1684__se9608011680e423ca403d5207c374daa"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1684__a38fb70b283514ab78798b4fe0c90e0b6">Performing this operation can concurrently modify file and directory permissions and access control tools in a cluster.</p>
|
|
<div class="note" id="mrs_01_1684__note17408121619136"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1684__p788202018132">This section applies to MRS 3.<em id="mrs_01_1684__i1581922711583">x</em> or later clusters.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1684__s96b8a9dc800a44b7b35e3a2d8fafa0e1"><h4 class="sectiontitle">Impact on the System</h4><p id="mrs_01_1684__p1365278267">Performing concurrent file modification operations in a cluster has adverse impacts on the cluster performance. Therefore, you are advised to do so when the cluster is idle.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1684__sa8a135a114cf4cbc8674242bd0cfebd7"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1684__ul6387132716574"><li id="mrs_01_1684__li8601330165719">The HDFS client or clients including HDFS has been installed. For example, the installation directory is <strong id="mrs_01_1684__b205976078852019">/opt/client</strong>.</li><li id="mrs_01_1684__la8a352e20c414015a0e60f8023ecc0a5">Service component users are created by the administrator as required. In security mode, machine-machine users need to download the keytab file. A human-machine user needs to change the password upon the first login. (This operation is not required in normal mode.)</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1684__section10593839103019"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1684__o65a723c3f0644a5ca5bc95b50051ef8f"><li id="mrs_01_1684__l3a2e1b7a8d004152865881b7d9bda58b"><span>Log in to the node where the client is installed as the client installation user.</span></li><li id="mrs_01_1684__l918962d747ac488fae8d369a02cf0b01"><span>Run the following command to go to the client installation directory:</span><p><p id="mrs_01_1684__ae6fabba0d23e4ae4bc35c6ef79cff605"><strong id="mrs_01_1684__a5a32ca4aacc5401ca0e8e8a12040e570">cd /opt/client</strong></p>
|
|
</p></li><li id="mrs_01_1684__lf110d935709c4ac48ee0ad1420665c2e"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_1684__acfff651122a74259a5e01741838c29e9"><strong id="mrs_01_1684__a468a1a99aae541fbab02686dddf0e45a">source bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1684__lc24dad0c1fb34eec930f24087726a7fd"><span>If the cluster is in security mode, the user executing the DistCp command must belong to the <strong id="mrs_01_1684__b6672427184217">supergroup</strong> group and run the following command to perform user authentication. In normal mode, user authentication is not required.</span><p><p id="mrs_01_1684__p26289282124"><strong id="mrs_01_1684__b6628112811123">kinit</strong> <em id="mrs_01_1684__i862892861215">Component service user</em></p>
|
|
</p></li><li id="mrs_01_1684__li19814121391317"><span>Increase the JVM size of the client to prevent out of memory (OOM). (32 GB is recommended for 100 million files.)</span><p><div class="note" id="mrs_01_1684__note1436184214133"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1684__p320316370353">The HDFS client exits abnormally and the error message "java.lang.OutOfMemoryError" is displayed after the HDFS client command is executed.</p>
|
|
<p id="mrs_01_1684__p127670271150">This problem occurs because the memory required for running the HDFS client exceeds the preset upper limit (128 MB by default). You can change the memory upper limit of the client by modifying <strong id="mrs_01_1684__b190951605352019">CLIENT_GC_OPTS</strong> in <em id="mrs_01_1684__i3404754052019"><Client installation path></em><strong id="mrs_01_1684__b72682366052019">/HDFS/component_env</strong>. For example, if you want to set the upper limit to 1 GB, run the following command:</p>
|
|
<p id="mrs_01_1684__p2244181342020">CLIENT_GC_OPTS="-Xmx1G"</p>
|
|
<p id="mrs_01_1684__p411125882013">After the modification, run the following command to make the modification take effect:</p>
|
|
<p id="mrs_01_1684__p10140131102115"><strong id="mrs_01_1684__b148666076252019">source </strong><<em id="mrs_01_1684__i48222422052019">Client installation path</em>>/<strong id="mrs_01_1684__b208682495952019">/bigdata_env</strong></p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_1684__li6626247810502"><span>Run the concurrent commands shown in the following table.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1684__table41323024102735" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_1684__row3454690102735"><th align="left" class="cellrowborder" valign="top" width="31.810000000000006%" id="mcps1.3.4.2.6.2.1.1.4.1.1"><p id="mrs_01_1684__p35441715102735">Command</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="45.080000000000005%" id="mcps1.3.4.2.6.2.1.1.4.1.2"><p id="mrs_01_1684__p881443102735">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="23.110000000000003%" id="mcps1.3.4.2.6.2.1.1.4.1.3"><p id="mrs_01_1684__p179022511156">Function</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1684__row38974829102735"><td class="cellrowborder" valign="top" width="31.810000000000006%" headers="mcps1.3.4.2.6.2.1.1.4.1.1 "><p id="mrs_01_1684__p8953334668">hdfs quickcmds [-t threadsNumber] [-p principal] [-k keytab] -setrep <rep> <path> ...</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="45.080000000000005%" headers="mcps1.3.4.2.6.2.1.1.4.1.2 "><p id="mrs_01_1684__p151191933274"><strong id="mrs_01_1684__b88424853552019">threadsNumber</strong> indicates the number of concurrent threads. The default value is the number of vCPUs of the local host.</p>
|
|
<p id="mrs_01_1684__p10119333274"><strong id="mrs_01_1684__b157431642652019">principal</strong> indicates the Kerberos user.</p>
|
|
<p id="mrs_01_1684__p11199336710"><strong id="mrs_01_1684__b117623924052019">keytab</strong> indicates the Keytab file.</p>
|
|
<p id="mrs_01_1684__p311933312710"><strong id="mrs_01_1684__b122591206752019">rep</strong> indicates the number of replicas.</p>
|
|
<p id="mrs_01_1684__p1811912334716"><strong id="mrs_01_1684__b4534803152019">path</strong> indicates the HDFS directory.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="23.110000000000003%" headers="mcps1.3.4.2.6.2.1.1.4.1.3 "><p id="mrs_01_1684__p32772204103835">Used to concurrently set the number of copies of all files in a directory.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1684__row5050198102735"><td class="cellrowborder" valign="top" width="31.810000000000006%" headers="mcps1.3.4.2.6.2.1.1.4.1.1 "><p id="mrs_01_1684__p14541442102718">hdfs quickcmds [-t threadsNumber] [-p principal] [-k keytab] -chown [owner][:[group]] <path> ...</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="45.080000000000005%" headers="mcps1.3.4.2.6.2.1.1.4.1.2 "><p id="mrs_01_1684__p37505471379"><strong id="mrs_01_1684__b185253775652019">threadsNumber</strong> indicates the number of concurrent threads. The default value is the number of vCPUs of the local host.</p>
|
|
<p id="mrs_01_1684__p4750647774"><strong id="mrs_01_1684__b73774731752019">principal</strong> indicates the Kerberos user.</p>
|
|
<p id="mrs_01_1684__p137507473714"><strong id="mrs_01_1684__b27140137452019">keytab</strong> indicates the Keytab file.</p>
|
|
<p id="mrs_01_1684__p97503474710"><strong id="mrs_01_1684__b159929268652019">owner</strong> indicates the owner.</p>
|
|
<p id="mrs_01_1684__p77505477715"><strong id="mrs_01_1684__b150394601552019">group</strong> indicates the group to which the user belongs.</p>
|
|
<p id="mrs_01_1684__p275017471777"><strong id="mrs_01_1684__b111163000852019">path</strong> indicates the HDFS directory.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="23.110000000000003%" headers="mcps1.3.4.2.6.2.1.1.4.1.3 "><p id="mrs_01_1684__p49658254103835">Used to concurrently set the owner group of all files in the directory.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1684__row45034927102735"><td class="cellrowborder" valign="top" width="31.810000000000006%" headers="mcps1.3.4.2.6.2.1.1.4.1.1 "><p id="mrs_01_1684__p165399564610">hdfs quickcmds [-t threadsNumber] [-p principal] [-k keytab] -chmod <mode> <path> ...</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="45.080000000000005%" headers="mcps1.3.4.2.6.2.1.1.4.1.2 "><p id="mrs_01_1684__p16812114782417"><strong id="mrs_01_1684__b202692739952019">threadsNumber</strong> indicates the number of concurrent threads. The default value is the number of vCPUs of the local host.</p>
|
|
<p id="mrs_01_1684__p881324722415"><strong id="mrs_01_1684__b79563875952019">principal</strong> indicates the Kerberos user.</p>
|
|
<p id="mrs_01_1684__p981304710245"><strong id="mrs_01_1684__b72998667252019">keytab</strong> indicates the Keytab file.</p>
|
|
<p id="mrs_01_1684__p2081384715241"><strong id="mrs_01_1684__b24468493552019">mode</strong> indicates the permission (for example, 754).</p>
|
|
<p id="mrs_01_1684__p10813134792415"><strong id="mrs_01_1684__b19401837352019">path</strong> indicates the HDFS directory.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="23.110000000000003%" headers="mcps1.3.4.2.6.2.1.1.4.1.3 "><p id="mrs_01_1684__p53746577103835">Used to concurrently set permissions for all files in a directory.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1684__row34511355102735"><td class="cellrowborder" valign="top" width="31.810000000000006%" headers="mcps1.3.4.2.6.2.1.1.4.1.1 "><p id="mrs_01_1684__p1261755713">hdfs quickcmds [-t threadsNumber] [-p principal] [-k keytab] -setfacl [{-b|-k} {-m|-x <acl_spec>} <path> ...]|[--set <acl_spec> <path> ...]</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="45.080000000000005%" headers="mcps1.3.4.2.6.2.1.1.4.1.2 "><p id="mrs_01_1684__p136471258152419"><strong id="mrs_01_1684__b96870977452019">threadsNumber</strong> indicates the number of concurrent threads. The default value is the number of vCPUs of the local host.</p>
|
|
<p id="mrs_01_1684__p11647125852419"><strong id="mrs_01_1684__b36417191752019">principal</strong> indicates the Kerberos user.</p>
|
|
<p id="mrs_01_1684__p664715584248"><strong id="mrs_01_1684__b121935783652019">keytab</strong> indicates the Keytab file.</p>
|
|
<p id="mrs_01_1684__p5647115822419"><strong id="mrs_01_1684__b48245735952019">acl_spec</strong> indicates the ACL list separated by commas (,).</p>
|
|
<p id="mrs_01_1684__p2064725812410"><strong id="mrs_01_1684__b42429238852019">path</strong> indicates the HDFS directory.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="23.110000000000003%" headers="mcps1.3.4.2.6.2.1.1.4.1.3 "><p id="mrs_01_1684__p57843426103835">Used to concurrently set ACL information for all files in a directory.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0790.html">Using HDFS</a></div>
|
|
</div>
|
|
</div>
|
|
|