forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
22 lines
4.1 KiB
HTML
22 lines
4.1 KiB
HTML
<a name="mrs_01_1707"></a><a name="mrs_01_1707"></a>
|
|
|
|
<h1 class="topictitle1">When Does a Balance Process in HDFS, Shut Down and Fail to be Executed Again?</h1>
|
|
<div id="body1597735020654"><div class="section" id="mrs_01_1707__sd3ab029f20ac450e9ec2149e59da02d9"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1707__a55902bff08644d62b2a15f6b71adb6bc">After I start a Balance process in HDFS, the process is shut down abnormally. If I attempt to execute the Balance process again, it fails again.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1707__sf16474cfeae24c5ebccbf82417168fde"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1707__aa842d0f2cd034e759c13138e2f238084">After a Balance process is executed in HDFS, another Balance process can be executed only after the <span class="filepath" id="mrs_01_1707__fc2cc5bc0ec2c41e29adbe65ca356810d"><b>/system/balancer.id</b></span> file is automatically released.</p>
|
|
<p id="mrs_01_1707__ae1fa0c99b4df49fea7aea5b59a6b9172">However, if a Balance process is shut down abnormally, the <span class="filepath" id="mrs_01_1707__fded3a890876a4de1aa0f268370dfc8e2"><b>/system/balancer.id</b></span> has not been released when the Balance is executed again, which triggers the <b><span class="cmdname" id="mrs_01_1707__cmdname83102016111113">append /system/balancer.id</span></b> operation.</p>
|
|
<ul id="mrs_01_1707__ufa93f942ac4d4014897cc446d0656fd6"><li id="mrs_01_1707__le6ef658af49c4f44be5697c830afc148">If the time spent on releasing the <span class="filepath" id="mrs_01_1707__ff614016aa1cd48f290d8ffcda32cbb4c"><b>/system/balancer.id</b></span> file exceeds the soft-limit lease period 60 seconds, executing the Balance process again triggers the append operation, which preempts the lease. The last block is in construction or under recovery status, which triggers the block recovery operation. The <span class="filepath" id="mrs_01_1707__f6a593e5206d8456986fe106b7b86e49f"><b>/system/balancer.id</b></span> file cannot be closed until the block recovery completes. Therefore, the append operation fails.<p id="mrs_01_1707__aa4578d90433f423ab1f08bbdd0880f50">After the <b><span class="cmdname" id="mrs_01_1707__cmdname33104166119">append /system/balancer.id</span></b> operation fails, the exception message <strong id="mrs_01_1707__a9b15f56affb242518676f38da78c6f35">RecoveryInProgressException</strong> is displayed.</p>
|
|
<pre class="screen" id="mrs_01_1707__s0e26454d1a884f908254b56875d1b750"><span id="mrs_01_1707__ph3571122531111">org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.protocol.RecoveryInProgressException): Failed to APPEND_FILE /system/balancer.id for DFSClient because lease recovery is in progress. Try again later.</span></pre>
|
|
</li><li id="mrs_01_1707__lc2372a9fa785434ab5c9cf16b47b2aa1">If the time spent on releasing the <span class="filepath" id="mrs_01_1707__f94cbafcc56104c53997c2a8e6e62cbf7"><b>/system/balancer.id</b></span> file is within 60 seconds, the original client continues to own the lease and the exception<strong id="mrs_01_1707__a53456dd6dfcd417e842d98ed9f9f2543"> </strong>AlreadyBeingCreatedException occurs and null is returned to the client. The following exception message is displayed on the client:<pre class="screen" id="mrs_01_1707__se186c0ab11544fe79afc07f3fb4195de"><span id="mrs_01_1707__ph1257312551110">java.io.IOException: Cannot create any NameNode Connectors.. Exiting...</span></pre>
|
|
</li></ul>
|
|
<p id="mrs_01_1707__ac9d64b006940430ca858d93ac42529e3">Either of the following methods can be used to solve the problem:</p>
|
|
<ul id="mrs_01_1707__u211b4532f9714502bf5fcce464a6ebf4"><li id="mrs_01_1707__lc0cf9e5f34514a5d925462c888f9cd6a">Execute the Balance process again after the hard-limit lease period expires for 1 hour, when the original client has released the lease.</li><li id="mrs_01_1707__la655803454f8406cb999c6dda8e8a90d">Delete the <span class="filepath" id="mrs_01_1707__fed1082c5dbb14755a89156debebbca3f"><b>/system/balancer.id</b></span> file before executing the Balance process again.</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1690.html">FAQ</a></div>
|
|
</div>
|
|
</div>
|
|
|