forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
40 lines
5.2 KiB
HTML
40 lines
5.2 KiB
HTML
<a name="mrs_01_1653"></a><a name="mrs_01_1653"></a>
|
|
|
|
<h1 class="topictitle1">Why Does HBase Become Faulty When I Set a Quota for the Directory Used by HBase in HDFS?</h1>
|
|
<div id="body1596003895084"><div class="section" id="mrs_01_1653__sffb39691888c451aa69944e660c690ef"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1653__af6f3affaa2064ce2ad676d5939bc5774">Why does HBase become faulty when I set quota for the directory used by HBase in HDFS?</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1653__s7a749ad8c43b4502bf3048b6c309f8f6"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1653__a9310dd8b981e4c4f928be741f2dc082f">The flush operation of a table is to write memstore data to HDFS.</p>
|
|
<p id="mrs_01_1653__aeea7851b5fe94e298f9c4128d0aa8e50">If the HDFS directory does not have sufficient disk space quota, the flush operation will fail and the region server will stop.</p>
|
|
<pre class="screen" id="mrs_01_1653__s56931a45944a4593ae923df3e5ca76a1">Caused by: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /hbase/data/<namespace>/<tableName> is exceeded: quota = 1024 B = 1 KB but diskspace consumed = 402655638 B = 384.00 MB
|
|
?at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyStoragespaceQuota(DirectoryWithQuotaFeature.java:211)
|
|
?at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:239)
|
|
?at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:882)
|
|
?at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:711)
|
|
?at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:670)
|
|
?at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:495)</pre>
|
|
<p id="mrs_01_1653__ac6f43fd6052f48518580f0638510aecb">In the preceding exception, the disk space quota of the <span class="filepath" id="mrs_01_1653__f040243053c8d4b14b92ffd10a2319436"><b>/hbase/data/<namespace>/<tableName></b></span> table is 1 KB, but the memstore data is 384.00 MB. Therefore, the flush operation fails and the region server stops.</p>
|
|
<p id="mrs_01_1653__a6a23816689df4acf8ff935ce3f01381a">When the region server is terminated, HMaster replays the WAL file of the terminated region server to restore data. The disk space quota is limited. As a result, the replay operation of the WAL file fails, and the HMaster process exits unexpectedly.</p>
|
|
<pre class="screen" id="mrs_01_1653__s99eee56d30e24d2f8cb562c228b482a1">2016-07-28 19:11:40,352 | FATAL | MASTER_SERVER_OPERATIONS-10-91-9-131:16000-0 | Caught throwable while processing event M_SERVER_SHUTDOWN | org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2474)
|
|
java.io.IOException: failed log splitting for 10-91-9-131,16020,1469689987884, will retry
|
|
?at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:365)
|
|
?at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:220)
|
|
?at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
|
|
?at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
|
|
?at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
|
|
?at java.lang.Thread.run(Thread.java:745)
|
|
Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://hacluster/hbase/WALs/<RS-Hostname>,<RS-Port>,<startcode>-splitting] Task = installed = 6 done = 3 error = 3
|
|
?at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290)
|
|
?at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:402)
|
|
?at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:375)</pre>
|
|
<p id="mrs_01_1653__a1b4905b84129411e998fbd1df3258f46">Therefore, you cannot set the quota value for the HBase directory in HDFS. If the exception occurs, perform the following operations: </p>
|
|
</div>
|
|
<ol id="mrs_01_1653__obeac96afb4cf4970b8d243bcd896a84e"><li id="mrs_01_1653__l0f70d4f92dde4b09b5c950c0898dfca5"><span>Run the <b><span class="cmdname" id="mrs_01_1653__cmdname92874461427">kinit</span></b> <em id="mrs_01_1653__i0485154104216">Username</em> command on the client to enable the HBase user to obtain security authentication.</span></li><li id="mrs_01_1653__ladb748f730234a7a91f084e665dd9d48"><span>Run the <b><span class="cmdname" id="mrs_01_1653__cmdname1449355916425">hdfs dfs -count -q</span></b> <i><span class="varname" id="mrs_01_1653__v1dd10370c130469e80db47380680094e">/hbase/data/<namespace>/<tableName></span></i> command to check the allocated disk space quota.</span></li><li id="mrs_01_1653__l86624b7443114bfda259160dd7dcd04a"><span>Run the following command to cancel the quota limit and restore HBase:</span><p><p id="mrs_01_1653__a738006ae1863435fb1f5ba64031c55cd"><b><span class="cmdname" id="mrs_01_1653__cmdname102251354311">hdfs dfsadmin -clrSpaceQuota</span></b> <i><span class="varname" id="mrs_01_1653__v0e84267d0cbf4c25a8e7b8a3243ae60c">/hbase/data/<namespace>/<tableName></span></i></p>
|
|
</p></li></ol>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1638.html">Common Issues About HBase</a></div>
|
|
</div>
|
|
</div>
|
|
|