forked from docs/doc-exports
Yang, Tong
3b1f73dece
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
91 lines
15 KiB
HTML
91 lines
15 KiB
HTML
<a name="alm_12017"></a><a name="alm_12017"></a>
|
|
|
|
<h1 class="topictitle1">ALM-12017 Insufficient Disk Capacity</h1>
|
|
<div id="body8662426"><div class="section" id="alm_12017__en-us_topic_0191813938_section33728370102144"><h4 class="sectiontitle">Description</h4><p id="alm_12017__en-us_topic_0191813938_p23344103102139">The system checks the host disk usage every 30 seconds and compares the actual disk usage with the threshold. The disk usage has a default threshold. This alarm is generated if the disk usage exceeds the threshold.</p>
|
|
<p id="alm_12017__en-us_topic_0191813938_p8770339102139">To change the threshold, choose <strong id="alm_12017__b0752118367">System</strong> > <strong id="alm_12017__b1980111153612">Threshold Configuration</strong>.</p>
|
|
<p id="alm_12017__en-us_topic_0191813938_p11824190102139">This alarm is cleared when the host disk usage is less than or equal to the threshold.</p>
|
|
</div>
|
|
<div class="section" id="alm_12017__en-us_topic_0191813938_section12259062102156"><h4 class="sectiontitle">Attribute</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="alm_12017__en-us_topic_0191813938_table29900204102139" frame="border" border="1" rules="all"><thead align="left"><tr id="alm_12017__en-us_topic_0191813938_row23378999102139"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.1"><p id="alm_12017__en-us_topic_0191813938_p14650739102139"><strong id="alm_12017__b1033528143620">Alarm ID</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.2"><p id="alm_12017__en-us_topic_0191813938_p10079948102139"><strong id="alm_12017__b4456915362">Alarm Severity</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.3"><p id="alm_12017__en-us_topic_0191813938_p33416452102139"><strong id="alm_12017__b28202911366">Auto Clear</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="alm_12017__en-us_topic_0191813938_row22378115102139"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.1 "><p id="alm_12017__en-us_topic_0191813938_p688022102139">12017</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.2 "><p id="alm_12017__en-us_topic_0191813938_p55729830102139">Major</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.3 "><p id="alm_12017__en-us_topic_0191813938_p17822362102139">Yes</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="alm_12017__en-us_topic_0191813938_section6282523010223"><h4 class="sectiontitle">Parameters</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="alm_12017__en-us_topic_0191813938_table40491443102139" frame="border" border="1" rules="all"><thead align="left"><tr id="alm_12017__en-us_topic_0191813938_row24645368102139"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.1"><p id="alm_12017__en-us_topic_0191813938_p50117754102139"><strong id="alm_12017__b1187313133363">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.2"><p id="alm_12017__en-us_topic_0191813938_p28621152102139"><strong id="alm_12017__b1955581473615">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="alm_12017__en-us_topic_0191813938_row36611985102139"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="alm_12017__en-us_topic_0191813938_p12780810102139">ServiceName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="alm_12017__en-us_topic_0191813938_p28612724102139">Specifies the service for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="alm_12017__en-us_topic_0191813938_row56187931102139"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="alm_12017__en-us_topic_0191813938_p54928561102139">RoleName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="alm_12017__en-us_topic_0191813938_p20028427102139">Specifies the role for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="alm_12017__en-us_topic_0191813938_row46038117102139"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="alm_12017__en-us_topic_0191813938_p38100001102139">HostName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="alm_12017__en-us_topic_0191813938_p66201237102139">Specifies the host for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="alm_12017__en-us_topic_0191813938_row58940228102139"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="alm_12017__en-us_topic_0191813938_p9429184102139">PartitionName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="alm_12017__en-us_topic_0191813938_p25566473102139">Specifies the disk partition for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="alm_12017__en-us_topic_0191813938_row28771670102139"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="alm_12017__en-us_topic_0191813938_p48803900102139">Trigger Condition</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="alm_12017__en-us_topic_0191813938_p60801827102139">Generates an alarm when the actual indicator value exceeds the specified threshold.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="alm_12017__en-us_topic_0191813938_section14307814102210"><h4 class="sectiontitle">Impact on the System</h4><p id="alm_12017__en-us_topic_0191813938_p32681929102139">Service processes become unavailable.</p>
|
|
</div>
|
|
<div class="section" id="alm_12017__en-us_topic_0191813938_section21983945102215"><h4 class="sectiontitle">Possible Causes</h4><p id="alm_12017__en-us_topic_0191813938_p1479659102139">The disk configuration cannot meet service requirements. The disk usage reaches the upper limit.</p>
|
|
</div>
|
|
<div class="section" id="alm_12017__en-us_topic_0191813938_section43305119102220"><h4 class="sectiontitle">Procedure</h4><ol id="alm_12017__en-us_topic_0191813938_ol57563713102351"><li id="alm_12017__en-us_topic_0191813938_li5172684010247"><span>Log in to MRS Manager and check whether the threshold is appropriate. </span><p><ol type="a" id="alm_12017__en-us_topic_0191813938_ol14909425103210"><li id="alm_12017__en-us_topic_0191813938_li55829103103210">The default threshold is 90%. You can change the threshold to meet service requirements.<ul id="alm_12017__en-us_topic_0191813938_ul28895019103214"><li id="alm_12017__en-us_topic_0191813938_li55869546103214">If yes, go to <a href="#alm_12017__en-us_topic_0191813938_li1589085510271">2</a>.</li><li id="alm_12017__en-us_topic_0191813938_li11479123103214">If no, go to <a href="#alm_12017__en-us_topic_0191813938_li39303074103210">1.b</a>.</li></ul>
|
|
</li><li id="alm_12017__en-us_topic_0191813938_li39303074103210"><a name="alm_12017__en-us_topic_0191813938_li39303074103210"></a><a name="en-us_topic_0191813938_li39303074103210"></a>Choose <strong id="alm_12017__b18799134193713">System</strong> > <strong id="alm_12017__b14799193417379">Threshold Configuration</strong> and change the alarm threshold based on the actual disk usage.</li><li id="alm_12017__en-us_topic_0191813938_li35214269103210">Wait 2 minutes and check whether the alarm is cleared.<ul id="alm_12017__en-us_topic_0191813938_ul1656602103210"><li id="alm_12017__en-us_topic_0191813938_li34817640103210">If yes, no further action is required.</li><li id="alm_12017__en-us_topic_0191813938_li44923309103210">If no, go to <a href="#alm_12017__en-us_topic_0191813938_li1589085510271">2</a>.</li></ul>
|
|
</li></ol>
|
|
</p></li><li id="alm_12017__en-us_topic_0191813938_li1589085510271"><a name="alm_12017__en-us_topic_0191813938_li1589085510271"></a><a name="en-us_topic_0191813938_li1589085510271"></a><span>Check whether the disk is a system disk.</span><p><ol type="a" id="alm_12017__en-us_topic_0191813938_ol33593014103158"><li id="alm_12017__en-us_topic_0191813938_li4203435103158"><a name="alm_12017__en-us_topic_0191813938_li4203435103158"></a><a name="en-us_topic_0191813938_li4203435103158"></a>Go to the MRS cluster details page. In the alarm list on the alarm management tab page, click the row that contains the alarm. In the alarm details, view the host name and disk partition information.</li><li id="alm_12017__en-us_topic_0191813938_li64105165103158">Log in to the node for which the alarm is generated.</li><li id="alm_12017__en-us_topic_0191813938_li2223823103328">Run the <strong id="alm_12017__b0975135583715">df -h</strong> command to check the system disk partition usage. Check whether the disk is mounted to any of the following directories by using the disk partition name obtained in <a href="#alm_12017__en-us_topic_0191813938_li4203435103158">2.a</a>: <strong id="alm_12017__b116113596373">/</strong>, <strong id="alm_12017__b861195953710">/boot</strong>, <strong id="alm_12017__b14612195917379">/home</strong>, <strong id="alm_12017__b1661295923720">/opt</strong>, <strong id="alm_12017__b5612659143720">/tmp</strong>, <strong id="alm_12017__b136121359163719">/var</strong>, <strong id="alm_12017__b1561311593378">/var/log</strong>, <strong id="alm_12017__b11613259183712">/boot</strong>, and <strong id="alm_12017__b1561315916378">/srv/BigData</strong>.<ul id="alm_12017__en-us_topic_0191813938_ul34510250103333"><li id="alm_12017__en-us_topic_0191813938_li41049479103336">If yes, the disk is a system disk. Then go to <a href="#alm_12017__en-us_topic_0191813938_li3904890010377">3.a</a>.</li><li id="alm_12017__en-us_topic_0191813938_li60332096103333">If no, the disk is not a system disk. Then go to <a href="#alm_12017__en-us_topic_0191813938_li22825392103158">2.d</a>.</li></ul>
|
|
</li><li id="alm_12017__en-us_topic_0191813938_li22825392103158"><a name="alm_12017__en-us_topic_0191813938_li22825392103158"></a><a name="en-us_topic_0191813938_li22825392103158"></a>Run the <strong id="alm_12017__b072342389">df -h</strong> command to check the system disk partition usage. Determine the role of the disk based on the disk partition name obtained in <a href="#alm_12017__en-us_topic_0191813938_li4203435103158">2.a</a>.</li><li id="alm_12017__en-us_topic_0191813938_li14263648103535">Check whether the disk is used by HDFS or Yarn.<ul id="alm_12017__en-us_topic_0191813938_ul57168968103616"><li id="alm_12017__en-us_topic_0191813938_li20089527103617">If yes, expand the disk capacity for the Core node. Then go to <a href="#alm_12017__en-us_topic_0191813938_li23401589103652">2.f</a>.</li><li id="alm_12017__en-us_topic_0191813938_li64424918103616">If no, go to <a href="#alm_12017__en-us_topic_0191813938_li572522141314">4</a>.</li></ul>
|
|
</li><li id="alm_12017__en-us_topic_0191813938_li23401589103652"><a name="alm_12017__en-us_topic_0191813938_li23401589103652"></a><a name="en-us_topic_0191813938_li23401589103652"></a>Wait 2 minutes and check whether the alarm is cleared.<ul id="alm_12017__en-us_topic_0191813938_ul30826480103657"><li id="alm_12017__en-us_topic_0191813938_li10354094103658">If yes, no further action is required.</li><li id="alm_12017__en-us_topic_0191813938_li52200367103657">If no, go to <a href="#alm_12017__en-us_topic_0191813938_li1854606410341">3</a>.</li></ul>
|
|
</li></ol>
|
|
</p></li><li id="alm_12017__en-us_topic_0191813938_li1854606410341"><a name="alm_12017__en-us_topic_0191813938_li1854606410341"></a><a name="en-us_topic_0191813938_li1854606410341"></a><span>Check whether large files are written to the disk.</span><p><ol type="a" id="alm_12017__en-us_topic_0191813938_ol764693010347"><li id="alm_12017__en-us_topic_0191813938_li3904890010377"><a name="alm_12017__en-us_topic_0191813938_li3904890010377"></a><a name="en-us_topic_0191813938_li3904890010377"></a>Run the <strong id="alm_12017__b73993391441">find / -xdev -size +500M -exec ls -l {} \;</strong> command to view files larger than 500 MB on the node. Check whether such files are written to the disk.<ul id="alm_12017__en-us_topic_0191813938_ul4855760810378"><li id="alm_12017__en-us_topic_0191813938_li43328979103712">If yes, go to <a href="#alm_12017__en-us_topic_0191813938_li65656242103715">3.b</a>.</li><li id="alm_12017__en-us_topic_0191813938_li425651110378">If no, go to <a href="#alm_12017__en-us_topic_0191813938_li572522141314">4</a>.</li></ul>
|
|
</li><li id="alm_12017__en-us_topic_0191813938_li65656242103715"><a name="alm_12017__en-us_topic_0191813938_li65656242103715"></a><a name="en-us_topic_0191813938_li65656242103715"></a>Handle the large files and check whether the alarm is cleared 2 minutes later.<ul id="alm_12017__en-us_topic_0191813938_ul34292994103718"><li id="alm_12017__en-us_topic_0191813938_li2997556103720">If yes, no further action is required.</li><li id="alm_12017__en-us_topic_0191813938_li5927267103718">If no, go to <a href="#alm_12017__en-us_topic_0191813938_li572522141314">4</a>.</li></ul>
|
|
</li><li id="alm_12017__en-us_topic_0191813938_li53383510347">Expand the disk capacity.</li><li id="alm_12017__en-us_topic_0191813938_li9758982104043">Wait 2 minutes and check whether the alarm is cleared.<ul id="alm_12017__en-us_topic_0191813938_ul20473597104045"><li id="alm_12017__en-us_topic_0191813938_li63647078104048">If yes, no further action is required.</li><li id="alm_12017__en-us_topic_0191813938_li50664922104045">If no, go to <a href="#alm_12017__en-us_topic_0191813938_li572522141314">4</a>.</li></ul>
|
|
</li></ol>
|
|
</p></li><li id="alm_12017__en-us_topic_0191813938_li572522141314"><a name="alm_12017__en-us_topic_0191813938_li572522141314"></a><a name="en-us_topic_0191813938_li572522141314"></a><span>Collect fault information.</span><p><ol type="a" id="alm_12017__en-us_topic_0191813938_en-us_topic_0191813935_ol6089206913036"><li id="alm_12017__en-us_topic_0191813938_en-us_topic_0191813935_li4478836213036">On MRS Manager, choose <span class="menucascade" id="alm_12017__menucascade644583219513"><b><span class="uicontrol" id="alm_12017__uicontrol194441032255">System</span></b> > <b><span class="uicontrol" id="alm_12017__uicontrol20445532358">Export Log</span></b></span>.</li><li id="alm_12017__li18574327401">Contact technical support engineers for help. For details, see <a href="https://docs.otc.t-systems.com/en-us/public/learnmore.html" target="_blank" rel="noopener noreferrer">technical support</a>.</li></ol>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="alm_12017__en-us_topic_0191813938_section13081136172452"><h4 class="sectiontitle"><strong id="alm_12017__b191056341350">Reference</strong></h4><p id="alm_12017__en-us_topic_0191813938_p509006751263">None</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0241.html">Alarm Reference (Applicable to Versions Earlier Than MRS 3.x)</a></div>
|
|
</div>
|
|
</div>
|
|
|