vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/umn/ALM-18002.html
Yang, Tong 3b1f73dece MRS UMN 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-13 12:03:34 +00:00

101 lines
16 KiB
HTML
Raw Permalink Blame History

<a name="ALM-18002"></a><a name="ALM-18002"></a>
<h1 class="topictitle1">ALM-18002 NodeManager Heartbeat Lost</h1>
<div id="body27209449"><div class="section" id="ALM-18002__s105852954c6b4aeda15055a2c4af68b3"><h4 class="sectiontitle">Description</h4><p id="ALM-18002__p6283125116462">The system checks the number of lost NodeManager nodes every 30 seconds, and compares the number with the threshold. The Number of Lost Nodes indicator has a default threshold. The alarm is generated when the value of Number of Lost Nodes exceeds the threshold.</p>
<p id="ALM-18002__p1728315154611">To change the threshold, on FusionInsight Manager, choose <strong id="ALM-18002__b1528365134612">Cluster</strong> &gt; <em id="ALM-18002__i7717173153913">Name of the desired cluster</em> &gt; <strong id="ALM-18002__b102835517464">Services</strong> &gt; <strong id="ALM-18002__b1728317512466">Yarn</strong>. On the displayed page, choose <strong id="ALM-18002__b0283751104611">Configurations</strong> &gt; <strong id="ALM-18002__b202836516468">All Configurations</strong>, and change the value of <strong id="ALM-18002__b2283175117464">yarn.nodemanager.lost.alarm.threshold</strong>. You do not need to restart Yarn to make the change take effect.</p>
<p id="ALM-18002__p328325164612">The default threshold is 0. The alarm is generated when the number of lost nodes exceeds the threshold, and is cleared when the number of lost nodes is less than the threshold.</p>
<p id="ALM-18002__p516051092951"></p>
</div>
<div class="section" id="ALM-18002__s145a8d7937fe4de88d027511391f940b"><h4 class="sectiontitle">Attribute</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-18002__en-us_topic_0070543682_table10453175" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-18002__en-us_topic_0070543682_row41172979"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.1"><p id="ALM-18002__en-us_topic_0070543682_p46677008">Alarm ID</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.2"><p id="ALM-18002__en-us_topic_0070543682_p22741322">Alarm Severity</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.3"><p id="ALM-18002__en-us_topic_0070543682_p30107784">Automatically Cleared</p>
</th>
</tr>
</thead>
<tbody><tr id="ALM-18002__en-us_topic_0070543682_row22811443"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.1 "><p id="ALM-18002__en-us_topic_0070543682_p35787570">18002</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.2 "><p id="ALM-18002__en-us_topic_0070543682_p13112046">Major</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.3 "><p id="ALM-18002__en-us_topic_0070543682_p55442766">Yes</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="ALM-18002__s6a7b937be2f045ac807963eb31955440"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-18002__en-us_topic_0070543682_table61679034" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-18002__en-us_topic_0070543682_row20168614"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.1"><p id="ALM-18002__en-us_topic_0070543682_p23045050">Name</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.2"><p id="ALM-18002__en-us_topic_0070543682_p54709725">Meaning</p>
</th>
</tr>
</thead>
<tbody><tr id="ALM-18002__row599711372315"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18002__p192431315431">Source</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18002__p692551319435">Specifies the cluster for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-18002__en-us_topic_0070543682_row2302778"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18002__en-us_topic_0070543682_p52307319">ServiceName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18002__en-us_topic_0070543682_p9034428">Specifies the service for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-18002__en-us_topic_0070543682_row14200994"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18002__en-us_topic_0070543682_p9429831">RoleName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18002__en-us_topic_0070543682_p25618824">Specifies the role for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-18002__en-us_topic_0070543682_row29242831"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18002__en-us_topic_0070543682_p19859143">HostName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18002__en-us_topic_0070543682_p65086735">Specifies the host for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-18002__en-us_topic_0070543682_row48909704"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18002__p1788433477">Lost Host</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18002__en-us_topic_0070543682_p49094708">Specifies the list of hosts with lost nodes.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="ALM-18002__s78f41e01881b4ed6a29cd3fed9330bd2"><h4 class="sectiontitle">Impact on the System</h4><ul id="ALM-18002__en-us_topic_0070543682_ul17248425"><li id="ALM-18002__en-us_topic_0070543682_li21018099">The lost NodeManager node cannot provide the Yarn service.</li><li id="ALM-18002__en-us_topic_0070543682_li54945170">The number of containers decreases, so the cluster performance deteriorates.</li></ul>
</div>
<div class="section" id="ALM-18002__s5dd736069743435ea254bc0ebfeb89af"><h4 class="sectiontitle">Possible Causes</h4><ul id="ALM-18002__en-us_topic_0070543682_ul21373784"><li id="ALM-18002__en-us_topic_0070543682_li58146329">NodeManager is forcibly deleted without decommission.</li><li id="ALM-18002__en-us_topic_0070543682_li53554914">All the NodeManager instances are stopped or the NodeManager process is faulty.</li><li id="ALM-18002__en-us_topic_0070543682_li12232180">The host where the NodeManager node resides is faulty.</li><li id="ALM-18002__en-us_topic_0070543682_li42980760">The network between the NodeManager and ResourceManager is disconnected or busy.</li></ul>
</div>
<div class="section" id="ALM-18002__en-us_topic_0070543682_section380170"><h4 class="sectiontitle">Procedure</h4><p class="tableheading" id="ALM-18002__en-us_topic_0070543682_p58889572"><strong id="ALM-18002__b17685380183753">Check the NodeManager status.</strong></p>
<ol id="ALM-18002__ol42269643183852"><li id="ALM-18002__li45221725183746"><a name="ALM-18002__li45221725183746"></a><a name="li45221725183746"></a><span>On the FusionInsight Manager, and choose <strong id="ALM-18002__b11702552125017">O&amp;M</strong> &gt; <strong id="ALM-18002__b127021852185017">Alarm &gt; Alarms</strong>. Click <span><img id="ALM-18002__image14702195211506" src="en-us_image_0269417391.png"></span> before the alarm and obtain lost nodes in <strong id="ALM-18002__b107021752155016">Additional Information</strong>.</span></li><li id="ALM-18002__li50532462183746"><span>Check whether the lost nodes are hosts that have been manually deleted without decommission.</span><p><ul class="subitemlist" id="ALM-18002__ul27984339183746"><li id="ALM-18002__li32626230183746">If yes, go to <a href="#ALM-18002__li10848276183746">3</a>.</li><li id="ALM-18002__li25478992183746">If no, go to <a href="#ALM-18002__li58395585183746">5</a>.</li></ul>
</p></li><li id="ALM-18002__li10848276183746"><a name="ALM-18002__li10848276183746"></a><a name="li10848276183746"></a><span>After the setting, Choose <strong id="ALM-18002__b128921327163915">Cluster </strong>&gt;<strong id="ALM-18002__b14892527153920"> </strong><em id="ALM-18002__i1785262318399">Name of the desired cluster</em> &gt; <strong id="ALM-18002__b19147160155311">Services</strong> &gt; <strong id="ALM-18002__b1414715075319">Yarn</strong>. On the displayed page, choose <strong id="ALM-18002__b13147160165318">Configurations </strong>&gt; <strong id="ALM-18002__b1614711014534">All Configurations</strong>. Search for <strong id="ALM-18002__b131477015311">yarn.nodemanager.lost.alarm.threshold</strong> and change its value to the number of hosts that are not out of service and proactively deleted. After the setting, check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-18002__ul1205364183746"><li id="ALM-18002__li62507714183746">If yes, no further action is required.</li><li id="ALM-18002__li29960091183746">If no, go to <a href="#ALM-18002__li30525620183746">4</a>.</li></ul>
</p></li><li id="ALM-18002__li30525620183746"><a name="ALM-18002__li30525620183746"></a><a name="li30525620183746"></a><span>Manually clear the alarm. Note that decommission must be performed before deleting hosts.</span></li><li id="ALM-18002__li58395585183746"><a name="ALM-18002__li58395585183746"></a><a name="li58395585183746"></a><span>On the FusionInsight Manager portal, choose <strong id="ALM-18002__b2481739193914">Cluster &gt; Hosts</strong>, and check whether the nodes obtained in <a href="#ALM-18002__li45221725183746">1</a> are healthy.</span><p><ul class="subitemlist" id="ALM-18002__ul43771100183746"><li id="ALM-18002__li25743681183746">If yes, go to <a href="#ALM-18002__li26422237183746">7</a>.</li><li id="ALM-18002__li4863455183746">If no, go to <a href="#ALM-18002__li3142189183746">6</a>.</li></ul>
</p></li><li id="ALM-18002__li3142189183746"><a name="ALM-18002__li3142189183746"></a><a name="li3142189183746"></a><span>Rectify the node fault based on <strong id="ALM-18002__b1956303141612">ALM-12006 Node Fault</strong> and check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-18002__ul13294864183746"><li id="ALM-18002__li32421968183746">If yes, no further action is required.</li><li id="ALM-18002__li8933747183746">If no, go to <a href="#ALM-18002__li26422237183746">7</a>.</li></ul>
</p></li></ol>
<p class="tableheading" id="ALM-18002__p52544915183746"><strong id="ALM-18002__b5074326218394">Check the process status.</strong></p>
<ol start="7" id="ALM-18002__ol28090046113217"><li id="ALM-18002__li26422237183746"><a name="ALM-18002__li26422237183746"></a><a name="li26422237183746"></a><span>On the FusionInsight Manager, choose <strong id="ALM-18002__b6896185110395">Cluster </strong>&gt; <em id="ALM-18002__i20918185120391">Name of the desired cluster</em><strong id="ALM-18002__b78976513392"> &gt; Services</strong> &gt; <strong id="ALM-18002__b53190738183746">Yarn</strong> &gt; <strong id="ALM-18002__b8954600183746">Instance</strong>, and check whether there are NodeManager instances whose status is not <strong id="ALM-18002__b13482536183746">Good</strong>.</span><p><ul class="subitemlist" id="ALM-18002__ul17848885183746"><li id="ALM-18002__li18343638183746">If yes, go to <a href="#ALM-18002__li10762250183746">10</a>.</li><li id="ALM-18002__li9439749183746">If no, go to <a href="#ALM-18002__li1508966183746">8</a>.</li></ul>
</p></li><li id="ALM-18002__li1508966183746"><a name="ALM-18002__li1508966183746"></a><a name="li1508966183746"></a><span>Check whether the NodeManager instance is deleted.</span><p><ul class="subitemlist" id="ALM-18002__ul59819986183746"><li id="ALM-18002__li59826437183746">If yes, go to <a href="#ALM-18002__li42859132183746">9</a>.</li><li id="ALM-18002__li14103205183746">If no, go to <a href="#ALM-18002__li13800982183746">11</a>.</li></ul>
</p></li><li id="ALM-18002__li42859132183746"><a name="ALM-18002__li42859132183746"></a><a name="li42859132183746"></a><span>Restart the active and standby ResourceManager instances, and check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-18002__ul35326313183746"><li id="ALM-18002__li13580698183746">If yes, no further action is required.</li><li id="ALM-18002__li26294767183746">If no, go to <a href="#ALM-18002__li34738119183746">13</a>.</li></ul>
</p></li></ol>
<p class="tableheading" id="ALM-18002__p49501368183746"><strong id="ALM-18002__b27304079113228">Check the instance status.</strong></p>
<ol start="10" id="ALM-18002__ol54064508113232"><li id="ALM-18002__li10762250183746"><a name="ALM-18002__li10762250183746"></a><a name="li10762250183746"></a><span>Select NodeManager instances which running state is not <strong id="ALM-18002__b1321810286441">Normal</strong> and restart them. Check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-18002__ul46529118183746"><li id="ALM-18002__li49037647183746">If yes, no further action is required.</li><li id="ALM-18002__li12626442183746">If no, go to <a href="#ALM-18002__li13800982183746">11</a>.</li></ul>
</p></li></ol>
<p class="tableheading" id="ALM-18002__p16108886183746"><strong id="ALM-18002__b37047335183929">Check the network status.</strong></p>
<ol start="11" id="ALM-18002__ol59981049183938"><li id="ALM-18002__li13800982183746"><a name="ALM-18002__li13800982183746"></a><a name="li13800982183746"></a><span>Log in to the management node, <strong id="ALM-18002__b455654121815">ping</strong> the IP address of the lost NodeManager node to check whether the network is disconnected or busy. <span id="ALM-18002__text11209122803817"></span></span><p><ul class="subitemlist" id="ALM-18002__ul46272685183746"><li id="ALM-18002__li66435889183746">If yes, go to <a href="#ALM-18002__li13119611183746">12</a>.</li><li id="ALM-18002__li12597949183746">If no, go to <a href="#ALM-18002__li34738119183746">13</a>.</li></ul>
</p></li><li id="ALM-18002__li13119611183746"><a name="ALM-18002__li13119611183746"></a><a name="li13119611183746"></a><span>Rectify the network, and check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-18002__ul18389069183746"><li id="ALM-18002__li57099979183746">If yes, no further action is required.</li><li id="ALM-18002__li61695553183746">If no, go to <a href="#ALM-18002__li34738119183746">13</a>.</li></ul>
</p></li></ol>
<p class="tableheading" id="ALM-18002__p31283896183746"><strong id="ALM-18002__b10410025183956">Collect fault information.</strong></p>
<ol start="13" id="ALM-18002__ol41109733183958"><li id="ALM-18002__li34738119183746"><a name="ALM-18002__li34738119183746"></a><a name="li34738119183746"></a><span>On the FusionInsight Manager in the active cluster, choose <strong id="ALM-18002__b148441545144416">O&amp;M</strong> &gt; <strong id="ALM-18002__b56055574183746">Log &gt; Download</strong>.</span></li><li id="ALM-18002__li15101241183746"><span>Select <strong id="ALM-18002__b44207621183746">Yarn</strong> in the required cluster from the <strong id="ALM-18002__b62324272183746">Service</strong>.</span></li><li id="ALM-18002__li1145664103113"><span>Click <span><img id="ALM-18002__image1945644173117" src="en-us_image_0269417392.png"></span> in the upper right corner, and set <strong id="ALM-18002__b6456941173117">Start Date</strong> and <strong id="ALM-18002__b11456154113318">End Date</strong> for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click <strong id="ALM-18002__b13456164113319">Download</strong>.</span></li><li id="ALM-18002__li3799590183746"><span>Contact the <span id="ALM-18002__text4614151421417">O&amp;M personnel</span> and send the collected logs.</span></li></ol>
</div>
<div class="section" id="ALM-18002__section1529716184534"><h4 class="sectiontitle">Alarm Clearing</h4><p id="ALM-18002__p4677152685316">After the fault is rectified, the system automatically clears this alarm.</p>
</div>
<div class="section" id="ALM-18002__s88a757d28ebd4daaa114de507c6aedfd"><h4 class="sectiontitle">Related Information</h4><p id="ALM-18002__en-us_topic_0070543682_p8735271">None</p>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1298.html">Alarm Reference (Applicable to MRS 3.x)</a></div>
</div>
</div>
View Git Blame Copy Permalink