yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
fix_python
doc-exports/docs/mrs/umn/ALM-45642.html
Yang, Tong 5914b67d13 MRS UMN Doc 20240802 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2024-09-28 19:04:58 +00:00

105 lines
18 KiB
HTML
Raw Permalink Blame History

<a name="ALM-45642"></a><a name="ALM-45642"></a>
<h1 class="topictitle1">ALM-45642 RocksDB Continuously Triggers Write Traffic Limiting</h1>
<div id="body0000002008101573"><p id="ALM-45642__p12261122253615">This section applies to MRS 3.3.0 or later.</p>
<div class="section" id="ALM-45642__section663215"><h4 class="sectiontitle"><span id="ALM-45642__text516373020197">Alarm Description</span></h4><p id="ALM-45642__p69451818165215">The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (<strong id="ALM-45642__b93546054313">metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration</strong>, 180s by default). This alarm is generated when RocksDB for a job continuously triggers write traffic limiting, that is, the RocksDB write rate is not 0. This alarm is cleared when the RocksDB write rate of the job becomes 0.</p>
<p id="ALM-45642__p168873136352">The <strong id="ALM-45642__b9928194317231">rocksdb.actual-delayed-write-rate</strong> parameter specifies the RocksDB write rate of a job. Value <strong id="ALM-45642__b1165232262412">0</strong> indicates that the rate is not limited, and other values indicate traffic limiting.</p>
</div>
<div class="section" id="ALM-45642__section5968939"><h4 class="sectiontitle"><span id="ALM-45642__text20591447192117">Alarm Attributes</span></h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-45642__table10143581" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-45642__row61411666"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.2.1.4.1.1"><p id="ALM-45642__p17386810"><span id="ALM-45642__text1864783145211">Alarm ID</span></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.2.1.4.1.2"><p id="ALM-45642__p66154394"><span id="ALM-45642__text297913110521">Alarm Severity</span></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.2.1.4.1.3"><p id="ALM-45642__p49230886"><span id="ALM-45642__text0890175712305">Auto Cleared</span></p>
</th>
</tr>
</thead>
<tbody><tr id="ALM-45642__row49774232"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.2.1.4.1.1 "><p id="ALM-45642__p5180964">45642</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.2.1.4.1.2 "><p id="ALM-45642__p17004965">Minor</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.2.1.4.1.3 "><p id="ALM-45642__p35224963">Yes</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="ALM-45642__section53720453"><h4 class="sectiontitle"><span id="ALM-45642__text18171442142214">Alarm Parameters</span></h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-45642__table34649765" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-45642__row18974100"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.4.2.1.3.1.1"><p id="ALM-45642__p42699947"><span id="ALM-45642__text6203173410617">Parameter</span></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.4.2.1.3.1.2"><p id="ALM-45642__p36143663"><span id="ALM-45642__text10819164319610">Description</span></p>
</th>
</tr>
</thead>
<tbody><tr id="ALM-45642__row16272251424"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.1 "><p id="ALM-45642__p17935380415">Source</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.2 "><p id="ALM-45642__p187931338134115">Specifies the cluster for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-45642__row38292076"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.1 "><p id="ALM-45642__p14650458">ServiceName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.2 "><p id="ALM-45642__p45836489">Specifies the service for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-45642__row73049270124"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.1 "><p id="ALM-45642__p11305162719127">ApplicationName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.2 "><p id="ALM-45642__p12493836125511">Specifies the name of the application for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-45642__row9875225"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.1 "><p id="ALM-45642__p61695723">RoleName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.2 "><p id="ALM-45642__p31297682">Specifies the role for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-45642__row13243689"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.1 "><p id="ALM-45642__p66105898">JobName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.1.3.1.2 "><p id="ALM-45642__p52977523">Specifies the job for which the alarm is generated.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="ALM-45642__section13722030"><h4 class="sectiontitle"><span id="ALM-45642__text98201443182317">Impact on the System</span></h4><p id="ALM-45642__p25040094">This alarm has no adverse impact on the system.</p>
</div>
<div class="section" id="ALM-45642__section56389407"><h4 class="sectiontitle"><span id="ALM-45642__text11871546172411">Possible Causes</span></h4><p id="ALM-45642__p1142674517133">When the rate at which Flink jobs write data to RocksDB is not 0, write traffic limiting is triggered. The possible causes are as follows:</p>
<ul id="ALM-45642__ul746322813145"><li id="ALM-45642__li446362801419">There are too many MemTables. As a result, write traffic is limited or write stops, and <strong id="ALM-45642__b123061437102815">ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold</strong> is generated.</li><li id="ALM-45642__li101302021929">The size of SST files at level 0 is too large, and <strong id="ALM-45642__b999155511116">ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold</strong> is generated.</li><li id="ALM-45642__li12463028161414">The estimated compaction size exceeds the threshold, and <strong id="ALM-45642__b1564254120213">ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold</strong> is generated.</li></ul>
</div>
<div class="section" id="ALM-45642__section178029519112"><h4 class="sectiontitle"><span id="ALM-45642__text79051154102518">Handling Procedure</span></h4><p class="tableheading" id="ALM-45642__p135760113337"><strong id="ALM-45642__b1357618111335">Check whether write traffic limiting or write stop is caused due to too many MemTables.</strong></p>
<ol id="ALM-45642__ol982443111184"><li id="ALM-45642__li88241531181819"><span>On FusionInsight Manager, choose <strong id="ALM-45642__b1271617891515">O&amp;M</strong> &gt; <strong id="ALM-45642__b27179861519">Alarm</strong> &gt; <strong id="ALM-45642__b1071714818154">Alarms</strong>.</span></li><li id="ALM-45642__li19824331181813"><span>In the alarm list, check whether <strong id="ALM-45642__b2613625181718">ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold</strong> exists.</span><p><ul id="ALM-45642__ul1824231121813"><li id="ALM-45642__li138241531191813">If yes, go to <a href="#ALM-45642__li1982483112189">3</a>.</li><li id="ALM-45642__li882419316180">If no, go to <a href="#ALM-45642__li19823113101810">5</a>.</li></ul>
</p></li><li id="ALM-45642__li1982483112189"><a name="ALM-45642__li1982483112189"></a><a name="li1982483112189"></a><span>Handle the alarm by following the instructions provided in section <strong id="ALM-45642__b151337483252">ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold</strong>.</span></li><li id="ALM-45642__li1382415317188"><span>After ALM-45643 is cleared, wait a few minutes and check whether this alarm is cleared.</span><p><ul id="ALM-45642__ul172191622161214"><li id="ALM-45642__li8219122211121">If yes, no further action is required.</li><li id="ALM-45642__li3219922111211">If no, go to <a href="#ALM-45642__li19823113101810">5</a>.</li></ul>
</p></li></ol>
<p id="ALM-45642__p69711949311"><strong id="ALM-45642__b192141735132813">Check whether write traffic limiting or write stop is caused due to too many SST files at level 0.</strong></p>
<ol start="5" id="ALM-45642__ol188231431101818"><li id="ALM-45642__li19823113101810"><a name="ALM-45642__li19823113101810"></a><a name="li19823113101810"></a><span>On FusionInsight Manager, choose <strong id="ALM-45642__b637202443020">O&amp;M</strong> &gt; <strong id="ALM-45642__b1738192415302">Alarm</strong> &gt; <strong id="ALM-45642__b113882410304">Alarms</strong>.</span></li><li id="ALM-45642__li1282393131814"><span>In the alarm list, check whether <strong id="ALM-45642__b5137102617306">ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold</strong> exists.</span><p><ul id="ALM-45642__ul3823173118186"><li id="ALM-45642__li6823183117187">If yes, go to <a href="#ALM-45642__li6823133191818">7</a>.</li><li id="ALM-45642__li28231314187">If no, go to <a href="#ALM-45642__li1485215516483">9</a>.</li></ul>
</p></li><li id="ALM-45642__li6823133191818"><a name="ALM-45642__li6823133191818"></a><a name="li6823133191818"></a><span>Handle the alarm by following the instructions provided in section <strong id="ALM-45642__b14482191216310">ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold</strong>.</span></li><li id="ALM-45642__li88231631161811"><span>After ALM-45644 is cleared, wait a few minutes and check whether this alarm is cleared.</span><p><ul id="ALM-45642__ul482318318185"><li id="ALM-45642__li78239313180">If yes, no further action is required.</li><li id="ALM-45642__li178231131131815">If no, go to <a href="#ALM-45642__li1485215516483">9</a>.</li></ul>
</p></li></ol>
<p id="ALM-45642__p3916636204711"><strong id="ALM-45642__b03631318144413">Check whether write traffic limiting or write stop is caused because the estimated compaction size exceeds the threshold.</strong></p>
<ol start="9" id="ALM-45642__ol1929554484711"><li id="ALM-45642__li1485215516483"><a name="ALM-45642__li1485215516483"></a><a name="li1485215516483"></a><span>In the alarm list, check whether <strong id="ALM-45642__b10854142717216">ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold</strong> exists.</span><p><ul id="ALM-45642__ul188529516481"><li id="ALM-45642__li158521854484">If yes, go to <a href="#ALM-45642__li1285312554818">10</a>.</li><li id="ALM-45642__li4853759485">If no, go to <a href="#ALM-45642__li1826072651812">12</a>.</li></ul>
</p></li><li id="ALM-45642__li1285312554818"><a name="ALM-45642__li1285312554818"></a><a name="li1285312554818"></a><span>Handle the alarm by following the instructions provided in section <strong id="ALM-45642__b460616261318">ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold</strong>.</span></li><li id="ALM-45642__li1585311516489"><span>After ALM-45647 is cleared, wait a few minutes and check whether this alarm is cleared.</span><p><ul id="ALM-45642__ul1560851651215"><li id="ALM-45642__li260821611124">If yes, no further action is required.</li><li id="ALM-45642__li176081216161213">If no, go to <a href="#ALM-45642__li1826072651812">12</a>.</li></ul>
</p></li></ol>
<p id="ALM-45642__p17751320141817"><strong id="ALM-45642__b18172195611319">Collect fault information.</strong></p>
<ol start="12" id="ALM-45642__ol626019269181"><li id="ALM-45642__li1826072651812"><a name="ALM-45642__li1826072651812"></a><a name="li1826072651812"></a><span>Log in to FusionInsight Manager as a user who has the FlinkServer management permission.</span></li><li id="ALM-45642__li143241423112412"><span>Choose <strong id="ALM-45642__b131881514943">O&amp;M</strong> &gt; <strong id="ALM-45642__b1818816141442">Alarm</strong> &gt; <strong id="ALM-45642__b7188111410416">Alarms</strong> &gt; <strong id="ALM-45642__b51885141547">ALM-45642 RocksDB Continuously Triggers Write Traffic Limiting</strong>, view <strong id="ALM-45642__b181889143420">Location</strong>, and obtain the name of the task for which the alarm is generated.</span></li><li id="ALM-45642__li112601426191813"><span>Choose <strong id="ALM-45642__b8941824162610">Cluster</strong> &gt; <strong id="ALM-45642__b1994119248262">Services</strong> &gt; <strong id="ALM-45642__b15941024142618">Yarn</strong> and click the link next to <strong id="ALM-45642__b189411224192619">ResourceManager WebUI</strong> to go to the native Yarn page.</span></li></ol><ol start="15" id="ALM-45642__ol53961455194712"><li id="ALM-45642__li18750195794719"><span>Locate the abnormal task based on its name displayed in <strong id="ALM-45642__b760014421954">Location</strong>, search for and record the application ID of the job, and check whether the job logs are available on the native Yarn page.</span><p><div class="fignone" id="ALM-45642__en-us_topic_0000001445372489_fig1390461517192"><span class="figcap"><b>Figure 1 </b>Application ID of a job</span><br><span><img id="ALM-45642__image203641438145416" src="en-us_image_0000001971808406.png"></span></div>
<ul id="ALM-45642__ul112631311128"><li id="ALM-45642__li22611331219">If yes, go to <a href="#ALM-45642__li14941184217233">16</a>.</li><li id="ALM-45642__li1626151371216">If no, go to <a href="#ALM-45642__li42141433171631">18</a>.</li></ul>
</p></li><li id="ALM-45642__li14941184217233"><a name="ALM-45642__li14941184217233"></a><a name="li14941184217233"></a><span>Click the application ID of the failed job to go to the job page.</span><p><ol type="a" id="ALM-45642__en-us_topic_0000001445372489_ol18905161513191"><li id="ALM-45642__en-us_topic_0000001445372489_li090431510192">Click <strong id="ALM-45642__b287710491079">Logs</strong> in the <strong id="ALM-45642__b198779497717">Logs</strong> column to view JobManager logs.<div class="fignone" id="ALM-45642__en-us_topic_0000001445372489_fig0904115131915"><span class="figcap"><b>Figure 2 </b>Clicking Logs</span><br><span><img id="ALM-45642__en-us_topic_0000001445372489_image290471501913" src="en-us_image_0000002008248417.png"></span></div>
</li><li id="ALM-45642__en-us_topic_0000001445372489_li232434015269">Click the ID in the <strong id="ALM-45642__b142178144277">Attempt ID</strong> column and click <strong id="ALM-45642__b172181614152713">Logs</strong> in the <strong id="ALM-45642__b5218101414278">Logs</strong> column to view and save TaskManager logs.<div class="fignone" id="ALM-45642__en-us_topic_0000001445372489_fig16904101571920"><span class="figcap"><b>Figure 3 </b>Clicking the ID in the Attempt ID column</span><br><span><img id="ALM-45642__en-us_topic_0000001445372489_image1890411511199" src="en-us_image_0000001971648670.png"></span></div>
<div class="fignone" id="ALM-45642__en-us_topic_0000001445372489_fig67971748144610"><span class="figcap"><b>Figure 4 </b>Clicking Logs</span><br><span><img id="ALM-45642__en-us_topic_0000001445372489_image1620681118112" src="en-us_image_0000002008128989.png"></span></div>
<div class="note" id="ALM-45642__en-us_topic_0000001445372489_note126111528152718"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="ALM-45642__en-us_topic_0000001445372489_p14611162814277">You can also log in to Manager as a user who has the FlinkServer management permission. Choose <strong id="ALM-45642__b2038831202811">Cluster</strong> &gt; <strong id="ALM-45642__b1038813113285">Services</strong> &gt; <strong id="ALM-45642__b63891015289">Flink</strong>, and click the link next to <strong id="ALM-45642__b9389191122810">Flink WebUI</strong>. On the displayed Flink web UI, click <strong id="ALM-45642__b938991152819">Job Management</strong>, click <strong id="ALM-45642__b183897111281">More</strong> in the <strong id="ALM-45642__b1338941152812">Operation</strong> column, and select <strong id="ALM-45642__b1439031162812">Job Monitoring</strong> to view TaskManager logs.</p>
</div></div>
</li></ol>
</p></li><li id="ALM-45642__li7740734122412"><span>View the job logs to rectify the fault, or contact the <span id="ALM-45642__en-us_topic_0000001445372489_text526817207207">O&amp;M personnel</span> and send the collected fault logs. No further action is required.</span></li></ol>
<p class="tableheading" id="ALM-45642__p22784204171641"><strong id="ALM-45642__b8247101919311">If logs are unavailable on the Yarn page, download logs from HDFS.</strong></p>
<ol start="18" id="ALM-45642__ol34389030171648"><li id="ALM-45642__li42141433171631"><a name="ALM-45642__li42141433171631"></a><a name="li42141433171631"></a><span>On Manager, choose <strong id="ALM-45642__b111359236314">Cluster</strong> &gt; <strong id="ALM-45642__b10136192333118">Services</strong> &gt; <strong id="ALM-45642__b101362236317">HDFS</strong>, click the link next to <strong id="ALM-45642__b2136132393120">NameNode WebUI</strong> to go to the HDFS page, choose <strong id="ALM-45642__b1713682319318">Utilities</strong> &gt; <strong id="ALM-45642__b171361423153118">Browse the file system</strong>, and download logs in the <strong id="ALM-45642__b1413702333118">/tmp/logs/</strong><em id="ALM-45642__i81371723193118">Username</em><strong id="ALM-45642__b913722313117">/bucket-logs-tfile/</strong><em id="ALM-45642__i2013772319316">Last four digits of the task application ID/Application ID of the task</em> directory.</span></li><li id="ALM-45642__li12839885171631"><span>View the logs of the failed job to rectify the fault, or contact the <span id="ALM-45642__text193191283474">O&amp;M personnel</span> and send the collected fault logs.</span></li></ol>
</div>
<div class="section" id="ALM-45642__section169311343318"><h4 class="sectiontitle"><span id="ALM-45642__text195945622616">Alarm Clearance</span></h4><p id="ALM-45642__p754913417333">This alarm is automatically cleared after the fault is rectified.</p>
</div>
<div class="section" id="ALM-45642__section4139237"><h4 class="sectiontitle"><span id="ALM-45642__text143698488285">Related Information</span></h4><p id="ALM-45642__p33559471"><span id="ALM-45642__text19275105817121">None.</span></p>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1298.html">Alarm Reference (Applicable to MRS 3.x)</a></div>
</div>
</div>
View Git Blame Copy Permalink