forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
95 lines
11 KiB
HTML
95 lines
11 KiB
HTML
<a name="mrs_01_0836"></a><a name="mrs_01_0836"></a>
|
|
|
|
<h1 class="topictitle1">Configuring the Log Archiving and Clearing Mechanism</h1>
|
|
<div id="body1590130628527"><div class="section" id="mrs_01_0836__s78a6bfc4c2bf4b73a47ca47fdb377c66"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0836__a626cdd0bab9746628faa69d8129cff28">Job and task logs are generated during execution of a MapReduce application.</p>
|
|
<ul id="mrs_01_0836__ul2054378171710"><li id="mrs_01_0836__li1854315819176">Job logs are generated by the MRApplicationMaster, which record details about the start and running time of jobs and each task, Counter value, and other information. After being analyzed by HistoryServer, the job logs are used to view job execution details.</li><li id="mrs_01_0836__li254312815177">A task log records the log information generated by each task running in a container. By default, task logs are stored only on the local disk of each NodeManager. After the log aggregation function is enabled, the NodeManager merges local task logs and writes them into HDFS after job execution completes.</li></ul>
|
|
<p id="mrs_01_0836__ac68be3cdec2949768edcbdc74e637786">The job logs and task logs of the MapReduce are stored on HDFS (when the log aggregation function is enabled). If the mechanism for periodically archiving and deleting log files is not configured for a cluster with a large number of computation tasks, the log files will occupy large memory space of HDFS and increase the cluster load.</p>
|
|
<p id="mrs_01_0836__a1fb0756a844d48a4b5c2a22aa79238b3">Log archive is implemented by Hadoop Archives. The number (number of Map tasks) of concurrent archiving tasks started by the Hadoop Archives is related to the total size of log files to be archived. The formula is as follows: Number of concurrent archive tasks = Total size of log files to be archived/Size of archive files.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0836__s02b22021995349908f01f74827ee1f83"><h4 class="sectiontitle">Configuration</h4><p id="mrs_01_0836__p21172338305">Go to the <strong id="mrs_01_0836__b178253143160">All Configurations</strong> page of the MapReduce service. For details, see <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.</p>
|
|
<p id="mrs_01_0836__p6381324103816">Enter the parameter name in the search box, change the parameter value, and save the configuration. On the <strong id="mrs_01_0836__b6399151417115">Dashboard</strong> tab page of the Mapreduce service, choose <strong id="mrs_01_0836__b06544179114">More</strong> > <strong id="mrs_01_0836__b5460211112">Synchronize Configuration</strong>. After the synchronization is complete, restart the Mapreduce service.</p>
|
|
<ul id="mrs_01_0836__ul6546166143619"><li id="mrs_01_0836__li135479619365">Job log parameters:
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0836__tdfac6c3b09f44f9c91cc1dafc3b9dcf0" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0836__rb1db0bdcb11c453eac569fbe5c319cc6"><th align="left" class="cellrowborder" valign="top" width="28.13%" id="mcps1.3.2.4.1.1.2.4.1.1"><p id="mrs_01_0836__a2ff371644c6d483a8af567e9e0b81f91"><strong id="mrs_01_0836__b860314814245">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="49%" id="mcps1.3.2.4.1.1.2.4.1.2"><p id="mrs_01_0836__a7eddb89ef90845aca1d38635ca23ff94"><strong id="mrs_01_0836__b1168615109249">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="22.869999999999997%" id="mcps1.3.2.4.1.1.2.4.1.3"><p id="mrs_01_0836__a4f63c038db1e446588075403a992c07d"><strong id="mrs_01_0836__b156611213192413">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0836__rc505b041d6eb4b4c85510950eef46b5c"><td class="cellrowborder" valign="top" width="28.13%" headers="mcps1.3.2.4.1.1.2.4.1.1 "><p id="mrs_01_0836__a4076fd7d350047bda4ec050f36e3aea7"><span id="mrs_01_0836__pbb323b37624d4fb4bd6db46bbc155ff1">mapreduce.jobhistory.cleaner.enable</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49%" headers="mcps1.3.2.4.1.1.2.4.1.2 "><p id="mrs_01_0836__ac6bb6b4d1e9d45149d8e6cc6f7e1e03b"><span id="mrs_01_0836__pedcc96816d4e4ceea0be3043f8b0f794">Whether to enable the job log file deletion function.</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.869999999999997%" headers="mcps1.3.2.4.1.1.2.4.1.3 "><p id="mrs_01_0836__a0031e49cf185474e9c25b172766e159c"><span id="mrs_01_0836__p16f60789d5594fbea8ac249e825f2949">true</span></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0836__rcad8b39ea72c49c38cb41e2b27e899ec"><td class="cellrowborder" valign="top" width="28.13%" headers="mcps1.3.2.4.1.1.2.4.1.1 "><p id="mrs_01_0836__a084f6381be0f4cf39d11fb3bbc298d58"><span id="mrs_01_0836__p82d9677bd6d34be59e26b28e548076b9">mapreduce.jobhistory.cleaner.interval-ms</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49%" headers="mcps1.3.2.4.1.1.2.4.1.2 "><p id="mrs_01_0836__a1f3c6eeba298470cbba9514a89ef6064"><span id="mrs_01_0836__p208a62b9ae24435c9ecbe15040e714ff">Period for starting a log file cleanup. Only log files whose retention period is longer than the time specified by <span class="parmname" id="mrs_01_0836__p8c2d2b034ae2498abb28dcb6403571c9"><b>mapreduce.jobhistory.max-age-ms</b></span></span><span id="mrs_01_0836__p3837a92a0ba940fd8d248fac64f23b09"> can be deleted.</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.869999999999997%" headers="mcps1.3.2.4.1.1.2.4.1.3 "><p id="mrs_01_0836__a704f93e445854a5fbf562926b5323dc8"><span id="mrs_01_0836__p29eeec59a1874c2e9721c71b129dd446">86,400,000 ms (1 day)</span></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0836__r59fd99bd1bef4c8abb1c8280e88283c5"><td class="cellrowborder" valign="top" width="28.13%" headers="mcps1.3.2.4.1.1.2.4.1.1 "><p id="mrs_01_0836__aca46ac808d7744ae87ef4100270f414d"><span id="mrs_01_0836__p2d5600d852224f88a5a45f7c51bd111a">mapreduce.jobhistory.max-age-ms</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49%" headers="mcps1.3.2.4.1.1.2.4.1.2 "><p id="mrs_01_0836__a2b99dd1685b641ea94c119d29bed851b"><span id="mrs_01_0836__p5c50fff5394b45d7872171e34f57299a">Log files whose retention period is longer than the retention period in milliseconds specified by this parameter will be deleted.</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.869999999999997%" headers="mcps1.3.2.4.1.1.2.4.1.3 "><p id="mrs_01_0836__ae5f18075868f4e47981340d3d96f3e5f">1,296,000,000 ms (15 days)</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_0836__li132641224367">Task log parameters:
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0836__table192165382365" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0836__row7217183816362"><th align="left" class="cellrowborder" valign="top" width="28.26%" id="mcps1.3.2.4.2.1.2.4.1.1"><p id="mrs_01_0836__p102172038203612"><strong id="mrs_01_0836__b1026730366">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="51.190000000000005%" id="mcps1.3.2.4.2.1.2.4.1.2"><p id="mrs_01_0836__p122172038113610"><strong id="mrs_01_0836__b1816496279">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20.549999999999997%" id="mcps1.3.2.4.2.1.2.4.1.3"><p id="mrs_01_0836__p72176381363"><strong id="mrs_01_0836__b156916547276">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0836__row9217738203611"><td class="cellrowborder" valign="top" width="28.26%" headers="mcps1.3.2.4.2.1.2.4.1.1 "><p id="mrs_01_0836__p132176384365">yarn.log-aggregation.archive.files.minimum</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.190000000000005%" headers="mcps1.3.2.4.2.1.2.4.1.2 "><p id="mrs_01_0836__p1621716383360">Indicates the minimum number of archived MapReduce job log files. The archiving task starts when the number of files in the <span class="parmname" id="mrs_01_0836__p6475996e2be047249049e65e12dd6c67"><b>yarn.nodemanager.remote-app-log-dir</b></span> folder is greater than or equal to the value of this parameter.</p>
|
|
<p id="mrs_01_0836__p192187388369">This parameter applies to MRS 3.<em id="mrs_01_0836__i699322981714">x</em>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.549999999999997%" headers="mcps1.3.2.4.2.1.2.4.1.3 "><p id="mrs_01_0836__p6218153893617">5,000</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0836__row52181338173616"><td class="cellrowborder" valign="top" width="28.26%" headers="mcps1.3.2.4.2.1.2.4.1.1 "><p id="mrs_01_0836__p12187386368">yarn.log-aggregation.archive-check-interval-seconds</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.190000000000005%" headers="mcps1.3.2.4.2.1.2.4.1.2 "><p id="mrs_01_0836__p4218438193614">Indicates the MapReduce job log archiving interval, in seconds. Log files are archived only when the number of log files reaches the value of <span class="parmname" id="mrs_01_0836__p238a8bdf7dc64e869ea0d8bdc4b68a1a"><b>yarn.log-aggregation.archive.files.minimum</b></span>. The archiving function is disabled when the period is set to <span class="parmvalue" id="mrs_01_0836__p94c7eee71c244325a14ce9d40bbdc548"><b>0</b></span> or <span class="parmvalue" id="mrs_01_0836__p2aeaae6bacf842d4b423a6ba78223e42"><b>-1</b></span>.</p>
|
|
<p id="mrs_01_0836__p182181238173619">This parameter applies to MRS 3.<em id="mrs_01_0836__i17878154717179">x</em>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.549999999999997%" headers="mcps1.3.2.4.2.1.2.4.1.3 "><p id="mrs_01_0836__p18218138113619">-1</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0836__row2218638193614"><td class="cellrowborder" valign="top" width="28.26%" headers="mcps1.3.2.4.2.1.2.4.1.1 "><p id="mrs_01_0836__p112187385363">yarn.log-aggregation.retain-seconds</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.190000000000005%" headers="mcps1.3.2.4.2.1.2.4.1.2 "><p id="mrs_01_0836__p1621810382368">Indicates the retention period on HDFS for archiving the MapReduce job logs. The value <span class="parmvalue" id="mrs_01_0836__p8965ccfdcf6242399657ad6f918a56b6"><b>-1</b></span> indicates that log files are stored permanently.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.549999999999997%" headers="mcps1.3.2.4.2.1.2.4.1.3 "><p id="mrs_01_0836__p13218113873619">1,296,000</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0836__row142185380365"><td class="cellrowborder" valign="top" width="28.26%" headers="mcps1.3.2.4.2.1.2.4.1.1 "><p id="mrs_01_0836__p142186388364"><span id="mrs_01_0836__ph9218193814362">yarn.log-aggregation.retain-check-interval-seconds</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="51.190000000000005%" headers="mcps1.3.2.4.2.1.2.4.1.2 "><p id="mrs_01_0836__p52183389367"><span id="mrs_01_0836__pfe0004a709ef4c63b0445a0b80512f1d">Indicates the check period (in seconds) of the MapReduce job log deletion task. If this parameter is set to <span class="parmvalue" id="mrs_01_0836__pd1893fde87e54cdcb6e44709d4e67c99"><b>-1</b></span>,</span><span id="mrs_01_0836__p84cfc80600e2411eabe1a6e5115ad837"> the check period is one tenth of the log retention period</span>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.549999999999997%" headers="mcps1.3.2.4.2.1.2.4.1.3 "><p id="mrs_01_0836__p42188383361">86400</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0834.html">Using MapReduce</a></div>
|
|
</div>
|
|
</div>
|
|
|