forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
43 lines
6.8 KiB
HTML
43 lines
6.8 KiB
HTML
<a name="mrs_01_1949"></a><a name="mrs_01_1949"></a>
|
|
|
|
<h1 class="topictitle1">Viewing Aggregated Container Logs on the Web UI</h1>
|
|
<div id="body1595920207013"><div class="section" id="mrs_01_1949__s8eab832973474ee2b7d10f857b4ce59a"><h4 class="sectiontitle">Scenarios</h4><p id="mrs_01_1949__a07a4208870594874b1c31e55edafaf65">When <span class="parmname" id="mrs_01_1949__parmname18416017491"><b>yarn.log-aggregation-enable</b></span> of Yarn is set to <span class="parmvalue" id="mrs_01_1949__parmvalue084710134918"><b>true</b></span>, the container log aggregation function is enabled. Log aggregation indicates that after applications are run on Yarn, NodeManager aggregates all container logs of the node to HDFS and deletes local logs. For details, see <a href="mrs_01_0858.html">Configuring Container Log Aggregation</a>.</p>
|
|
<p id="mrs_01_1949__ae8afabb222c443a6868dfb2cc612beeb">However, all logs will be aggregated to an HDFS directory and can only be viewed by accessing an HDFS file. Open-source Spark and Yarn do not support the function of viewing aggregated logs on the web UI.</p>
|
|
<p id="mrs_01_1949__af914439db28f41a0995648bd390fb45e">Spark supports this function. As shown in <a href="#mrs_01_1949__fig157091128102512">Figure 1</a>, the <span class="wintitle" id="mrs_01_1949__wb1c7b0e2233d41f1a36500d302274b73"><b>AggregatedLogs</b></span> tab is added to the HistoryServer page. You can click <span class="parmname" id="mrs_01_1949__pae5c6f2d904147d7a8fcb8a64afd0c36"><b>logs</b></span> to view aggregated logs.</p>
|
|
<div class="fignone" id="mrs_01_1949__fig157091128102512"><a name="mrs_01_1949__fig157091128102512"></a><a name="fig157091128102512"></a><span class="figcap"><b>Figure 1 </b>Log aggregation page</span><br><span><img id="mrs_01_1949__image15830111917257" src="en-us_image_0000001387894476.png"></span></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1949__sddf08557702c41eca697e470c8e674e4"><h4 class="sectiontitle">Configuration Description</h4><p id="mrs_01_1949__a9291737594254f96b3f65fa7ad6d1688">To display logs on the web UI, aggregated logs need to be parsed and presented. Spark parses aggregation logs using JobHistoryServer of Hadoop. Therefore, you can use the <span class="parmname" id="mrs_01_1949__pc729fc9a7f0f4475a7a6229300326516"><b>spark.jobhistory.address</b></span> parameter to specify the URL of the JobHistoryServer page to parse and present the logs.</p>
|
|
<p id="mrs_01_1949__a00fd6dbbba8b410793143e9663be285a"><strong id="mrs_01_1949__a9edeaf7c110b4f818dd79da24af5c068">Navigation path for setting parameters:</strong></p>
|
|
<p id="mrs_01_1949__a11b0680af22c453882f772db6705f2a4">When submitting an application, set these parameters using <span class="parmname" id="mrs_01_1949__pd11b4e78945d4301884ee18c3f0de8b6"><b>--conf</b></span> or adjust the following parameter in the <span class="filepath" id="mrs_01_1949__f9b2f1f043ab94897a2f2a2cdb4ea4084"><b>spark-defaults.conf</b></span> configuration file on the client.</p>
|
|
<div class="note" id="mrs_01_1949__ne9f0932cba6540398adde4984146173f"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1949__uc8b7f451aa6448bfa46dac69656c10a8"><li id="mrs_01_1949__l1ec4b07acc544a97b24a70522df623e3">This function depends on JobHistoryServer of Hadoop. Therefore, ensure that JobHistoryServer is running properly before using the log aggregation function.</li><li id="mrs_01_1949__l12909dc4dcbd47d1a0a1034759fc4450">If the parameter value is empty, the <span class="wintitle" id="mrs_01_1949__wintitle2053822960113231"><b>AggregatedLogs</b></span> tab page still exists, but you cannot view logs by clicking <strong id="mrs_01_1949__b2122110342113231">logs</strong>.</li><li id="mrs_01_1949__lf92ce12bd3774df49af0c903ad699de1">The aggregated container logs can be viewed only when the application is running and event log files of the application exist on HDFS.</li><li id="mrs_01_1949__lad07931d8202452388379d66b758c7de">You can click the log link on the <span class="wintitle" id="mrs_01_1949__wintitle1390664030113231"><b>Executors</b></span> page to view the logs of a running task. After the task completes, the logs are aggregated to HDFS, and the log link on the <span class="wintitle" id="mrs_01_1949__wintitle340136682113231"><b>Executors</b></span> page becomes invalid. In this case, you can click <strong id="mrs_01_1949__b1015542336113231">logs</strong> on the <span class="wintitle" id="mrs_01_1949__wintitle2084740360113231"><b>AggregatedLogs</b></span> page to view the aggregated logs.</li></ul>
|
|
</div></div>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1949__t7f090357d37c49328051a7a4bc0fd4f8" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1949__r3ae0207e5618488dbd9409356e64e62e"><th align="left" class="cellrowborder" valign="top" width="24.79%" id="mcps1.3.2.6.2.4.1.1"><p id="mrs_01_1949__afd81cf0147ed4c95b5f03c882e932fad"><strong id="mrs_01_1949__a43a5e46c70624f4293abb0c92f715020">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="64.95%" id="mcps1.3.2.6.2.4.1.2"><p id="mrs_01_1949__af82abc2bec3b43ccb4f67ae2b6fe5991"><strong id="mrs_01_1949__afa3597bde7e5496eac76cbe4b1a2d434">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="10.26%" id="mcps1.3.2.6.2.4.1.3"><p id="mrs_01_1949__adc08f858813d47e985c767b3bf1bcbb0"><strong id="mrs_01_1949__a66aa31559a244770be22ba97c10e1c6e">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1949__r7920d336445d4c79a58d1873d019820b"><td class="cellrowborder" valign="top" width="24.79%" headers="mcps1.3.2.6.2.4.1.1 "><p id="mrs_01_1949__ad5b782c6cf5243e5b8e7dfab9aeb85e5">spark.jobhistory.address</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="64.95%" headers="mcps1.3.2.6.2.4.1.2 "><p id="mrs_01_1949__aad894612c5214e8396a236efc310b5d5">URL of the JobHistoryServer page. The format is <em id="mrs_01_1949__a507aaf39e0e2444ab2e81c9682bd37ec">http(s)://ip:port/jobhistory</em>. For example, <span class="parmvalue" id="mrs_01_1949__p1fcd156546a54517b95ccc9578b17dac"><b>https://10.92.115.1:26014/jobhistory</b></span>.</p>
|
|
<p id="mrs_01_1949__aa7b2d1ca00544be7a2ef3ad3cc22285a">The default value is empty, indicating that container aggregation logs cannot be viewed on the web UI.</p>
|
|
<p id="mrs_01_1949__a6b64f86cf1b54a648488124286c2fb9f">Restart the service for the configuration to take effect.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.26%" headers="mcps1.3.2.6.2.4.1.3 "><p id="mrs_01_1949__a11b228b346264d72846278152b785479">-</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|