forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
118 lines
13 KiB
HTML
118 lines
13 KiB
HTML
<a name="mrs_01_0596"></a><a name="mrs_01_0596"></a>
|
|
|
|
<h1 class="topictitle1">Flink Log Overview</h1>
|
|
<div id="body1589421638568"><div class="section" id="mrs_01_0596__section8415510499"><h4 class="sectiontitle">Log Description</h4><div class="p" id="mrs_01_0596__p128661441497"><strong id="mrs_01_0596__b181051025123915">Log path:</strong><ul id="mrs_01_0596__ul686615441890"><li id="mrs_01_0596__li98669441691">Run logs of a Flink job: <strong id="mrs_01_0596__b719310245113">${BIGDATA_DATA_HOME}/hadoop/data${i}/nm/containerlogs/application_${appid}/container_{$contid}</strong><div class="note" id="mrs_01_0596__note1583913511576"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0596__p383912512718">The logs of executing tasks are stored in the preceding path. After the execution is complete, the Yarn configuration determines whether these logs are gathered to the HDFS directory.</p>
|
|
</div></div>
|
|
</li><li id="mrs_01_0596__li58661044192">FlinkResource run logs: <strong id="mrs_01_0596__b2270319829">/var/log/Bigdata/flink/flinkResource</strong></li></ul>
|
|
</div>
|
|
<p id="mrs_01_0596__p986614449919"><strong id="mrs_01_0596__b323282025713">Log archive rules:</strong></p>
|
|
<ol id="mrs_01_0596__ol1731462014464"><li id="mrs_01_0596__li142661403474">FlinkResource run logs:<ul id="mrs_01_0596__ul720161518399"><li id="mrs_01_0596__li82041514393">By default, service logs are backed up each time when the log size reaches 20 MB. A maximum of 20 logs can be reserved without being compressed.<div class="note" id="mrs_01_0596__note1551019429533"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0596__p14511042105319">For versions earlier than MRS 3.x, The executor logs are backed up each time when the log size reaches 30 MB. A maximum of 20 logs can be reserved without being compressed.</p>
|
|
</div></div>
|
|
</li><li id="mrs_01_0596__li252395619538">You can set the log size and number of compressed logs on the Manager page or modify the corresponding configuration items in <strong id="mrs_01_0596__b6772111161617">log4j-cli.properties</strong>, <strong id="mrs_01_0596__b8537113313161">log4j.properties</strong>, and <strong id="mrs_01_0596__b9704737101612">log4j-session.properties</strong> in <strong id="mrs_01_0596__b1388035911419">/opt/client/Flink/flink/conf/</strong> on the client. <strong id="mrs_01_0596__b82441748131712">/opt</strong><strong id="mrs_01_0596__b19245948141714"></strong><strong id="mrs_01_0596__b8246184812179">/client</strong> is the client installation directory.</li></ul>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0596__table1020915123917" frame="border" border="1" rules="all"><caption><b>Table 1 </b>FlinkResource log list</caption><thead align="left"><tr id="mrs_01_0596__row721121516394"><th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.1.4.1.2.2.4.1.1"><p id="mrs_01_0596__p18211215173912"><strong id="mrs_01_0596__b19771125375311">Type</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.1.4.1.2.2.4.1.2"><p id="mrs_01_0596__p4211415193914"><strong id="mrs_01_0596__b2292258775">Name</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.1.4.1.2.2.4.1.3"><p id="mrs_01_0596__p1821161583911"><strong id="mrs_01_0596__b18215155393">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0596__row174211839173915"><td class="cellrowborder" rowspan="5" valign="top" width="33.333333333333336%" headers="mcps1.3.1.4.1.2.2.4.1.1 "><p id="mrs_01_0596__p0421839153918">FlinkResource run logs</p>
|
|
<p id="mrs_01_0596__p1118711862620"></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.1.4.1.2.2.4.1.2 "><p id="mrs_01_0596__p592915333913">checkService.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.1.4.1.2.2.4.1.3 "><p id="mrs_01_0596__p124221939123916">Health check log</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0596__row51077910402"><td class="cellrowborder" valign="top" headers="mcps1.3.1.4.1.2.2.4.1.1 "><p id="mrs_01_0596__p1230311654015">kinit.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.4.1.2.2.4.1.2 "><p id="mrs_01_0596__p6108159194019">Initialization log</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0596__row143173919408"><td class="cellrowborder" valign="top" headers="mcps1.3.1.4.1.2.2.4.1.1 "><p id="mrs_01_0596__p17143154154018">postinstall.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.4.1.2.2.4.1.2 "><p id="mrs_01_0596__p43539124013">Service installation log</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0596__row63539144014"><td class="cellrowborder" valign="top" headers="mcps1.3.1.4.1.2.2.4.1.1 "><p id="mrs_01_0596__p7641145915409">prestart.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.4.1.2.2.4.1.2 "><p id="mrs_01_0596__p73939174016">Prestart script log</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0596__row3318391403"><td class="cellrowborder" valign="top" headers="mcps1.3.1.4.1.2.2.4.1.1 "><p id="mrs_01_0596__p023915718413">start.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.4.1.2.2.4.1.2 "><p id="mrs_01_0596__p14393914408">Startup log</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0596__section19463015192"><h4 class="sectiontitle">Log Level</h4><p id="mrs_01_0596__p87214571397"><a href="#mrs_01_0596__table63318572917">Table 2</a> describes the log levels supported by Flink. The priorities of log levels are ERROR, WARN, INFO, and DEBUG in descending order. Logs whose levels are higher than or equal to the specified level are printed. The number of printed logs decreases as the specified log level increases.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0596__table63318572917"></a><a name="table63318572917"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0596__table63318572917" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Log levels</caption><thead align="left"><tr id="mrs_01_0596__row12721657892"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.2.3.2.3.1.1"><p id="mrs_01_0596__p1202134412918"><strong id="mrs_01_0596__b2077582416129">Level</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="70%" id="mcps1.3.2.3.2.3.1.2"><p id="mrs_01_0596__p3202544891"><strong id="mrs_01_0596__b3913152661211">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0596__row107314575911"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_0596__p57314576913">ERROR</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_0596__p117385719912">Error information about the current event processing</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0596__row0733571398"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_0596__p1731557595">WARN</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_0596__p1273185720917">Exception information about the current event processing</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0596__row4733571694"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_0596__p9737578914">INFO</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_0596__p073155711910">Normal running status information about the system and events</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0596__row177320571996"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_0596__p197319571492">DEBUG</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_0596__p137345716911">System information and system debugging information</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_0596__p4731157197">To modify log levels, perform the following steps:</p>
|
|
<ol id="mrs_01_0596__ol2049469418545"><li id="mrs_01_0596__li4027424618545"><span>Go to the <strong id="mrs_01_0596__b1037082416568">All Configurations</strong> page of Flink by referring to <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.</span></li><li id="mrs_01_0596__li2692390118545"><span>On the menu bar on the left, select the log menu of the target role.</span></li><li id="mrs_01_0596__li4098852518545"><span>Select a desired log level.</span></li><li id="mrs_01_0596__li1719026918545"><span>Save the configuration. In the displayed dialog box, click <strong id="mrs_01_0596__b176757401010724">OK</strong> to make the configurations take effect.</span></li></ol>
|
|
<div class="note" id="mrs_01_0596__note173412056141514"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0596__ul210713781611"><li id="mrs_01_0596__li81071377169">After the configuration is complete, you do not need to restart the service. Download the client again for the configuration to take effect.</li><li id="mrs_01_0596__li1710711712165">You can also change the configuration items corresponding to the log level in <strong id="mrs_01_0596__b1841434619194">log4j-cli.properties</strong>, <strong id="mrs_01_0596__b6947105616192">log4j.properties</strong>, and <strong id="mrs_01_0596__b15525182112015">log4j-session.properties</strong> in <strong id="mrs_01_0596__b987411081919">/opt</strong><strong id="mrs_01_0596__b1087420191918"></strong><strong id="mrs_01_0596__b18874120181911">/client/Flink/flink/conf/</strong> on the client. <strong id="mrs_01_0596__b892019374528">/opt</strong><strong id="mrs_01_0596__b792053765210"></strong><strong id="mrs_01_0596__b59216374529">/client</strong> is the client installation directory.</li><li id="mrs_01_0596__li19676384546">When a job is submitted using a client, a log file is generated in the <strong id="mrs_01_0596__b1853524735712">log</strong> folder on the client. The default umask value is <strong id="mrs_01_0596__b5858632145817">0022</strong>. Therefore, the default log permission is <strong id="mrs_01_0596__b1777564295817">644</strong>. To change the file permission, you need to change the umask value. For example, to change the umask value of user <strong id="mrs_01_0596__b65529544582">omm</strong>:<ul id="mrs_01_0596__ul740172651510"><li id="mrs_01_0596__li152762258151">Add <strong id="mrs_01_0596__b14487184917597">umask 0026</strong> to the end of the <strong id="mrs_01_0596__b153401850115210">/home/omm/.baskrc</strong> file.</li><li id="mrs_01_0596__li16762182815165">Run the <strong id="mrs_01_0596__b19541527013">source /home/omm/.baskrc</strong> command to make the file permission take effect.</li></ul>
|
|
</li></ul>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_0596__section1435319261199"><h4 class="sectiontitle">Log Format</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0596__table131491213116" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Log formats</caption><thead align="left"><tr id="mrs_01_0596__row12149112191110"><th align="left" class="cellrowborder" valign="top" width="11.981198119811982%" id="mcps1.3.3.2.2.4.1.1"><p id="mrs_01_0596__p683323501119"><strong id="mrs_01_0596__b1099664183114">Type</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="28.612861286128616%" id="mcps1.3.3.2.2.4.1.2"><p id="mrs_01_0596__p13833173514110"><strong id="mrs_01_0596__b1765264416206">Format</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="59.4059405940594%" id="mcps1.3.3.2.2.4.1.3"><p id="mrs_01_0596__p183373591114"><strong id="mrs_01_0596__b164921147132015">Example</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0596__row181507210112"><td class="cellrowborder" valign="top" width="11.981198119811982%" headers="mcps1.3.3.2.2.4.1.1 "><p id="mrs_01_0596__p99601845151110">Run log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.612861286128616%" headers="mcps1.3.3.2.2.4.1.2 "><p id="mrs_01_0596__p10960114516118"><<em id="mrs_01_0596__i182715010188">yyyy-MM-dd HH:mm:ss,SSS</em>>|<<em id="mrs_01_0596__i3830950131813">Log level</em>>|<<em id="mrs_01_0596__i1083045061814">Name of the thread that generates the log</em>>|<<em id="mrs_01_0596__i9830175041813">Message in the log</em>>|<<em id="mrs_01_0596__i1683014509189">Location where the log event occurs</em>></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.4059405940594%" headers="mcps1.3.3.2.2.4.1.3 "><p id="mrs_01_0596__p996012457115">2019-06-27 21:30:31,778 | INFO | [flink-akka.actor.default-dispatcher-3] | TaskManager container_e10_1498290698388_0004_02_000007 has started. | org.apache.flink.yarn.YarnFlinkResourceManager (FlinkResourceManager.java:368)</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0591.html">Using Flink</a></div>
|
|
</div>
|
|
</div>
|
|
|