doc-exports/docs/cce/umn/cce_10_0555.html

<a name="cce_10_0555"></a><a name="cce_10_0555"></a>

<h1 class="topictitle1">Collecting Container Logs Using the Cloud Native Log Collection Add-on</h1>
<div id="body0000001757561941"><p id="cce_10_0555__p1525318481827">The Cloud Native Log Collection add-on (<a href="cce_10_0416.html">Cloud Native Log Collection</a>) is developed based on Fluent Bit and OpenTelemetry for collecting logs and Kubernetes events. This add-on supports CRD-based log collection policies. It collects and forwards stdout logs, container file logs, node logs, and Kubernetes events in a cluster based on configured policies.</p>
<div class="section" id="cce_10_0555__section1995018241214"><h4 class="sectiontitle">Notes and Constraints</h4><ul id="cce_10_0555__ul38996572032"><li id="cce_10_0555__li1437118282351">Up to 100 log rules can be created for each cluster.</li><li id="cce_10_0555__li3899057134">The Cloud Native Log Collection add-on cannot collect .gz, .tar, and .zip logs and cannot access symbolic links of logs.</li><li id="cce_10_0555__li48992575320">If the node <a href="cce_10_0341.html#cce_10_0341__section1473612279214">storage driver</a> is Device Mapper, container file logs must be collected from the path where the data disk is attached to the node.</li><li id="cce_10_0555__li789910571632">If the container runtime is containerd, each stdout log cannot be in multiple lines. (This does not apply to the Cloud Native Log Collection add-on of version 1.3.0 or later.)</li><li id="cce_10_0555__li1319974510478">If a volume is mounted to the directory of a service container, this add-on cannot collect data from the parent directory. In this case, you need to configure a complete data directory.</li><li id="cce_10_0555__li930217527219">If the lifetime of a container is less than 1 minute, logs cannot be collected in a timely manner. As a result, logs may be lost.</li><li id="cce_10_0555__li331118441166">When you create a log stream using a collection policy, ensure that the log stream name is unique because LTS does not allow log streams with the same name (both before and after renaming), even in different log groups.</li></ul>
</div>
<div class="section" id="cce_10_0555__section53625241484"><h4 class="sectiontitle">Enabling Logging on the Console</h4><ol id="cce_10_0555__ol7318196796"><li id="cce_10_0555__li03181161493"><span>Enable logging.</span><p><p id="cce_10_0555__p18411191814578"><strong id="cce_10_0555__b7437453122312">Enabling logging during cluster creation</strong></p>
<ol type="a" id="cce_10_0555__ol183617113584"><li id="cce_10_0555__li9293104917265">Log in to the <span id="cce_10_0555__cce_10_0047_ph1519791153812">CCE console</span>.</li><li id="cce_10_0555__li1283621110581">Click <strong id="cce_10_0555__b250511268500">Create Cluster</strong>.</li><li id="cce_10_0555__li1076913581016">Configure the parameters by referring to <a href="cce_10_0028.html">Creating a CCE Standard/Turbo Cluster</a>. Then, click <span class="uicontrol" id="cce_10_0555__uicontrol133868598013"><b>Next: Select Add-on</b></span> in the lower right corner.</li><li id="cce_10_0555__li163681377132">On the <strong id="cce_10_0555__b553513241554">Select Add-on</strong> page, select <strong id="cce_10_0555__b25361624053">Cloud Native Log Collection</strong>.</li><li id="cce_10_0555__en-us_topic_0000001757561941_li1483631175820">Click <strong id="cce_10_0555__b1787703317193">Next: Configure Add-on</strong> in the lower right corner and select the required logs.<ul id="cce_10_0555__ul15382152171319"><li id="cce_10_0555__li11382142171315">Container logs: A log collection policy named <strong id="cce_10_0555__b665404815196">default-stdout</strong> will be created, and stdout logs in all namespaces will be reported to LTS.</li><li id="cce_10_0555__li03821021171318">Kubernetes events: A log collection policy named <strong id="cce_10_0555__b163281917172013">default-event</strong> will be created, and Kubernetes events in all namespaces will be reported to LTS.</li></ul>
</li><li id="cce_10_0555__en-us_topic_0000001757561941_li19681527171613">Click <strong id="cce_10_0555__b205211845354">Next: Confirm Settings</strong> in the lower right corner. On the displayed page, click <strong id="cce_10_0555__b55217452057">Submit</strong>.</li></ol>
</p></li><li id="cce_10_0555__li193114172910"><span>View and configure log collection policies.</span><p><ol type="a" id="cce_10_0555__ol1431973318574"><li id="cce_10_0555__li1931973335715">In the navigation pane, choose <strong id="cce_10_0555__b5928138195013"><span id="cce_10_0555__text10928178155014">Logging</span></strong>.</li><li id="cce_10_0555__li10603839205711"><p id="cce_10_0555__p2604131442418"><a name="cce_10_0555__li10603839205711"></a><a name="li10603839205711"></a>Click <strong id="cce_10_0555__b17604111432413">View Log Collection Policies</strong> in the upper right corner.</p>
<p id="cce_10_0555__li10603839205711p0">All log collection policies reported to LTS are displayed.</p>
</li><li id="cce_10_0555__li1876513521447">Click <strong id="cce_10_0555__b14931123151011">Create Log Collection Policy</strong>.<div class="note" id="cce_10_0555__note9771843165312"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="cce_10_0555__ul970311961213"><li id="cce_10_0555__li1676544725820">To keep the logs organized, you are advised to select different log streams for different types of logs when configuring the log collection policies.</li><li id="cce_10_0555__li87655474588"><a name="cce_10_0555__li87655474588"></a><a name="li87655474588"></a>The following are requirements for configuring the container and node file log paths:<ul id="cce_10_0555__ul10247101135917"><li id="cce_10_0555__li170317918123">Log directory: Enter an absolute path, for example, <strong id="cce_10_0555__b643314810145">/log</strong>. The path must start with a slash (/) and contain a maximum of 512 characters. Only uppercase letters, lowercase letters, digits, hyphens (-), underscores (_), slashes (/), asterisks (*), and question marks (?) are allowed.</li><li id="cce_10_0555__li370349101210">Log file name: It can contain only uppercase letters, lowercase letters, digits, hyphens (-), underscores (_), asterisks (*), question marks (?), and periods (.). Logs in the format of .gz, .tar, and .zip are not supported.</li></ul>
<p id="cce_10_0555__p18533101418205">The directory and file names must be complete and support asterisks (*) and question marks (?) as wildcards. A maximum of three levels of directories can be matched using wildcards. The level-1 directory does not support wildcards. An asterisk (*) can match multiple characters. A question mark (?) can match only one character. For example:</p>
<ul id="cce_10_0555__ul1272693314525"><li id="cce_10_0555__li19409105316435">If the directory is <strong id="cce_10_0555__b1629982310212">/var/logs/*</strong> and the file name is <strong id="cce_10_0555__b2426182882119">*.log</strong>, the match expression is <strong id="cce_10_0555__b19572195941013">/var/logs/*/*.log</strong>, indicating that any files with the extension .log in all level-1 directories in the <strong id="cce_10_0555__b111644610221">/var/logs</strong> directory are matched. Note that this expression cannot match any files with the extension .log in the <strong id="cce_10_0555__b3668181723119">/var/logs</strong> directory and multi-level directories in the <strong id="cce_10_0555__b121751355241">/var/logs</strong> directory.</li><li id="cce_10_0555__li572643318529">If the directory is <strong id="cce_10_0555__b152661611261">/var/logs/app_*</strong> and the file name is <strong id="cce_10_0555__b1252610167265">*.log</strong>, any log files with the extension .log in all directories that match <strong id="cce_10_0555__b18446154735913">app_*</strong> in the <strong id="cce_10_0555__b15526171652614">/var/logs</strong> directory will be reported.</li></ul>
</li></ul>
</div></div>

<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="cce_10_0555__table1069321010618" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters of a custom policy</caption><thead align="left"><tr id="cce_10_0555__row1469317101963"><th align="left" class="cellrowborder" valign="top" id="mcps1.3.3.2.2.2.1.3.3.2.6.1.1"><p id="cce_10_0555__p9693101011611">Parameter</p>
</th>
<th align="left" class="cellrowborder" colspan="4" valign="top" id="mcps1.3.3.2.2.2.1.3.3.2.6.1.2"><p id="cce_10_0555__p46931110961">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="cce_10_0555__row669314101768"><td class="cellrowborder" valign="top" width="8%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.1 "><p id="cce_10_0555__p169316101963">Log Type</p>
</td>
<td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><p id="cce_10_0555__p5693610564"><strong id="cce_10_0555__b20259161811450">Container standard output</strong>: used to collect container stdout logs. You can create a log collection policy by namespace, workload name, or instance label.</p>
</td>
<td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><p id="cce_10_0555__p6693010767"><strong id="cce_10_0555__b174211216144512">Container file log</strong>: used to collect text logs. You can specify a workload or instance label to create a log collection policy.</p>
</td>
<td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><p id="cce_10_0555__p369317101617"><strong id="cce_10_0555__b186410191176">Node file log</strong>: used to collect logs from a node. Only one file path can be configured for a log collection policy.</p>
</td>
<td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><p id="cce_10_0555__p1930312593485"><strong id="cce_10_0555__b1175692611177">Kubernetes Events</strong>: used to collect Kubernetes events. You can configure collection policies by namespace.</p>
</td>
</tr>
<tr id="cce_10_0555__row1869371011614"><td class="cellrowborder" valign="top" width="8%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.1 "><p id="cce_10_0555__p96934102619">Log Source</p>
</td>
<td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><ul id="cce_10_0555__ul1469317101666"><li id="cce_10_0555__li1769314101061"><strong id="cce_10_0555__b28141700">All containers</strong>: You can specify all containers in a namespace. If this parameter is not specified, logs of containers in all namespaces will be collected.</li><li id="cce_10_0555__li1693151015616"><strong id="cce_10_0555__b175575866">Workload</strong>: You can specify a workload and its containers. If this parameter is not specified, logs of all containers running the workload will be collected.</li><li id="cce_10_0555__li06938104617"><strong id="cce_10_0555__b318252200">Workload with target label</strong>: You can specify a workload by label and its containers. If this parameter is not specified, logs of all containers running the workload will be collected.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><ul id="cce_10_0555__ul269381019616"><li id="cce_10_0555__li16933101664"><strong id="cce_10_0555__b1547376853">Workload</strong>: You can specify a workload and its containers. If this parameter is not specified, logs of all containers running the workload will be collected.</li><li id="cce_10_0555__li1669311101161"><strong id="cce_10_0555__b1750190740">Workload with target label</strong>: You can specify a workload by label and its containers. If this parameter is not specified, logs of all containers running the workload will be collected.</li></ul>
<p id="cce_10_0555__p16693171011611">You also need to specify the log collection path. For details, see the <a href="#cce_10_0555__li87655474588">log path configuration requirements</a>.</p>
</td>
<td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><p id="cce_10_0555__p116933101664"><strong id="cce_10_0555__b16893544192019">Collection Path</strong>: used to configure the log collection path. For details, see the <a href="#cce_10_0555__li87655474588">log path configuration requirements</a>.</p>
</td>
<td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><p id="cce_10_0555__p230325994811">You can specify a namespace whose Kubernetes events are to be collected. If no namespace is specified, Kubernetes events in all namespaces are collected.</p>
</td>
</tr>
<tr id="cce_10_0555__row769351010619"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.1 "><p id="cce_10_0555__p269491010614">Log Format</p>
</td>
<td class="cellrowborder" colspan="4" valign="top" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><ul id="cce_10_0555__ul869412103615"><li id="cce_10_0555__li86941101269"><strong id="cce_10_0555__b1045211673">Single-line</strong><p id="cce_10_0555__p869431018612">Each log contains only one line of text. The newline character \n denotes the start of a new log.</p>
</li><li id="cce_10_0555__li369420105617"><strong id="cce_10_0555__b2010611684">Multi-line</strong><p id="cce_10_0555__p1969418108616">Some programs (for example, Java program) print a log that occupies multiple lines. By default, logs are collected by line. If you want to display logs as a single message, you can enable multi-line logging and use the regular pattern. When you select <strong id="cce_10_0555__b1869790206">Multi-line</strong>, configure <strong id="cce_10_0555__b342683931">Log Matching Format</strong>.</p>
<p id="cce_10_0555__p8694510161">For example, if logs need to be collected by line and each log starts with a date and occupies three lines, you can set <strong id="cce_10_0555__b1252115195">Log Matching Format</strong> to the regular expression of the date, for example, <strong id="cce_10_0555__b1072582879">\d{4}-\d{2}-\d{2} \d{2}\:\d{2}\:\d{2}.*</strong>.</p>
<div class="p" id="cce_10_0555__p369418101463">The three lines starting with the date are regarded as a log.<pre class="screen" id="cce_10_0555__screen16694191014611">2022-01-01 00:00:00 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
at com.myproject.module.MyProject.badMethod(MyProject.java:22)
at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)</pre>
</div>
</li></ul>
<p id="cce_10_0555__p1246585017714"></p>
</td>
</tr>
<tr id="cce_10_0555__row1369491011610"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.1 "><p id="cce_10_0555__p56941410768">LTS Collection</p>
</td>
<td class="cellrowborder" colspan="4" valign="top" headers="mcps1.3.3.2.2.2.1.3.3.2.6.1.2 "><p id="cce_10_0555__p1469414108617">This parameter is used to configure the log group and log stream for log reporting.</p>
<ul id="cce_10_0555__ul8694310061"><li id="cce_10_0555__li269410109613"><strong id="cce_10_0555__b1547792719220">Centralized</strong>: The default log group (<strong id="cce_10_0555__b1947711274227">k8s-log-</strong><em id="cce_10_0555__i16477727132215">{Cluster ID}</em>) and default log stream (<strong id="cce_10_0555__b4477172792213">stdout-</strong><em id="cce_10_0555__i6477227132215">{Cluster ID}</em>) are automatically selected.</li><li id="cce_10_0555__li1969431013614"><strong id="cce_10_0555__b6140638191813">Custom</strong>: Select a log group and log stream from the drop-down list.<ul id="cce_10_0555__ul26946100611"><li id="cce_10_0555__li169411107614"><strong id="cce_10_0555__b19866758122313">Log Group</strong>: A log group is the basic unit for LTS to manage logs. If you do not have a log group, CCE prompts you to create one. The default name is <strong id="cce_10_0555__b1379904026">k8s-log-</strong><em id="cce_10_0555__i513473411">{Cluster ID}</em>, for example, <strong id="cce_10_0555__b337850143">k8s-log-bb7eaa87-07dd-11ed-ab6c-0255ac1001b3</strong>.</li><li id="cce_10_0555__li1269415102611"><strong id="cce_10_0555__b2144999192">Log Stream</strong>: A log stream is the basic unit for reading and writing logs. You can put different types of logs into different streams to ease management. When you install the add-on or create a log collection policy based on the policy template, the following log streams are automatically created:<p id="cce_10_0555__p2694131010611">- <strong id="cce_10_0555__b1891398358">stdout-</strong><em id="cce_10_0555__i1401683398">{Cluster ID}</em> for container logs, for example, <strong id="cce_10_0555__b316977623">stdout-bb7eaa87-07dd-11ed-ab6c-0255ac1001b3</strong></p>
<p id="cce_10_0555__p116946104619">- <strong id="cce_10_0555__b1975377124">event-</strong><em id="cce_10_0555__i2026573786">{Cluster ID}</em> for Kubernetes events, for example, <strong id="cce_10_0555__b2006547906">event-bb7eaa87-07dd-11ed-ab6c-0255ac1001b3</strong></p>
</li></ul>
</li></ul>
<p id="cce_10_0555__p12465650672"></p>
</td>
</tr>
</tbody>
</table>
</div>
</li></ol>
</p></li><li id="cce_10_0555__li1174231917561"><span>View the logs.</span><p><ol type="a" id="cce_10_0555__ol260314783813"><li id="cce_10_0555__li166037715381">In the navigation pane, choose <strong id="cce_10_0555__b142741126508">Logging</strong>.</li><li id="cce_10_0555__li1679622425113">View different types of logs:</li></ol>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="cce_10_0182.html">Collecting Container Logs</a></div>
</div>
</div>