doc-exports/docs/cce/umn/cce_10_0406.html

<a name="cce_10_0406"></a><a name="cce_10_0406"></a>

<h1 class="topictitle1">Cloud Native Cluster Monitoring</h1>
<div id="body0000001343671985"><div class="section" id="cce_10_0406__section25311744154917"><h4 class="sectiontitle">Introduction</h4><p id="cce_10_0406__p108091025011">The Cloud Native Cluster Monitoring add-on (formerly kube-prometheus-stack) uses Prometheus-operator and Prometheus and provides easy-to-use, end-to-end Kubernetes cluster monitoring.</p>
<p id="cce_10_0406__p1062083475716"><strong id="cce_10_0406__b449618166517">Open-source community</strong>: <a href="https://github.com/prometheus/prometheus" target="_blank" rel="noopener noreferrer">https://github.com/prometheus/prometheus</a></p>
</div>
<div class="section" id="cce_10_0406__section199451341919"><h4 class="sectiontitle">Notes and Constraints</h4><ul id="cce_10_0406__ul1269514507288"><li id="cce_10_0406__li11695195062815">By default, the kube-state-metrics component of the add-on does not collect labels and annotations of Kubernetes resources. To collect these labels and annotations, manually enable the collection function in the startup parameters and check whether the corresponding metrics are added to the collection trustlist of ServiceMonitor named <strong id="cce_10_0406__b20833715191811">kube-state-metrics</strong>. For details, see <a href="#cce_10_0406__section163751639151115">Collecting All Labels and Annotations of a Pod</a>.</li><li id="cce_10_0406__li167721419155620">In 3.8.0 and later versions, component metrics in the kube-system and monitoring namespaces are not collected by default. If you have workloads in the two namespaces, use <a href="cce_10_0373.html#cce_10_0373__section3497144813213">Pod Monitor</a> or <a href="cce_10_0373.html#cce_10_0373__section92241661317">Service Monitor</a> to collect these metrics.</li><li id="cce_10_0406__li3318171215615">In 3.8.0 and later versions, etcd-server, kube-controller, kube-scheduler, autoscaler, fluent-bit, volcano-agent, volcano-scheduler and otel-collector metrics are not collected by default. Enable the collection as required.<p id="cce_10_0406__p206018134568"><a name="cce_10_0406__li3318171215615"></a><a name="li3318171215615"></a>To enable this function, on the <strong id="cce_10_0406__b1134072516416">ConfigMaps and Secrets</strong> page, expand the dropdown list of <strong id="cce_10_0406__b729293216428">Namespace</strong>, and select <strong id="cce_10_0406__b152291534104213">monitoring</strong>. Locate the row that contains the configuration item named <strong id="cce_10_0406__b12914546184310">persistent-user-config</strong>, and click <strong id="cce_10_0406__b956191411438">Edit YAML</strong> in the operation column. Remove the <strong id="cce_10_0406__b1471421564716">serviceMonitorDisable</strong> or <strong id="cce_10_0406__b14901021154711">podMonitorDisable</strong> configuration in the <strong id="cce_10_0406__b72011251476">customSettings</strong> field as required or set the configuration to an empty array.</p>
<pre class="screen" id="cce_10_0406__screen1890358131817">...
   customSettings:
      podMonitorDisable: []
      serviceMonitorDisable: []</pre>
</li></ul>
</div>
<div class="section" id="cce_10_0406__section158021093142"><h4 class="sectiontitle">Permissions</h4><p id="cce_10_0406__p125082050122613">The node-exporter component of the Cloud Native Cluster Monitoring add-on needs to read the Docker info data from the <strong id="cce_10_0406__b242316206214">/var/run/docker.sock</strong> directory on the host for monitoring the Docker disk space.</p>
<p id="cce_10_0406__p145091350122610">The following permission is required for running node-exporter:</p>
<ul id="cce_10_0406__ul117815436270"><li id="cce_10_0406__li20178743182710">cap_dac_override: reads the Docker info data.</li></ul>
</div>
<div class="section" id="cce_10_0406__section186134814119"><a name="cce_10_0406__section186134814119"></a><a name="section186134814119"></a><h4 class="sectiontitle">Installing the Add-on</h4><div class="note" id="cce_10_0406__note152084181520"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="cce_10_0406__p1752024116150">The Cloud Native Cluster Monitoring add-on automatically selects a deployment mode based on <strong id="cce_10_0406__b1389013472818"><a href="#cce_10_0406__li15556183414307">Data Storage Configuration</a></strong>. This is supported by Cloud Native Cluster Monitoring 3.7.1 or later.</p>
<ul id="cce_10_0406__ul128792089306"><li id="cce_10_0406__li6656205432717">Original agent mode: Disable <strong id="cce_10_0406__b20127191055812">Local Data Storage</strong> and enable at least one of <strong id="cce_10_0406__b8127210135811">Report Monitoring Data to AOM</strong> and <strong id="cce_10_0406__b111271510125810">Report Monitoring Data to a Third-Party Platform</strong>.</li></ul>
<ul id="cce_10_0406__ul565575402715"><li id="cce_10_0406__li5655175432710">Original server mode: Enable <strong id="cce_10_0406__b10938825102619">Local Data Storage</strong> and <strong id="cce_10_0406__b59381525132612">Report Monitoring Data to AOM</strong> or <strong id="cce_10_0406__b69384256265">Report Monitoring Data to a Third-Party Platform</strong>.</li></ul>
</div></div>
<ol id="cce_10_0406__ol9183433182510"><li id="cce_10_0406__li330462393220"><span>Log in to the <span id="cce_10_0406__cce_10_0004_ph18314322182">CCE console</span> and click the cluster name to access the cluster console.</span></li><li id="cce_10_0406__li13183153352515"><span>In the navigation pane, choose <strong id="cce_10_0406__b51842425265"><span id="cce_10_0406__text51842042102610">Add-ons</span></strong>. Locate <strong id="cce_10_0406__b0184194210265">Cloud Native Cluster Monitoring</strong> on the right and click <span class="uicontrol" id="cce_10_0406__uicontrol51841642172619"><b>Install</b></span>.</span></li><li id="cce_10_0406__li15556183414307"><a name="cce_10_0406__li15556183414307"></a><a name="li15556183414307"></a><span>On the <strong id="cce_10_0406__b1813011813218">Install Add-on</strong> page, enable at least one item in the <span class="uicontrol" id="cce_10_0406__uicontrol738210234336"><b>Data Storage Configuration</b></span> area.</span><p><ul id="cce_10_0406__ul14526143113393"><li id="cce_10_0406__li953119336397"><strong id="cce_10_0406__b143131455124117">Report Monitoring Data to AOM</strong>: Report Prometheus data to AOM. After this function is enabled, you can select the corresponding AOM instance. The collected basic metrics are free of charge. Custom metrics are charged by AOM. To interconnect with AOM, you must have certain permissions. Only <strong id="cce_10_0406__b75491191223">users in the </strong><strong id="cce_10_0406__b1254918920225">admin</strong><strong id="cce_10_0406__b954914912224"> user group</strong> can perform this operation.</li><li id="cce_10_0406__li2526203153919"><strong id="cce_10_0406__b5956191316337">Reporting Monitoring Data to a Third-Party Monitoring Platform</strong>: To report Prometheus data to a third-party monitoring system, you need to enter the address and token of the third-party monitoring system and determine whether to skip certificate authentication.</li><li id="cce_10_0406__li185331058123918"><strong id="cce_10_0406__b108317571412">Local Data Storage</strong>: Select the type and size of a disk for storing monitoring data to store Prometheus data in PVCs in the cluster. <strong id="cce_10_0406__b12961482422">Storage volumes are not deleted along with the add-on.</strong> If <strong id="cce_10_0406__b7247641124717">Local Data Storage</strong> is enabled, all components will be deployed. For details, see <a href="#cce_10_0406__section0377457163618">Components</a>.<div class="note" id="cce_10_0406__note59616874216"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="cce_10_0406__p1197168134212">An available PVC named <strong id="cce_10_0406__b1753452513212">pvc-prometheus-server-0</strong> exists in namespace <strong id="cce_10_0406__b12534925122110">monitoring</strong> and will be used as the storage source.</p>
</div></div>
</li></ul>
</p></li><li id="cce_10_0406__li184381758104315"><span>Configure the add-on <span class="uicontrol" id="cce_10_0406__uicontrol04381658154315"><b>specifications</b></span> as needed.</span><p><ul id="cce_10_0406__ul13438195818437"><li id="cce_10_0406__li654520301400"><strong id="cce_10_0406__b13567194814118">Add-on Specifications</strong>:<ul id="cce_10_0406__ul18716136164011"><li id="cce_10_0406__li17438758184312">If you selected <span class="uicontrol" id="cce_10_0406__uicontrol17771938154414"><b>Preset</b></span>, the system will configure the number of pods and resource quotas for the add-on based on the preset specifications. You can see the configurations on the console.</li><li id="cce_10_0406__li2438145819439">If you selected <strong id="cce_10_0406__b153325156444">Custom</strong>, you can adjust the number of pods and resource quotas as needed. High availability is not possible with a single pod. If an error occurs on the node where the add-on instance runs, the add-on will fail.</li></ul>
</li><li id="cce_10_0406__li1613154094010"><strong id="cce_10_0406__b19609253163415">Prometheus HA</strong>: The Prometheus-server, Prometheus-operator, thanos-query, custom-metrics-apiserver, alertmanager, and kube-state-metrics components are deployed in multi-instance mode in the cluster.</li><li id="cce_10_0406__li4894113217440"><strong id="cce_10_0406__b2026818113613">Number of collected shards</strong> (available after <strong id="cce_10_0406__b1184192683718">Local Data Storage</strong> is disabled): When there is a lot of Prometheus data, you can configure this parameter to spread the data across a specific number of Prometheus instances. This will help with storage and querying. Increasing the number of shards reduces the data volume carried by each shard. This can increase the upper limit of the metric collection throughput and cause more resources to be consumed. By default, CCE automatically determines the number of shards based on the cluster scale. It is advised to allocate one shard for every 50 nodes. If you want to enhance collection performance by increasing the number of shards, carefully assess resource utilization and optimize based on your monitoring needs. To ensure system stability, keep the master node's memory usage below 50%.</li></ul>
</p></li><li id="cce_10_0406__li1066432619340"><span>Configure the add-on parameters.</span><p><ul id="cce_10_0406__ul4665523123418"><li id="cce_10_0406__li164964410414"><strong id="cce_10_0406__b41959918389">Custom Metric Collection</strong>: Application metrics are automatically collected in the form of service discovery. After this function is enabled, you need to add related configurations to the target application. For details, see <a href="cce_10_0373.html">Monitoring Custom Metrics Using Cloud Native Cluster Monitoring</a>.</li><li id="cce_10_0406__li13450171664216"><strong id="cce_10_0406__b14548610163818">Collection Interval</strong>: Configure the collection interval.</li><li id="cce_10_0406__li1545010161428"><strong id="cce_10_0406__b17450151616423">Data Retention</strong> (available only after <span class="uicontrol" id="cce_10_0406__uicontrol114365454479"><b>Local Data Storage</b></span> is enabled): Configure how long the monitoring data can be kept.</li><li id="cce_10_0406__li136161884431"><strong id="cce_10_0406__b2046161374315">node-exporter Listening Port</strong>: This port uses the host network to listen to and expose metrics on the node for Prometheus collection. The default port number is 9100, but it can be modified if there is a conflict with an existing port.</li><li id="cce_10_0406__li233584994219"><strong id="cce_10_0406__b13266110164216">Scheduling Policies</strong>: Support node affinity, taint, and tolerations. Multiple scheduling policies can be configured. If no affinity node label key or toleration node taint key is configured, this function is disabled by default.<ul id="cce_10_0406__ul1133564974214"><li id="cce_10_0406__li173357494428"><strong id="cce_10_0406__b5913204944510">Range</strong>: You can select the add-on pods for which the scheduling policy takes effect. By default, the scheduling policy takes effect for all pods. If a pod is specified, the scheduling policies configured for all pods are overwritten.</li><li id="cce_10_0406__li5335144964210"><strong id="cce_10_0406__b20434135974612">Affinity Node Label Key</strong>: Enter a node label key to set node affinity for the add-on pods.</li><li id="cce_10_0406__li633574915422"><strong id="cce_10_0406__b1985725164710">Affinity Node Label Value</strong>: Enter a node label value to set node affinity for the add-on pods.</li><li id="cce_10_0406__li203352049154217"><strong id="cce_10_0406__b1716103213428">Toleration Node Taint Key</strong>: A component can be scheduled to a node that has the taint key you specified.</li></ul>
</li></ul>
</p></li><li id="cce_10_0406__li9455819152615"><span>Click <strong id="cce_10_0406__b93619174715">Install</strong>.</span><p><p id="cce_10_0406__p1554916492396">After the add-on is installed, you may need to perform the following operations:</p>
<ul id="cce_10_0406__ul16483117104016"><li id="cce_10_0406__li98191651463">To use custom metrics to create an auto scaling policy, ensure that <strong id="cce_10_0406__b35111354815">Local Data Storage</strong> is enabled for the add-on and then take the following steps:<ol type="a" id="cce_10_0406__ol36708714713"><li id="cce_10_0406__li1367027979">Collect custom metrics reported by applications to Prometheus. For details, see <a href="cce_10_0373.html">Monitoring Custom Metrics Using Cloud Native Cluster Monitoring</a>.</li><li id="cce_10_0406__li1567012717715">Aggregate these custom metrics collected by Prometheus to the API server for the HPA policy to use. For details, see <a href="#cce_10_0406__section11927514174016">Creating an HPA Policy Using Custom Metrics</a>.</li></ol>
</li><li id="cce_10_0406__li1648367124015">To provide system resource metrics (such as CPU and memory usage) for workload auto scaling using this add-on, ensure that <strong id="cce_10_0406__b26891447124816">Local Data Storage</strong> is enabled for the add-on and then enable the Metrics API. For details, see <a href="#cce_10_0406__section17830202915211">Providing Basic Resource Metrics Through the Metrics API</a>. After the configuration, you can use Prometheus to collect system resource metrics. (This operation is not recommended because it may conflict with the Kubernetes Metric Server add-on.)</li></ul>
</p></li></ol>
</div>
<div class="section" id="cce_10_0406__section0377457163618"><a name="cce_10_0406__section0377457163618"></a><a name="section0377457163618"></a><h4 class="sectiontitle">Components</h4><p id="cce_10_0406__p5684174553719">All Kubernetes resources created during Cloud Native Cluster Monitoring add-on installation are created in the namespace named <strong id="cce_10_0406__b107961646279">monitoring</strong>.</p>

<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="cce_10_0406__table1965341035819" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Add-on components</caption><thead align="left"><tr id="cce_10_0406__row1565319102582"><th align="left" class="cellrowborder" valign="top" width="23%" id="mcps1.3.5.3.2.5.1.1"><p id="cce_10_0406__p14653141018584">Component</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.5.3.2.5.1.2"><p id="cce_10_0406__p065391025820">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="15%" id="mcps1.3.5.3.2.5.1.3"><p id="cce_10_0406__p136533107584">Supported Deployment Mode</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="12%" id="mcps1.3.5.3.2.5.1.4"><p id="cce_10_0406__p5653111015587">Resource Type</p>
</th>
</tr>
</thead>
<tbody><tr id="cce_10_0406__row872889165919"><td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.5.3.2.5.1.1 "><p id="cce_10_0406__p47281897596">prometheusOperator</p>
<p id="cce_10_0406__p810412717416">(workload name: prometheus-operator)</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.2.5.1.2 "><p id="cce_10_0406__p14728149195917">The control center of the entire system. It deploys and manages Prometheus server based on Prometheus CustomResourceDefinitions (CRDs), and monitors and processes the changes of these CRDs.</p>
</td>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.5.3.2.5.1.3 "><p id="cce_10_0406__p472814975920">All</p>
</td>
<td class="cellrowborder" valign="top" width="12%" headers="mcps1.3.5.3.2.5.1.4 "><p id="cce_10_0406__p772869115917">Deployment</p>
</td>
</tr>
<tr id="cce_10_0406__row2653710135812"><td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.5.3.2.5.1.1 "><p id="cce_10_0406__p03061232856">prometheus</p>
<p id="cce_10_0406__p18653201095818">(If the local data storage is used, the workload name is <strong id="cce_10_0406__b2814438194117">prometheus-server</strong>. If the local data storage is not used, the workload name is <strong id="cce_10_0406__b98521645154115">prometheus-lightweight</strong>.)</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.2.5.1.2 "><p id="cce_10_0406__p246914124816">The core component of the Prometheus monitoring system. It is used to collect and store metrics and report the metrics to AOM or a third-party monitoring platform. If local storage is enabled, the metric data can be stored in local PVCs.</p>
</td>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.5.3.2.5.1.3 "><p id="cce_10_0406__p335010301130">All</p>
</td>
<td class="cellrowborder" valign="top" width="12%" headers="mcps1.3.5.3.2.5.1.4 "><p id="cce_10_0406__p365411016585">StatefulSet</p>
</td>
</tr>
<tr id="cce_10_0406__row365413102582"><td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.5.3.2.5.1.1 "><p id="cce_10_0406__p96548103582">alertmanager</p>
<p id="cce_10_0406__p1018293419516">(workload name: alertmanager-alertmanager)</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.2.5.1.2 "><p id="cce_10_0406__p15654201019583">Alarm center of the add-on. It receives alarms sent by Prometheus and manages alarm information by deduplicating, grouping, and distributing.</p>
</td>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.5.3.2.5.1.3 "><p id="cce_10_0406__p1765431025818">Local data storage enabled</p>
</td>
<td class="cellrowborder" valign="top" width="12%" headers="mcps1.3.5.3.2.5.1.4 "><p id="cce_10_0406__p6654910185817">StatefulSet</p>
</td>
</tr>
<tr id="cce_10_0406__row868712385916"><td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.5.3.2.5.1.1 "><p id="cce_10_0406__p668718238593">thanosSidecar</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.2.5.1.2 "><p id="cce_10_0406__p1268762310591">Available only in HA mode. Runs with prometheus-server in the same pod to implement persistent storage of Prometheus metric data.</p>
</td>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.5.3.2.5.1.3 "><p id="cce_10_0406__p16547728556">Local data storage enabled</p>
</td>
<td class="cellrowborder" valign="top" width="12%" headers="mcps1.3.5.3.2.5.1.4 "><p id="cce_10_0406__p19687122310595">Container</p>
</td>
</tr>
<tr id="cce_10_0406__row1765471012585"><td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.5.3.2.5.1.1 "><p id="cce_10_0406__p1965451065818">thanosQuery</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.2.5.1.2 "><p id="cce_10_0406__p165431045811">Available only in HA mode. Entry for PromQL query when Prometheus is in HA scenarios. It can delete duplicate metrics from Store or Prometheus.</p>
</td>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.5.3.2.5.1.3 "><p id="cce_10_0406__p655414281751">Local data storage enabled</p>
</td>
<td class="cellrowborder" valign="top" width="12%" headers="mcps1.3.5.3.2.5.1.4 "><p id="cce_10_0406__p6418131361018">Deployment</p>
</td>
</tr>
<tr id="cce_10_0406__row1564041115919"><td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.5.3.2.5.1.1 "><p id="cce_10_0406__p74176249615">adapter</p>
<p id="cce_10_0406__p7564164185920">(workload name: custom-metrics-apiserver)</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.2.5.1.2 "><p id="cce_10_0406__p155641741105915">Aggregates custom metrics to the native Kubernetes API Server.</p>
</td>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.5.3.2.5.1.3 "><p id="cce_10_0406__p18562152818517">Local data storage enabled</p>
</td>
<td class="cellrowborder" valign="top" width="12%" headers="mcps1.3.5.3.2.5.1.4 "><p id="cce_10_0406__p3564204115917">Deployment</p>
</td>
</tr>
<tr id="cce_10_0406__row565431020582"><td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.5.3.2.5.1.1 "><p id="cce_10_0406__p4897749155914">kubeStateMetrics</p>
<p id="cce_10_0406__p81744361767">(workload name: kube-state-metrics)</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.2.5.1.2 "><p id="cce_10_0406__p17654910165815">Converts the Prometheus metric data into a format that can be identified by Kubernetes APIs. By default, the kube-state-metrics component does not collect all labels and annotations of Kubernetes resources. To collect all labels and annotations, see <a href="#cce_10_0406__section163751639151115">Collecting All Labels and Annotations of a Pod</a>.</p>
<div class="note" id="cce_10_0406__note137421311185918"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="cce_10_0406__p177427115597">If the components run in multiple pods, only one pod provides metrics.</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.5.3.2.5.1.3 "><p id="cce_10_0406__p1635314351135">All</p>
</td>
<td class="cellrowborder" valign="top" width="12%" headers="mcps1.3.5.3.2.5.1.4 "><p id="cce_10_0406__p365461005815">Deployment</p>
</td>
</tr>
<tr id="cce_10_0406__row19672124885814"><td class="cellrowborder" valign="top" width="23%" headers="mcps1.3.5.3.2.5.1.1 "><p id="cce_10_0406__p18115016013">nodeExporter</p>
<p id="cce_10_0406__p202796513616">(workload name: node-exporter)</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.2.5.1.2 "><p id="cce_10_0406__p1367384812582">Deployed on each node to collect node monitoring data.</p>
</td>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.5.3.2.5.1.3 "><p id="cce_10_0406__p53611983713">All</p>
</td>
<td class="cellrowborder" valign="top" width="12%" headers="mcps1.3.5.3.2.5.1.4 "><p id="cce_10_0406__p36731548195816">DaemonSet</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="cce_10_0406__section17830202915211"><a name="cce_10_0406__section17830202915211"></a><a name="section17830202915211"></a><h4 class="sectiontitle">Providing Basic Resource Metrics Through the Metrics API</h4><div class="note" id="cce_10_0406__note1599611485210"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="cce_10_0406__p899614481424">If <strong id="cce_10_0406__b73131855204811">Local Data Storage</strong> is enabled for the Cloud Native Cluster Monitoring add-on, basic resource metrics can only be provided through Metrics API.</p>
</div></div>
<p id="cce_10_0406__en-us_topic_0000001199501156_p188341734368">Resource metrics of containers and nodes, such as CPU and memory usage, can be obtained through the Kubernetes Metrics API. Resource metrics can be directly accessed, for example, by using the <strong id="cce_10_0406__b149931436770">kubectl top</strong> command, or used by HPA or CustomedHPA policies for auto scaling.</p>
<p id="cce_10_0406__en-us_topic_0000001199501156_p8834334668">The add-on can provide the Kubernetes Metrics API that is disabled by default. To enable the Metrics API, create the following APIService object:</p>
<pre class="screen" id="cce_10_0406__en-us_topic_0000001199501156_screen1820012451165">apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    app: custom-metrics-apiserver
    release: cceaddon-prometheus
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: custom-metrics-apiserver
    namespace: monitoring
    port: 443
  version: v1beta1
  versionPriority: 100</pre>
<p id="cce_10_0406__en-us_topic_0000001199501156_p115241332713">You can save the object as a file, name it <strong id="cce_10_0406__b1878815471774">metrics-apiservice.yaml</strong>, and run the following command:</p>
<pre class="screen" id="cce_10_0406__en-us_topic_0000001199501156_screen386018161172">kubectl create -f metrics-apiservice.yaml</pre>
<p id="cce_10_0406__en-us_topic_0000001199501156_p199681326879">Run the <strong id="cce_10_0406__b2918449676">kubectl top pod -n monitoring</strong> command. If information similar to the following is displayed, the Metrics API can be accessed:</p>
<pre class="screen" id="cce_10_0406__en-us_topic_0000001199501156_screen103261461689">NAME                                                      CPU(cores)   MEMORY(bytes)
......
custom-metrics-apiserver-d4f556ff9-l2j2m                  38m          44Mi
......</pre>
<div class="notice" id="cce_10_0406__en-us_topic_0000001199501156_note19821227125019"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="cce_10_0406__en-us_topic_0000001199501156_p18982227115018">To uninstall the add-on, run the following kubectl command and delete the APIService object. Otherwise, residual APIService resources may prevent the installation of the Kubernetes Metrics Server add-on.</p>
<pre class="screen" id="cce_10_0406__en-us_topic_0000001199501156_screen4632104316598">kubectl delete APIService v1beta1.metrics.k8s.io</pre>
</div></div>
</div>
<div class="section" id="cce_10_0406__section11927514174016"><a name="cce_10_0406__section11927514174016"></a><a name="section11927514174016"></a><h4 class="sectiontitle">Creating an HPA Policy Using Custom Metrics</h4><p id="cce_10_0406__p9443554203">HPA policies can only be used when Cloud Native Cluster Monitoring is deployed with <strong id="cce_10_0406__b10607103514480">Local Data Storage</strong> enabled. You can configure custom metrics required by HPA policies in the <strong id="cce_10_0406__b92831476242">user-adapter-config</strong> ConfigMap.</p>
<div class="notice" id="cce_10_0406__note10327155212376"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="cce_10_0406__p1484841315256">To use Prometheus to monitor custom metrics, the application needs to provide a metric monitoring API. For details, see <a href="cce_10_0373.html#cce_10_0373__section173671127160">Prometheus Monitoring Data Collection</a>.</p>
</div></div>
<p id="cce_10_0406__p1468357121818">In this section, the nginx metric (nginx_connections_accepted) in <a href="cce_10_0373.html">Monitoring Custom Metrics Using Cloud Native Cluster Monitoring</a> is used as an example.</p>
<ol id="cce_10_0406__ol26731013477"><li id="cce_10_0406__li1761820113263"><span>Log in to the <span id="cce_10_0406__cce_10_0004_ph18314322182_1">CCE console</span> and click the cluster name to access the cluster console.</span></li><li id="cce_10_0406__li49761537112"><span>In the navigation pane, choose <strong id="cce_10_0406__b538942572819"><span id="cce_10_0406__text1266312153316">ConfigMaps and Secrets</span></strong>. Switch to the <strong id="cce_10_0406__b195383611284">monitoring</strong> namespace, click the <strong id="cce_10_0406__b10290115352219">ConfigMaps</strong> tab, locate the row containing <strong id="cce_10_0406__b01901513112317">user-adapter-config</strong> (or <strong id="cce_10_0406__b1711291872311">adapter-config</strong>), and click <span class="uicontrol" id="cce_10_0406__uicontrol147501842314"><b>Update</b></span>.</span><p><p id="cce_10_0406__p11511973302"></p>
</p></li><li id="cce_10_0406__li3252123812205"><span>In <strong id="cce_10_0406__b49601134165211">Data</strong>, click <strong id="cce_10_0406__b13961173435219">Edit</strong> for the <strong id="cce_10_0406__b1696163413523">config.yaml</strong> file to add a custom metric collection rule under the <strong id="cce_10_0406__b20961173412523">rules</strong> field. Click <strong id="cce_10_0406__b8811164892718">OK</strong>.</span><p><p id="cce_10_0406__p2583125516253">You can add multiple collection rules by adding multiple configurations under the <strong id="cce_10_0406__b133961975317">rules</strong> field. For details, see <a href="https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config.md" target="_blank" rel="noopener noreferrer">Metrics Discovery and Presentation Configuration</a>.</p>
<div class="p" id="cce_10_0406__p24311556112513">Example custom metric rule:<pre class="screen" id="cce_10_0406__screen3583135511253">rules:
# Match the metric whose name is <strong id="cce_10_0406__b1825613529222">nginx_connections_accepted</strong>. The metric name must be confirmed. Otherwise, the HPA controller cannot get the metric.
<strong id="cce_10_0406__b12339131483213">- seriesQuery: '{__name__=~"nginx_connections_accepted",container!="POD",namespace!="",pod!=""}'</strong>
<strong id="cce_10_0406__b17343161413214">  resources:</strong>
    # Specify pod and namespace resources.
<strong id="cce_10_0406__b3344171419326">    overrides:</strong>
<strong id="cce_10_0406__b203454140327">      namespace:</strong>
<strong id="cce_10_0406__b6345111443219">        resource: namespace</strong>
<strong id="cce_10_0406__b153461914183214">      pod:</strong>
<strong id="cce_10_0406__b10346914133214">        resource: pod</strong>
<strong id="cce_10_0406__b1834761443219">  name:</strong>
    #Use nginx_connections_accepted"
    <strong id="cce_10_0406__b75724359212">matches: "nginx_connections_accepted"</strong>
    #Use nginx_connections_accepted_per_second to represent the metric. The name is the custom metric name in a custom HPA policy.
    <strong id="cce_10_0406__b1348412395219">as: "nginx_connections_accepted_per_second"</strong>
    # Calculate rate(nginx_connections_accepted[2m]) to specify the number of requests received per second.
  <strong id="cce_10_0406__b1135313520212">metricsQuery: 'rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;,container!="POD"}[2m])'</strong></pre>
</div>
<p id="cce_10_0406__p952824520334"></p>
</p></li><li id="cce_10_0406__li8948827192217"><span>Redeploy the <strong id="cce_10_0406__b1770585917308">custom-metrics-apiserver</strong> workload in the <strong id="cce_10_0406__b187718420315">monitoring</strong> namespace.</span></li><li id="cce_10_0406__li3494122464317"><span>In the navigation pane, choose <span class="uicontrol" id="cce_10_0406__uicontrol1521920375417"><b>Workloads</b></span>. Locate the workload for which you want to create an HPA policy and choose <span class="uicontrol" id="cce_10_0406__uicontrol39780441806"><b>More</b></span> &gt; <strong id="cce_10_0406__b1154541382218">Auto Scaling</strong>. In the <strong id="cce_10_0406__b1960941718507">Custom Policy</strong> area, you can select the preceding parameters to create an auto scaling policy.</span></li></ol>
</div>
<div class="section" id="cce_10_0406__section163751639151115"><a name="cce_10_0406__section163751639151115"></a><a name="section163751639151115"></a><h4 class="sectiontitle">Collecting All Labels and Annotations of a Pod</h4><ol id="cce_10_0406__ol4972103611195"><li id="cce_10_0406__li198296182613"><span>Log in to the <span id="cce_10_0406__cce_10_0004_ph18314322182_2">CCE console</span> and click the cluster name to access the cluster console.</span></li><li id="cce_10_0406__li1140182632412"><span>In the navigation pane, choose <strong id="cce_10_0406__b15916154915276">Workloads</strong>. Switch to the <strong id="cce_10_0406__b6290018162815">monitoring</strong> namespace, click the <strong id="cce_10_0406__b1629021822817">Deployments</strong> tab, and click the name of the <strong id="cce_10_0406__b3290171852813">kube-state-metrics</strong> workload. On the page displayed, click the <strong id="cce_10_0406__b1729141802813">Containers</strong> tab and click <strong id="cce_10_0406__b132911218132817">Edit</strong> on the right.</span></li><li id="cce_10_0406__li11145115692815"><span>In the <strong id="cce_10_0406__b934171314103">Lifecycle</strong> area of the container settings, edit the startup command.</span><p><div class="p" id="cce_10_0406__p673817271288">To collect labels, add the following information to the end of the original <strong id="cce_10_0406__b7875174481018">kube-state-metrics</strong> startup parameter:<pre class="screen" id="cce_10_0406__screen1434944831318">--metric-labels-allowlist=pods=[*],nodes=[node,failure-domain.beta.kubernetes.io/zone,topology.kubernetes.io/zone]</pre>
</div>
<div class="p" id="cce_10_0406__p7711154111911">To collect annotations, add parameters in the startup parameters in the same way.<pre class="screen" id="cce_10_0406__screen143831101919">--metric-annotations-allowlist=pods=[*],nodes=[node,failure-domain.beta.kubernetes.io/zone,topology.kubernetes.io/zone]</pre>
</div>
<div class="notice" id="cce_10_0406__note5441191217304"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="cce_10_0406__p1644114127304">When editing the startup command, do not modify other original startup parameters. Otherwise, the component may be abnormal.</p>
</div></div>
</p></li><li id="cce_10_0406__li1088317513196"><span><strong id="cce_10_0406__b183551524101118">kube-state-metrics</strong> starts to collect the labels/annotations of pods and nodes and checks whether <strong id="cce_10_0406__b10842144411112">kube_pod_labels/kube_pod_annotations</strong> is in the collection task of Prometheus.</span><p><pre class="screen" id="cce_10_0406__screen12842134013206">kubectl get servicemonitor kube-state-metrics -nmonitoring -oyaml | grep kube_pod_labels</pre>
</p></li></ol>
<p id="cce_10_0406__p597521071411">For more kube-state-metrics startup parameters, see <a href="https://github.com/kubernetes/kube-state-metrics/blob/v2.2.3/docs/cli-arguments.md" target="_blank" rel="noopener noreferrer">kube-state-metrics/cli-arguments</a>.</p>
</div>
<div class="section" id="cce_10_0406__section183121449435"><h4 class="sectiontitle">Release History</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="cce_10_0406__table4943143014277" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Cloud Native Cluster Monitoring add-on</caption><thead align="left"><tr id="cce_10_0406__en-us_topic_0000001559534266_row13459112313176"><th align="left" class="cellrowborder" valign="top" width="16.31%" id="mcps1.3.9.2.2.5.1.1"><p id="cce_10_0406__en-us_topic_0000001559534266_p206369328181">Add-on Version</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="21.349999999999998%" id="mcps1.3.9.2.2.5.1.2"><p id="cce_10_0406__en-us_topic_0000001559534266_p1663653221810">Supported Cluster Version</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="38.9%" id="mcps1.3.9.2.2.5.1.3"><p id="cce_10_0406__en-us_topic_0000001559534266_p26362032131817">New Feature</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="23.44%" id="mcps1.3.9.2.2.5.1.4"><p id="cce_10_0406__en-us_topic_0000001559534266_p445992311174">Community Version</p>
</th>
</tr>
</thead>
<tbody><tr id="cce_10_0406__en-us_topic_0000001559534266_row934413271416"><td class="cellrowborder" valign="top" width="16.31%" headers="mcps1.3.9.2.2.5.1.1 "><p id="cce_10_0406__en-us_topic_0000001559534266_p1860419376415">3.12.2</p>
</td>
<td class="cellrowborder" valign="top" width="21.349999999999998%" headers="mcps1.3.9.2.2.5.1.2 "><p id="cce_10_0406__en-us_topic_0000001559534266_p2604137745">v1.21</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p360419371042">v1.23</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p116047371410">v1.25</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p1260411379417">v1.27</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p36042378413">v1.28</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p14604143710417">v1.29</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p20604137547">v1.30</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p560410371347">v1.31</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p560414371345">v1.32</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p23131542843">v1.33</p>
</td>
<td class="cellrowborder" valign="top" width="38.9%" headers="mcps1.3.9.2.2.5.1.3 "><p id="cce_10_0406__en-us_topic_0000001559534266_p1660413718419">CCE clusters v1.33 are supported.</p>
</td>
<td class="cellrowborder" valign="top" width="23.44%" headers="mcps1.3.9.2.2.5.1.4 "><p id="cce_10_0406__en-us_topic_0000001559534266_p14604837740"><a href="https://github.com/prometheus/prometheus/releases/tag/v2.53.2" target="_blank" rel="noopener noreferrer">2.53.2</a></p>
</td>
</tr>
<tr id="cce_10_0406__en-us_topic_0000001559534266_row526916486421"><td class="cellrowborder" valign="top" width="16.31%" headers="mcps1.3.9.2.2.5.1.1 "><p id="cce_10_0406__en-us_topic_0000001559534266_p152771834184311">3.12.1</p>
</td>
<td class="cellrowborder" valign="top" width="21.349999999999998%" headers="mcps1.3.9.2.2.5.1.2 "><p id="cce_10_0406__en-us_topic_0000001559534266_p186644054314">v1.21</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p46674074320">v1.23</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p1766194015432">v1.25</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p1766440144315">v1.27</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p106644013439">v1.28</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p66674014310">v1.29</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p1566124024320">v1.30</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p266140134314">v1.31</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p1956962230">v1.32</p>
</td>
<td class="cellrowborder" valign="top" width="38.9%" headers="mcps1.3.9.2.2.5.1.3 "><p id="cce_10_0406__en-us_topic_0000001559534266_p646784894311">CCE clusters v1.32 are supported.</p>
</td>
<td class="cellrowborder" valign="top" width="23.44%" headers="mcps1.3.9.2.2.5.1.4 "><p id="cce_10_0406__en-us_topic_0000001559534266_p11665407430"><a href="https://github.com/prometheus/prometheus/releases/tag/v2.53.2" target="_blank" rel="noopener noreferrer">2.53.2</a></p>
</td>
</tr>
<tr id="cce_10_0406__en-us_topic_0000001559534266_row20671951192218"><td class="cellrowborder" valign="top" width="16.31%" headers="mcps1.3.9.2.2.5.1.1 "><p id="cce_10_0406__en-us_topic_0000001559534266_p66705115228">3.12.0</p>
</td>
<td class="cellrowborder" valign="top" width="21.349999999999998%" headers="mcps1.3.9.2.2.5.1.2 "><p id="cce_10_0406__en-us_topic_0000001559534266_p1819771115238">v1.21</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p131971211112317">v1.23</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p2197121192313">v1.25</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p14197911162313">v1.27</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p1119731113237">v1.28</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p71971711122315">v1.29</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p819710117232">v1.30</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p8931171313237">v1.31</p>
</td>
<td class="cellrowborder" valign="top" width="38.9%" headers="mcps1.3.9.2.2.5.1.3 "><ul id="cce_10_0406__en-us_topic_0000001559534266_ul16823318192316"><li id="cce_10_0406__en-us_topic_0000001559534266_li3824151802316">CCE clusters v1.31 are supported.</li><li id="cce_10_0406__en-us_topic_0000001559534266_li851102113238">Upgraded Prometheus.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="23.44%" headers="mcps1.3.9.2.2.5.1.4 "><p id="cce_10_0406__en-us_topic_0000001559534266_p11681251162211"><a href="https://github.com/prometheus/prometheus/releases/tag/v2.53.2" target="_blank" rel="noopener noreferrer">2.53.2</a></p>
</td>
</tr>
<tr id="cce_10_0406__en-us_topic_0000001559534266_row185517439467"><td class="cellrowborder" valign="top" width="16.31%" headers="mcps1.3.9.2.2.5.1.1 "><p id="cce_10_0406__en-us_topic_0000001559534266_p3734154510460">3.11.0</p>
</td>
<td class="cellrowborder" valign="top" width="21.349999999999998%" headers="mcps1.3.9.2.2.5.1.2 "><p id="cce_10_0406__en-us_topic_0000001559534266_p11734164518460">v1.21</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p473414534610">v1.23</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p4734164514617">v1.25</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p1673419456469">v1.27</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p573424518467">v1.28</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p10734145144610">v1.29</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p16594859184612">v1.30</p>
</td>
<td class="cellrowborder" valign="top" width="38.9%" headers="mcps1.3.9.2.2.5.1.3 "><p id="cce_10_0406__en-us_topic_0000001559534266_p98111922164615">CCE clusters v1.30 are supported.</p>
</td>
<td class="cellrowborder" valign="top" width="23.44%" headers="mcps1.3.9.2.2.5.1.4 "><p id="cce_10_0406__en-us_topic_0000001559534266_p147341745134611"><a href="https://github.com/prometheus/prometheus/releases/tag/v2.37.8" target="_blank" rel="noopener noreferrer">2.37.8</a></p>
</td>
</tr>
<tr id="cce_10_0406__en-us_topic_0000001559534266_row1863002115911"><td class="cellrowborder" valign="top" width="16.31%" headers="mcps1.3.9.2.2.5.1.1 "><p id="cce_10_0406__en-us_topic_0000001559534266_p8720641185913">3.10.1</p>
</td>
<td class="cellrowborder" valign="top" width="21.349999999999998%" headers="mcps1.3.9.2.2.5.1.2 "><p id="cce_10_0406__en-us_topic_0000001559534266_p872094115910">v1.21</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p472014113596">v1.23</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p6720154119595">v1.25</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p1872064116596">v1.27</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p07201741155917">v1.28</p>
<p id="cce_10_0406__en-us_topic_0000001559534266_p97205413597">v1.29</p>
</td>
<td class="cellrowborder" valign="top" width="38.9%" headers="mcps1.3.9.2.2.5.1.3 "><p id="cce_10_0406__en-us_topic_0000001559534266_p96301821155914">Upgraded NodeExporter to 1.8.0.</p>
</td>
<td class="cellrowborder" valign="top" width="23.44%" headers="mcps1.3.9.2.2.5.1.4 "><p id="cce_10_0406__en-us_topic_0000001559534266_p5630821155914"><a href="https://github.com/prometheus/prometheus/releases/tag/v2.37.8" target="_blank" rel="noopener noreferrer">2.37.8</a></p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="cce_10_0908.html">Cloud Native Observability Add-ons</a></div>
</div>
</div>