forked from docs/doc-exports
qiujiandong1
71d5c814e7
Reviewed-by: Eotvos, Oliver <oliver.eotvos@t-systems.com> Co-authored-by: qiujiandong1 <qiujiandong1@huawei.com> Co-committed-by: qiujiandong1 <qiujiandong1@huawei.com>
159 lines
15 KiB
HTML
159 lines
15 KiB
HTML
<a name="cce_10_0201"></a><a name="cce_10_0201"></a>
|
|
|
|
<h1 class="topictitle1">Monitoring Custom Metrics on AOM</h1>
|
|
<div id="body0000001176124020"><p id="cce_10_0201__p8060118">CCE allows you to upload custom metrics to AOM. ICAgent on a node periodically calls the metric monitoring API configured on a workload to read monitoring data and then uploads the data to AOM.</p>
|
|
<div class="fignone" id="cce_10_0201__fig119692414478"><span class="figcap"><b>Figure 1 </b>Using ICAgent to collect monitoring metrics</span><br><span><img id="cce_10_0201__image15650732403" src="en-us_image_0000002253619601.png"></span></div>
|
|
<p id="cce_10_0201__p1649415194016">The custom metric API of a workload can be configured when the workload is created. The following procedure uses an Nginx application as an example to describe how to report custom metrics to AOM.</p>
|
|
<ol id="cce_10_0201__ol755829153011"><li id="cce_10_0201__li12551729153011"><a href="#cce_10_0201__section14984815298">Preparing an Application</a><p id="cce_10_0201__p170175633014">Prepare an application image. The application must provide a metric monitoring API for ICAgent to collect data, and the monitoring data must <a href="#cce_10_0201__section173671127160">comply with the Prometheus specifications</a>.</p>
|
|
</li><li id="cce_10_0201__li139312392308"><a href="#cce_10_0201__section1539954011362">Deploying Applications and Converting Nginx Metrics</a><p id="cce_10_0201__p1854781214363">Use the application image to deploy a workload in a cluster. Custom metrics will be automatically reported.</p>
|
|
</li><li id="cce_10_0201__li9168154819302"><a href="#cce_10_0201__section42551081185">Verification</a><p id="cce_10_0201__p287816183915">Go to AOM to check whether the custom metrics are successfully collected.</p>
|
|
</li></ol>
|
|
<div class="section" id="cce_10_0201__section11634016215"><h4 class="sectiontitle">Constraints</h4><ul id="cce_10_0201__ul066520226214"><li id="cce_10_0201__li1935518502212">The ICAgent is compatible with the monitoring data specifications of <a href="https://prometheus.io/" target="_blank" rel="noopener noreferrer">Prometheus</a>. The custom metrics provided by pods can be collected by the ICAgent only when they meet the monitoring data specifications of Prometheus. For details, see <a href="#cce_10_0201__section173671127160">Prometheus Monitoring Data Collection</a>.</li><li id="cce_10_0201__li84311436320">The ICAgent supports only <a href="https://prometheus.io/docs/concepts/metric_types/" target="_blank" rel="noopener noreferrer">Gauge</a> metrics.</li><li id="cce_10_0201__li35904504518">The interval for the ICAgent to call the custom metric API is 1 minute, which cannot be changed.</li></ul>
|
|
</div>
|
|
<div class="section" id="cce_10_0201__section173671127160"><a name="cce_10_0201__section173671127160"></a><a name="section173671127160"></a><h4 class="sectiontitle">Prometheus Monitoring Data Collection</h4><p id="cce_10_0201__p23001123182112">Prometheus periodically calls the metric monitoring API (<span class="uicontrol" id="cce_10_0201__uicontrol5708652235"><b>/metrics</b></span> by default) of an application to obtain monitoring data. The application needs to provide the metric monitoring API for Prometheus to call, and the monitoring data must meet the following specifications of Prometheus:</p>
|
|
<pre class="screen" id="cce_10_0201__screen1584511211257"># TYPE nginx_connections_active gauge
|
|
nginx_connections_active 2
|
|
# TYPE nginx_connections_reading gauge
|
|
nginx_connections_reading 0</pre>
|
|
<p id="cce_10_0201__p1358113105720">Prometheus provides clients in various languages. For details about the clients, see <a href="https://prometheus.io/docs/instrumenting/clientlibs/" target="_blank" rel="noopener noreferrer">Prometheus CLIENT LIBRARIES</a>. For details about how to develop an exporter, see <a href="https://prometheus.io/docs/instrumenting/writing_exporters/" target="_blank" rel="noopener noreferrer">WRITING EXPORTERS</a>. The Prometheus community provides various third-party exporters that can be directly used. For details, see <a href="https://prometheus.io/docs/instrumenting/exporters/" target="_blank" rel="noopener noreferrer">EXPORTERS AND INTEGRATIONS</a>.</p>
|
|
</div>
|
|
<div class="section" id="cce_10_0201__section14984815298"><a name="cce_10_0201__section14984815298"></a><a name="section14984815298"></a><h4 class="sectiontitle">Preparing an Application</h4><p id="cce_10_0201__p12799784716">User-developed applications must provide a metric monitoring API, and the monitoring data must comply with the Prometheus specifications. For details, see <a href="#cce_10_0201__section173671127160">Prometheus Monitoring Data Collection</a>.</p>
|
|
<p id="cce_10_0201__p869125518296">This section uses Nginx as an example to describe how to collect monitoring data. There is a module named <strong id="cce_10_0201__b3952153014374">ngx_http_stub_status_module</strong> in Nginx, which provides basic monitoring functions. You can configure the <strong id="cce_10_0201__b1057321884510">nginx.conf</strong> file to provide an interface for external systems to access Nginx monitoring data.</p>
|
|
<ol id="cce_10_0201__ol47741617716"><li id="cce_10_0201__li172914434716"><span>Log in to a Linux VM that can access to the Internet and run Docker commands.</span></li><li id="cce_10_0201__li5774511276"><span>Create an <strong id="cce_10_0201__b1576091111477">nginx.conf</strong> file. Add the server configuration under <strong id="cce_10_0201__b982818481210">http</strong> to enable Nginx to provide an interface for the external systems to access the monitoring data.</span><p><pre class="screen" id="cce_10_0201__screen1773195816508">user nginx;
|
|
worker_processes auto;
|
|
|
|
error_log /var/log/nginx/error.log warn;
|
|
pid /var/run/nginx.pid;
|
|
|
|
events {
|
|
worker_connections 1024;
|
|
}
|
|
|
|
http {
|
|
include /etc/nginx/mime.types;
|
|
default_type application/octet-stream;
|
|
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
|
|
'$status $body_bytes_sent "$http_referer" '
|
|
'"$http_user_agent" "$http_x_forwarded_for"';
|
|
|
|
access_log /var/log/nginx/access.log main;
|
|
sendfile on;
|
|
#tcp_nopush on;
|
|
keepalive_timeout 65;
|
|
#gzip on;
|
|
include /etc/nginx/conf.d/*.conf;
|
|
|
|
server {
|
|
listen 8080;
|
|
server_name localhost;
|
|
location /stub_status {
|
|
stub_status on;
|
|
access_log off;
|
|
}
|
|
}
|
|
}</pre>
|
|
</p></li><li id="cce_10_0201__li1199121431218"><span>Use this configuration to create an image and a Dockerfile file.</span><p><pre class="screen" id="cce_10_0201__screen58471820191212">vi Dockerfile</pre>
|
|
<div class="p" id="cce_10_0201__p1079719162127">The content of Dockerfile is as follows:<pre class="screen" id="cce_10_0201__screen1412112118918">FROM nginx:1.21.5-alpine
|
|
ADD nginx.conf /etc/nginx/nginx.conf
|
|
EXPOSE 80
|
|
CMD ["nginx", "-g", "daemon off;"]</pre>
|
|
</div>
|
|
</p></li><li id="cce_10_0201__li171513248917"><span>Use this Dockerfile to create an image and upload it to SWR. The image name is <strong id="cce_10_0201__b19241320133619">nginx:exporter</strong>. </span><p><ol type="a" id="cce_10_0201__ol111731405102"><li id="cce_10_0201__en-us_topic_0000001148755333_li187221141362">In the navigation pane, choose <strong id="cce_10_0201__b18285167113913">My Images</strong>. In the upper right corner, click <strong id="cce_10_0201__b1428516710391">Upload Through Client</strong>. In the displayed dialog box, click <strong id="cce_10_0201__b728510713915">Generate a temporary login command</strong>. Then, click <span><img id="cce_10_0201__en-us_topic_0000001148755333_image18751161714" src="en-us_image_0000002253779505.png"></span> to copy the command.</li><li id="cce_10_0201__en-us_topic_0000001148755333_li4452185981216">Run the login command copied in the previous step on the node. If the login is successful, the message "Login Succeeded" is displayed.</li><li id="cce_10_0201__li820205451013">Run the following command to build an image named nginx. The image version is exporter.<pre class="screen" id="cce_10_0201__screen12404356141220">docker build -t nginx:exporter .</pre>
|
|
</li><li id="cce_10_0201__li195490338118">Tag the image and upload it to the image repository. Change the image repository address and organization name based on your requirements.<pre class="screen" id="cce_10_0201__screen0590911161715">docker tag nginx:exporter {swr-address}/{group}/nginx:exporter
|
|
docker push {swr-address}/{group}/nginx:exporter</pre>
|
|
</li></ol>
|
|
</p></li><li id="cce_10_0201__li3673432142319"><span>View application metrics.</span><p><ol type="a" id="cce_10_0201__ol93905514234"><li id="cce_10_0201__li20154355142319">Use <strong id="cce_10_0201__b20613053114315">nginx:exporter</strong> to create a workload.</li><li id="cce_10_0201__li839005118236"><a href="cce_10_00356.html">Log in to the container</a> and obtain the Nginx monitoring data through <strong id="cce_10_0201__b288233814399">http://</strong><em id="cce_10_0201__i3813114516398"><ip_address></em>:<strong id="cce_10_0201__b15343142113912">8080/stub_status</strong>. <em id="cce_10_0201__i265245318392"><ip_address></em> indicates the IP address of the container.<pre class="screen" id="cce_10_0201__screen941371015355">curl http://127.0.0.1:8080/stub_status</pre>
|
|
<p id="cce_10_0201__p12796133718238">The command output is as follows:</p>
|
|
<pre class="screen" id="cce_10_0201__screen99889917190">Active connections: 3
|
|
server accepts handled requests
|
|
146269 146269 212
|
|
Reading: 0 Writing: 1 Waiting: 2</pre>
|
|
</li></ol>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="cce_10_0201__section1539954011362"><a name="cce_10_0201__section1539954011362"></a><a name="section1539954011362"></a><h4 class="sectiontitle">Deploying Applications and Converting Nginx Metrics</h4><p id="cce_10_0201__p13696165713362">The format of the monitoring data provided by <strong id="cce_10_0201__b248994731115">nginx:exporter</strong> does not meet the requirements of Prometheus. Convert the data format to the format required by Prometheus. To convert the format of Nginx metrics, use <a href="https://github.com/nginxinc/nginx-prometheus-exporter" target="_blank" rel="noopener noreferrer"><strong id="cce_10_0201__b24336402133">nginx-prometheus-exporter</strong></a>, as shown in the following figure.</p>
|
|
<div class="fignone" id="cce_10_0201__fig16342448144712"><span class="figcap"><b>Figure 2 </b>Using exporter to convert the data format</span><br><span><img id="cce_10_0201__image11567167161312" src="en-us_image_0000002253619605.png"></span></div>
|
|
<p id="cce_10_0201__p13391926836">Deploy <strong id="cce_10_0201__b15666153321316">nginx:exporter</strong> and <strong id="cce_10_0201__b1324493616133">nginx-prometheus-exporter</strong> in the same pod.</p>
|
|
<pre class="screen" id="cce_10_0201__screen726020139814">kind: Deployment
|
|
apiVersion: apps/v1
|
|
metadata:
|
|
name: nginx-exporter
|
|
namespace: default
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: nginx-exporter
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: nginx-exporter
|
|
annotations:
|
|
metrics.alpha.kubernetes.io/custom-endpoints: '[{"api":"prometheus","path":"/metrics","port":"9113","names":""}]'
|
|
spec:
|
|
containers:
|
|
- name: container-0
|
|
image: 'nginx:exporter' # Replace it with the address of the image you uploaded to SWR.
|
|
resources:
|
|
limits:
|
|
cpu: 250m
|
|
memory: 512Mi
|
|
requests:
|
|
cpu: 250m
|
|
memory: 512Mi
|
|
- name: container-1
|
|
image: 'nginx/nginx-prometheus-exporter:0.9.0'
|
|
command:
|
|
- nginx-prometheus-exporter
|
|
args:
|
|
- '-nginx.scrape-uri=http://127.0.0.1:8080/stub_status'
|
|
imagePullSecrets:
|
|
- name: default-secret</pre>
|
|
<div class="note" id="cce_10_0201__note1789303032414"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="cce_10_0201__p4893143011248">The <strong id="cce_10_0201__b5441648133020">nginx/nginx-prometheus-exporter:0.9.0</strong> image needs to be pulled from the public network. Therefore, a public IP address needs to be bound to each node in the cluster.</p>
|
|
</div></div>
|
|
<p id="cce_10_0201__p94942045392">nginx-prometheus-exporter requires a startup command. <strong id="cce_10_0201__b778519222196">nginx-prometheus-exporter -nginx.scrape-uri=http://127.0.0.1:8080/stub_status</strong> is used to obtain Nginx monitoring data.</p>
|
|
<p id="cce_10_0201__p79431941101118">In addition, add an annotation <strong id="cce_10_0201__b14505215173318">metrics.alpha.kubernetes.io/custom-endpoints: '[{"api":"prometheus","path":"/metrics","port":"9113","names":""}]'</strong> to the pod.</p>
|
|
</div>
|
|
<div class="section" id="cce_10_0201__section42551081185"><a name="cce_10_0201__section42551081185"></a><a name="section42551081185"></a><h4 class="sectiontitle">Verification</h4><p id="cce_10_0201__p14723430121818">After an application is deployed, you can access Nginx to construct some access data and check whether the corresponding monitoring data can be obtained in AOM.</p>
|
|
<ol id="cce_10_0201__ol1940325434813"><li id="cce_10_0201__li17403254114819"><span>Obtain the pod name of Nginx.</span><p><pre class="screen" id="cce_10_0201__screen86986221360">kubectl get pod</pre>
|
|
<p id="cce_10_0201__p9181715113618">The command output is as follows:</p>
|
|
<pre class="screen" id="cce_10_0201__screen19233548172719">NAME READY STATUS RESTARTS AGE
|
|
nginx-exporter-78859765db-6j8sw 2/2 Running 0 4m</pre>
|
|
</p></li><li id="cce_10_0201__li11519182418493"><span>Run the following command to log in to the container:</span><p><pre class="screen" id="cce_10_0201__screen1551545810364">kubectl exec -it <i><span class="varname" id="cce_10_0201__varname557453716">nginx-exporter-78859765db-6j8sw</span></i> -- /bin/sh</pre>
|
|
<p id="cce_10_0201__p551716192382">Run the following command to access Nginx:</p>
|
|
<pre class="screen" id="cce_10_0201__screen8366425153819">curl http://localhost</pre>
|
|
<p id="cce_10_0201__p18508115415363">The command output is as follows:</p>
|
|
<pre class="screen" id="cce_10_0201__screen1558895115497"><!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<title>Welcome to nginx!</title>
|
|
<style>
|
|
html { color-scheme: light dark; }
|
|
body { width: 35em; margin: 0 auto;
|
|
font-family: Tahoma, Verdana, Arial, sans-serif; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<h1>Welcome to nginx!</h1>
|
|
<p>If you see this page, the nginx web server is successfully installed and
|
|
working. Further configuration is required.</p>
|
|
|
|
<p>For online documentation and support please refer to
|
|
<a href="http://nginx.org/">nginx.org</a>.<br/>
|
|
Commercial support is available at
|
|
<a href="http://nginx.com/">nginx.com</a>.</p>
|
|
|
|
<p><em>Thank you for using nginx.</em></p>
|
|
</body>
|
|
</html>
|
|
/ #</pre>
|
|
</p></li><li id="cce_10_0201__li7744182284917"><span>Log in to AOM. In the navigation pane, choose <strong id="cce_10_0201__b6265216103615">Monitoring</strong> > <strong id="cce_10_0201__b132665165364">Metric Monitoring</strong> to view Nginx-related metrics, for example, <strong id="cce_10_0201__b526621623612">nginx_connections_active</strong>.</span></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="cce_10_0811.html">O&M Best Practices</a></div>
|
|
</div>
|
|
</div>
|
|
|