forked from docs/doc-exports
zhengxiu
93d856d5c5
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: zhengxiu <zhengxiu@huawei.com> Co-committed-by: zhengxiu <zhengxiu@huawei.com>
185 lines
44 KiB
HTML
185 lines
44 KiB
HTML
<a name="EN-US_TOPIC_0000001955726478"></a><a name="EN-US_TOPIC_0000001955726478"></a>
|
|
|
|
<h1 class="topictitle1">Using In-house Built Logstash to Ingest Data into an OpenSearch Cluster</h1>
|
|
<div id="body0000001955726478"><p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p6252043134616">With <span id="EN-US_TOPIC_0000001955726478__text17435115331016">CSS</span>, you can use in-house developed Logstash to ingest data into OpenSearch for efficient search and exploration. Supported data formats include JSON and CSV.</p>
|
|
<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p2714111201213">Logstash is an open-source, server-side data processing pipeline that ingests data from multiple sources simultaneously, processes and transforms the data, and then sends it to OpenSearch. For more information about Logstash, visit <a href="https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html</a></p>
|
|
<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p651723317178">Depending on where Logstash is deployed, there are two data ingestion scenarios:</p>
|
|
<ul id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ul11548165271713"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li16548852101714"><a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_section072813417814">Ingesting Data When Logstash Is Deployed on an External Network</a></li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li0548652141717"><a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_section1098217174335">Ingesting Data When Logstash Is Deployed Using an ECS</a></li></ul>
|
|
<div class="section" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_section371994174412"><h4 class="sectiontitle">Prerequisites</h4><ul id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ul15513934114514"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li851317349456">To facilitate operations, you are advised to deploy Logstash on a host that runs a Linux operating system (OS).</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li105075367453">Logstash must use an OSS version that is consistent with that of the CSS cluster. To download Logstash, go to <a href="https://www.elastic.co/downloads/logstash-oss" target="_blank" rel="noopener noreferrer">https://www.elastic.co/downloads/logstash-oss</a>.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li9641131917016">The JDK must be installed before Logstash is installed. In a Linux OS, you can run the <strong id="EN-US_TOPIC_0000001955726478__b1028781311595">yum -y install java-1.8.0</strong> command to install JDK 1.8.0. In a Windows OS, you can download the required JDK version from the <a href="https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html" target="_blank" rel="noopener noreferrer">official website of JDK</a>, and install it by following the installation guide.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_li165290425171">After installing Logstash, perform the steps below to ingest data. For details about how to install Logstash, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/installing-logstash.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/installing-logstash.html</a></li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1995041711115">In the <a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_section1098217174335">Ingesting Data When Logstash Is Deployed Using an ECS</a> scenario, ensure that the ECS and the destination Elasticsearch cluster are in the same VPC.</li></ul>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_section072813417814"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_section072813417814"></a><a name="en-us_topic_0000001938218520_en-us_topic_0000001223914344_section072813417814"></a><h4 class="sectiontitle">Ingesting Data When Logstash Is Deployed on an External Network</h4><p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p4281513171111"><a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_fig471717481106">Figure 1</a> illustrates the data ingestion process when Logstash is deployed on an external network.</p>
|
|
<div class="fignone" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_fig471717481106"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_fig471717481106"></a><a name="en-us_topic_0000001938218520_en-us_topic_0000001223914344_fig471717481106"></a><span class="figcap"><b>Figure 1 </b>Data ingestion process when Logstash is deployed on an external network</span><br><span><img class="eddx" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_image1539410379715" src="figure/en-us_image_0000002412558009.png"></span></div>
|
|
<ol id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ol20180835132314"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1648853125014"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1648853125014"></a><a name="en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1648853125014"></a>Create a jump host and configure it as follows:<ul id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ul2807152315574"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li18807132310572">The jump host is an ECS running a Linux OS and an EIP has been associated with it.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1692633435810">The jump host resides in the same VPC as the destination cluster.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li129110535017">SSH local port forwarding is configured for the jump host to forward requests from a chosen local port to port <strong id="EN-US_TOPIC_0000001955726478__b158531227192519">9200</strong> on one node of the CSS cluster.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1277830155915">Refer to <a href="https://man.openbsd.org/ssh.1#L" target="_blank" rel="noopener noreferrer">SSH documentation</a> for the local port forwarding configuration.</li></ul>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li19164183513237">Use PuTTY to log in to the jump host via its EIP.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1316473511236">Run the following command to configure port mapping to forward requests sent to the opened port on the jump host to the destination cluster:<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen716453518231">ssh -g -L <em id="EN-US_TOPIC_0000001955726478__i149461540112510"><Local port of the jump host</em>:<em id="EN-US_TOPIC_0000001955726478__i20947440152515">Private network address and port number of a node></em> -N -f root@<em id="EN-US_TOPIC_0000001955726478__i1394734052516"><Private IP address of the jump host</em>></pre>
|
|
<ul id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ul3164163513237"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1216415353237">In the preceding command, <<em id="EN-US_TOPIC_0000001955726478__i19756164212519">Local port of the jump host</em>> refers to the jump host port configured in <a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1648853125014">1</a>.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li516410352234">In the preceding command, <em id="EN-US_TOPIC_0000001955726478__i138671722132818"><Private network address and port number of a node></em> refers to the private network address and port number of a node in the cluster. If the node is faulty, command execution will fail. If the cluster contains multiple nodes, you can replace the value of <strong id="EN-US_TOPIC_0000001955726478__b17862112592811"><private network address and port number of a node></strong> with the private network address and port number of any available node in the cluster. If the cluster contains only one node, restore the node and execute the command again.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li91643359239">Replace <<em id="EN-US_TOPIC_0000001955726478__i10977354563">Private IP address of the jump host</em>> in the preceding command with the IP address (with <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue139771358569"><b>Private IP</b></span>) of the created jump host in the <span class="parmname" id="EN-US_TOPIC_0000001955726478__parmname6978205165611"><b>IP Address</b></span> column in the ECS list on the ECS management console.</li></ul>
|
|
<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p1416483562310">For example, port <strong id="EN-US_TOPIC_0000001955726478__b17843143492819">9200</strong> on the jump host is accessible from the public network, the private network address and port number of the node are <strong id="EN-US_TOPIC_0000001955726478__b284393418286">192.168.0.81</strong> and <strong id="EN-US_TOPIC_0000001955726478__b188441434182812">9200</strong>, respectively, and the private IP address of the jump host is <strong id="EN-US_TOPIC_0000001955726478__b138441434122813">192.168.0.227</strong>. You need to run the following command to perform port mapping:</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen7164123516239">ssh -g -L 9200:192.168.0.81:9200 -N -f root@192.168.0.227</pre>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li5164153542312"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li5164153542312"></a><a name="en-us_topic_0000001938218520_en-us_topic_0000001223914344_li5164153542312"></a>Log in to the server where Logstash is deployed and store the files to be ingested on this server.<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p9164235142316"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li5164153542312"></a><a name="en-us_topic_0000001938218520_en-us_topic_0000001223914344_li5164153542312"></a>For example, data file <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath061013447285"><b>access_20181029_log</b></span> needs to be ingested, the file storage path is <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath146101844142816"><b>/tmp/access_log/</b></span> (create the access_log folder if it does not already exist), and the data file contains the following data:</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen916412354232">| All | Heap used for segments | | 18.6403 | MB |
|
|
| All | Heap used for doc values | | 0.119289 | MB |
|
|
| All | Heap used for terms | | 17.4095 | MB |
|
|
| All | Heap used for norms | | 0.0767822 | MB |
|
|
| All | Heap used for points | | 0.225246 | MB |
|
|
| All | Heap used for stored fields | | 0.809448 | MB |
|
|
| All | Segment count | | 101 | |
|
|
| All | Min Throughput | index-append | 66232.6 | docs/s |
|
|
| All | Median Throughput | index-append | 66735.3 | docs/s |
|
|
| All | Max Throughput | index-append | 67745.6 | docs/s |
|
|
| All | 50th percentile latency | index-append | 510.261 | ms |</pre>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li416420352232">On the server where Logstash is deployed, run the following command to create configuration file <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath13272639293"><b>logstash-simple.conf</b></span> in the Logstash installation directory:<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen8164235182310">cd /<em id="EN-US_TOPIC_0000001955726478__i14571351299"><Logstash installation directory></em>/
|
|
vi logstash-simple.conf</pre>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li416415355230">Enter the following content in <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath147593712298"><b>logstash-simple.conf</b></span>:<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen151641235102317">input {
|
|
<em id="EN-US_TOPIC_0000001955726478__i2316111019294">Location of data</em>
|
|
}
|
|
filter {
|
|
<em id="EN-US_TOPIC_0000001955726478__i921714132912">Related data processing</em>
|
|
}
|
|
output {
|
|
elasticsearch {
|
|
hosts => "<EIP of the <em id="EN-US_TOPIC_0000001955726478__i7324132473314"><em id="EN-US_TOPIC_0000001955726478__i432452443310">jump host</em></em>>:<em id="EN-US_TOPIC_0000001955726478__i1132452411338"><Number of the port assigned external network access permissions on the jump host></em>"
|
|
(Optional) If communication encryption has been enabled for the cluster, you need to add the following configuration:
|
|
ssl => true
|
|
ssl_certificate_verification => false
|
|
}
|
|
}</pre>
|
|
<ul id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ul1416414353230"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1216411359233">The <strong id="EN-US_TOPIC_0000001955726478__b616420264291">input</strong> parameter indicates the data source. Set this parameter based on the actual conditions. For details about the <strong id="EN-US_TOPIC_0000001955726478__b13854133017299">input</strong> parameter and its usage, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/input-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/input-plugins.html</a></li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li9164735172310">The <strong id="EN-US_TOPIC_0000001955726478__b1882123217293">filter</strong> parameter specifies the data processing method. For example, extract and process logs to convert unstructured information into structured information. For details about the <strong id="EN-US_TOPIC_0000001955726478__b690723812911">filter</strong> parameter and its usage, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/filter-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/filter-plugins.html</a></li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li121641735122315">The <strong id="EN-US_TOPIC_0000001955726478__b1954914172910">output</strong> parameter indicates the destination address of the data. For details about the <strong id="EN-US_TOPIC_0000001955726478__b774344517295">output</strong> parameter and its usage, visit <a href="https://www.elastic.co/guide/en/logstash/current/output-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/output-plugins.html</a>. Replace <<em id="EN-US_TOPIC_0000001955726478__i8299163115331">EIP address of the <em id="EN-US_TOPIC_0000001955726478__i182991331133317">jump host</em></em>> with the IP address (with <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue229910312332"><b>EIP</b></span>) of the created jump host in the <span class="parmname" id="EN-US_TOPIC_0000001955726478__parmname22991531103318"><b>IP Address</b></span> column in the ECS list on the ECS management console. <em id="EN-US_TOPIC_0000001955726478__i1666625532915"><The port accessible from the public network on the jump host></em> is the port number obtained in <a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1648853125014">1</a>, for example, <strong id="EN-US_TOPIC_0000001955726478__b36664555292">9200</strong>.</li></ul>
|
|
<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p1164143513237">Consider the data files in the <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath12369121017309"><b>/tmp/access_log/</b></span> path mentioned in <a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li5164153542312">4</a> as an example. Assume that data ingestion starts from the first row of the file, the filtering condition is left unspecified (indicating no data processing operations are performed), the public IP address and port number of the jump host are <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue4267132413301"><b>192.168.0.227</b></span> and <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue8268202443012"><b>9200</b></span>, respectively, and the name of the destination index is <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue1457653093015"><b>myindex</b></span>. Edit the configuration file as follows, and enter <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue9766171111436"><b>:wq</b></span> to save the change and exit.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen816443562316">input {
|
|
file{
|
|
path => "/tmp/access_log/*"
|
|
start_position => "beginning"
|
|
}
|
|
}
|
|
filter {
|
|
}
|
|
output {
|
|
elasticsearch {
|
|
hosts => "192.168.0.227:9200"
|
|
index => "myindex"
|
|
|
|
}
|
|
}</pre>
|
|
<div class="note" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_note29452405213"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p12945184092114">If a license error is reported, set <strong id="EN-US_TOPIC_0000001955726478__b3933538173012">ilm_enabled</strong> to <strong id="EN-US_TOPIC_0000001955726478__b39331438153015">false</strong> to try and rectify the error.</p>
|
|
</div></div>
|
|
<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p154751181388">If the cluster has the security mode enabled, you need to download a certificate first.</p>
|
|
<ol type="a" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ol45911713483"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001268314505_li07251912434"><a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_section697213217486">Obtaining the Security Certificate</a>.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li107224618910">Store the downloaded certificate to the server where Logstash is deployed.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li84281357121313">Modify the <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath81711353113018"><b>logstash-simple.conf</b></span> configuration file.<div class="p" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p14897136193">Consider the data files in the <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath1052913568300"><b>/tmp/access_log/</b></span> path mentioned in <a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li5164153542312">4</a> as an example. Assume that data ingestion starts from the first row of the file, the filtering condition is left unspecified (indicating no data processing operations are performed), the public IP address and port number of the jump host are <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue3683851203614"><b>192.168.0.227</b></span> and <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue1268315114367"><b>9200</b></span>, respectively, The name of the index for importing data is <strong id="EN-US_TOPIC_0000001955726478__b483953473311">myindex</strong>, and the certificate is stored in <strong id="EN-US_TOPIC_0000001955726478__b16839634183316">/logstash/config/CloudSearchService.cer</strong>. Edit the configuration file as follows, and enter <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue12181354438"><b>:wq</b></span> to save the change and exit.<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_screen12598143419580">input{
|
|
file {
|
|
path => "/tmp/access_log/*"
|
|
start_position => "beginning"
|
|
}
|
|
}
|
|
filter {
|
|
}
|
|
output{
|
|
elasticsearch{
|
|
hosts => ["https://192.168.0.227:9200"]
|
|
index => "myindex"
|
|
user => "admin" # Username for accessing the security-mode cluster
|
|
password => "******" # Password for accessing the security-mode cluster
|
|
cacert => "/logstash/config/CloudSearchService.cer"
|
|
manager_template => false
|
|
ilm_enabled => false
|
|
ssl => true
|
|
ssl_certificate_verification => false
|
|
}
|
|
}</pre>
|
|
</div>
|
|
</li></ol>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li14164163582313">Run the following command to import the data collected by Logstash to the cluster:<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen121641135152312">./bin/logstash -f logstash-simple.conf</pre>
|
|
<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p3258910205713">This command must be executed in the directory where the <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath13581810143219"><b>logstash-simple.conf</b></span> file is located. For example, if the <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath1668211163327"><b>logstash-simple.conf</b></span> file is stored in <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath5682116163216"><b>/root/logstash-7.1.1/</b></span>, navigate to this directory before executing the command.</p>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000002271391393_en-us_topic_0000001268594549_li5509181942419"><span id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_ph152259618315">Log in to the CSS management console.</span></li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000002271391393_li4250043448">In the navigation pane on the left, choose <span class="uicontrol" id="EN-US_TOPIC_0000001955726478__uicontrol35231521103216"><b>Clusters > OpenSearch</b></span>.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000002271391393_en-us_topic_0000001268154521_li6012554340">In the cluster list, find the destination cluster, and click <strong id="EN-US_TOPIC_0000001955726478__b6845182410324">Dashboards</strong> in the <strong id="EN-US_TOPIC_0000001955726478__b884592413214">Operation</strong> column to log in to OpenSearch Dashboards.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000002271391393_li270075935613">In the left navigation pane, choose <strong id="EN-US_TOPIC_0000001955726478__b0667830163214">Dev Tools</strong>.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li418033592313">On the <strong id="EN-US_TOPIC_0000001955726478__b17580103373210">Console</strong> page of OpenSearch Dashboards, search for the ingested data.<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p118053518234">Run the following command to search for data. Check the search results. If they are consistent with the ingested data, data ingestion has been successful.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen1718063513237">GET myindex/_search</pre>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_section1098217174335"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_section1098217174335"></a><a name="en-us_topic_0000001938218520_en-us_topic_0000001223914344_section1098217174335"></a><h4 class="sectiontitle">Ingesting Data When Logstash Is Deployed Using an ECS</h4><p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p110725613115"><a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_fig124034434127">Figure 2</a> illustrates the data ingestion process when Logstash is deployed on an ECS that resides in the same VPC as the destination cluster.</p>
|
|
<div class="fignone" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_fig124034434127"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_fig124034434127"></a><a name="en-us_topic_0000001938218520_en-us_topic_0000001223914344_fig124034434127"></a><span class="figcap"><b>Figure 2 </b>Data ingestion process when Logstash is deployed on an ECS</span><br><span><img class="eddx" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_image149477361988" src="figure/en-us_image_0000002378998570.png"></span></div>
|
|
<ol id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ol3524114322310"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1852474313235">Make sure the ECS where Logstash is deployed and the destination cluster reside in the same VPC, port <strong id="EN-US_TOPIC_0000001955726478__b1785301913814">9200</strong> is opened in the ECS's security group to allow external network access, and an EIP has been associated with the ECS.<ul id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_ul8341199611"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_li1134598620">If there are multiple servers in a VPC, you only need to associate an EIP with one of these servers. Switch to the node where Logstash is deployed from the node with which the EIP is associated.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_li5344913618">If a private line or VPN is available, there is no need for an EIP.</li></ul>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1652411439236"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1652411439236"></a><a name="en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1652411439236"></a>Use PuTTY to log in to the ECS.<div class="p" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p12524643162314"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1652411439236"></a><a name="en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1652411439236"></a>For example, the file <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath6430144011383"><b>access_20181029_log</b></span> is stored in the <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath1743084018383"><b>/tmp/access_log/</b></span> path of the ECS, and the file contains the following data:<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen19524243192319">| All | Heap used for segments | | 18.6403 | MB |
|
|
| All | Heap used for doc values | | 0.119289 | MB |
|
|
| All | Heap used for terms | | 17.4095 | MB |
|
|
| All | Heap used for norms | | 0.0767822 | MB |
|
|
| All | Heap used for points | | 0.225246 | MB |
|
|
| All | Heap used for stored fields | | 0.809448 | MB |
|
|
| All | Segment count | | 101 | |
|
|
| All | Min Throughput | index-append | 66232.6 | docs/s |
|
|
| All | Median Throughput | index-append | 66735.3 | docs/s |
|
|
| All | Max Throughput | index-append | 67745.6 | docs/s |
|
|
| All | 50th percentile latency | index-append | 510.261 | ms |</pre>
|
|
</div>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li9524143172314">Run the following command to create configuration file <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath2284815163917"><b>logstash-simple.conf</b></span> in the Logstash installation directory:<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen2524104318230">cd /<em id="EN-US_TOPIC_0000001955726478__i659422018391"><Logstash installation directory></em>/
|
|
vi logstash-simple.conf</pre>
|
|
<div class="p" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p15241243162314">Enter the following content in <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath1595572316398"><b>logstash-simple.conf</b></span>:<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen652484313231">input {
|
|
<em id="EN-US_TOPIC_0000001955726478__i169061627163913">Location of data</em>
|
|
}
|
|
filter {
|
|
<em id="EN-US_TOPIC_0000001955726478__i039533318397">Related data processing</em>
|
|
}
|
|
output {
|
|
elasticsearch{
|
|
hosts => "<em id="EN-US_TOPIC_0000001955726478__i128978365392"><Private network address and port number of the node</em>>"}
|
|
(Optional) If communication encryption has been enabled for the cluster, you need to add the following configuration:
|
|
ssl => true
|
|
ssl_certificate_verification => false
|
|
}</pre>
|
|
<ul id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ul165241743132318"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li952444318236">The <strong id="EN-US_TOPIC_0000001955726478__b4367125883913">input</strong> parameter indicates the data source. Set this parameter based on the actual conditions. For details about the <strong id="EN-US_TOPIC_0000001955726478__b7892151194015">input</strong> parameter and its usage, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/input-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/input-plugins.html</a></li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1524194372320">The <strong id="EN-US_TOPIC_0000001955726478__b6455101514018">filter</strong> parameter specifies the data processing method. For example, extract and process logs to convert unstructured information into structured information. For details about the <strong id="EN-US_TOPIC_0000001955726478__b167517812377">filter</strong> parameter and its usage, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/filter-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/filter-plugins.html</a></li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li552418439238">The <strong id="EN-US_TOPIC_0000001955726478__b1657152444015">output</strong> parameter indicates the destination address of the data. For details about the <strong id="EN-US_TOPIC_0000001955726478__b112352176379">output</strong> parameter and its usage, visit <a href="https://www.elastic.co/guide/en/logstash/current/output-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/output-plugins.html</a>. <em id="EN-US_TOPIC_0000001955726478__i16193162483719"><private network address and port number of a node></em> refers to the private network address and port number of a node in the cluster.<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p18524164317236">If the cluster contains multiple nodes, you are advised to replace the value of <em id="EN-US_TOPIC_0000001955726478__i16495734124010"><Private network address and port number of a node></em> with the private network addresses and port numbers of all nodes in the cluster to prevent node faults. Use commas (,) to separate the nodes' private network addresses and port numbers. The following is an example:</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen175241043182316">hosts => ["192.168.0.81:9200","192.168.0.24:9200"]</pre>
|
|
<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p452413433238">If the cluster contains only one node, the format is as follows:</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen1352474320237">hosts => "192.168.0.81:9200"</pre>
|
|
</li></ul>
|
|
</div>
|
|
<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p3524043192311">Consider the data files in the <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath716254117405"><b>/tmp/access_log/</b></span> path mentioned in <a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1652411439236">2</a> as an example. Assume that data ingestion starts from the first row of the file, the filtering condition is left unspecified (indicating no data processing operations are performed), the private network address and port number of the node in the destination cluster are <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue957905754018"><b>192.168.0.81</b></span> and <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue2057915576401"><b>9200</b></span>, respectively, and the name of the destination index is <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue15351160164117"><b>myindex</b></span>. Edit the configuration file as follows, and enter <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue451123124111"><b>:wq</b></span> to save the change and exit.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen952414316233">input {
|
|
file{
|
|
path => "/tmp/access_log/*"
|
|
start_position => "beginning"
|
|
}
|
|
}
|
|
filter {
|
|
}
|
|
output {
|
|
elasticsearch {
|
|
hosts => "192.168.0.81:9200"
|
|
index => "myindex"
|
|
|
|
}
|
|
}</pre>
|
|
<div class="p" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p0370830125513">If the cluster has the security mode enabled, you need to download a certificate first.<ol type="a" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_ol1817087143812"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_en-us_topic_0111222977_li45915135814"><a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_section697213217486">Obtaining the Security Certificate</a>.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_en-us_topic_0111222977_li107224618910">Store the downloaded certificate to the server where Logstash is deployed.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_en-us_topic_0111222977_li84281357121313">Modify the <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath136464224418"><b>logstash-simple.conf</b></span> configuration file.<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_en-us_topic_0111222977_p14897136193">Consider the data files in the <span class="filepath" id="EN-US_TOPIC_0000001955726478__filepath17188105016479"><b>/tmp/access_log/</b></span> path mentioned in step <a href="#EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1652411439236">2</a> as an example. Assume that data ingestion starts from the first row of the file, the filtering condition is left unspecified (indicating no data processing operations are performed), the public IP address and port number of the jump host are <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue09195824215"><b>192.168.0.227</b></span> and <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue1191916812428"><b>9200</b></span>, respectively, The name of the index for importing data is <strong id="EN-US_TOPIC_0000001955726478__b137085382335">myindex</strong>, and the certificate is stored in <strong id="EN-US_TOPIC_0000001955726478__b107081638133314">/logstash/config/CloudSearchService.cer</strong>. Edit the configuration file as follows, and enter <span class="parmvalue" id="EN-US_TOPIC_0000001955726478__parmvalue1131122793818"><b>:wq</b></span> to save the change and exit.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_screen11621184016579">input{
|
|
file {
|
|
path => "/tmp/access_log/*"
|
|
start_position => "beginning"
|
|
}
|
|
}
|
|
filter {
|
|
}
|
|
output{
|
|
elasticsearch{
|
|
hosts => ["https://192.168.0.227:9200"]
|
|
index => "myindex"
|
|
user => "admin" # Username for accessing the security-mode cluster
|
|
password => "******" # Password for accessing the security-mode cluster
|
|
cacert => "/logstash/config/CloudSearchService.cer"
|
|
manager_template => false
|
|
ilm_enabled => false
|
|
ssl => true
|
|
ssl_certificate_verification => false
|
|
}
|
|
}</pre>
|
|
</li></ol>
|
|
</div>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li115241243142312">Run the following command to import the ECS data collected by Logstash to the cluster:<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen3524143132312">./bin/logstash -f logstash-simple.conf</pre>
|
|
</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_li9597192517411"><span id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_ph45978258410">Log in to the CSS management console.</span></li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_li185989254410">In the navigation pane on the left, choose <span class="uicontrol" id="EN-US_TOPIC_0000001955726478__uicontrol127751049123811"><b>Clusters > OpenSearch</b></span>.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_li2598192511413">In the cluster list, find the target cluster, and click <strong id="EN-US_TOPIC_0000001955726478__b20824133711383">Dashboards</strong> in the <strong id="EN-US_TOPIC_0000001955726478__b178246372380">Operation</strong> column to log in to OpenSearch Dashboards.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_li85989251548">In the left navigation pane, choose <strong id="EN-US_TOPIC_0000001955726478__b1967923412384">Dev Tools</strong>.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_li1252410433236">On the <strong id="EN-US_TOPIC_0000001955726478__b205568554516">Console</strong> page of OpenSearch Dashboards, search for the ingested data.<p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_p115241343172312">Run the following command to search for data. Check the search results. If they are consistent with the ingested data, data ingestion has been successful.</p>
|
|
<pre class="screen" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_en-us_topic_0000001223914344_screen75241543182320">GET myindex/_search</pre>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_section697213217486"><a name="EN-US_TOPIC_0000001955726478__en-us_topic_0000001938218520_section697213217486"></a><a name="en-us_topic_0000001938218520_section697213217486"></a><h4 class="sectiontitle">Obtaining the Security Certificate</h4><p id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_p14588122314477">To access a security-mode OpenSearch cluster that uses HTTPS, a security certificate must be loaded. To obtain this security certificate (CloudSearchService.cer), follow these steps:</p>
|
|
<ol id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_ol65881523124710"><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_li3588723204713"><span id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_ph358812374717">Log in to the CSS management console.</span></li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_li1058815239474">In the navigation pane on the left, choose <span class="uicontrol" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_uicontrol724454458"><b>Clusters > OpenSearch</b></span>.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_li2588923184720">In the cluster list, click the name of the target cluster. The cluster information page is displayed.</li><li id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_li15588142316479">Click the <span class="wintitle" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_wintitle1205837049"><b>Overview</b></span> tab. In the <span class="wintitle" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_wintitle684020822"><b>Configuration</b></span> area, click <strong id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_b1541397834">Download Certificate</strong> next to <span class="uicontrol" id="EN-US_TOPIC_0000001955726478__en-us_topic_0000002338102386_uicontrol1874704973"><b>HTTPS Access</b></span>.</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0059.html">Importing Data to an OpenSearch Cluster</a></div>
|
|
</div>
|
|
</div>
|
|
|