forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
87 lines
13 KiB
HTML
87 lines
13 KiB
HTML
<a name="mrs_01_1061"></a><a name="mrs_01_1061"></a>
|
|
|
|
<h1 class="topictitle1">Typical Scenario: Collecting Local Static Logs and Uploading Them to Kafka</h1>
|
|
<div id="body1590374510129"><div class="section" id="mrs_01_1061__sb4292d86ee4d4301a6ed8d94c759bbf0"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1061__a1a87f111a4994e05bedaa228aa22d17e">This section describes how to use the Flume client to collect static logs from a local host and save them to the topic list (test1) of Kafka.</p>
|
|
<p id="mrs_01_1061__p19632144014147">This section applies to MRS 3.<em id="mrs_01_1061__i4485192917554">x</em> or later clusters.</p>
|
|
<div class="note" id="mrs_01_1061__n9a1d4f6c843f4dcb8debb5163332278a"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p class="text" id="mrs_01_1061__a40295110318540328da22f742b731724">By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see <a href="mrs_01_1069.html">Configuring the Encrypted Transmission</a>. The configuration applies to scenarios where only the Flume is configured, for example, Spooldir Source+Memory Channel+Kafka Sink.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1061__sfc4c684a812747cf8c3afc3b8b429735"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1061__u374541c747074b609f90117bae1a6198"><li id="mrs_01_1061__l8f5176ac0c5c4e94bf5c50c6e4633edc">The cluster has been installed, including the Kafka and Flume services.</li><li id="mrs_01_1061__li3520154014471">The Flume client has been installed. For details, see <span id="mrs_01_1061__ph63412418173"><a href="https://docs.otc.t-systems.com/cmpntguide/mrs/mrs_01_0392.html" target="_blank" rel="noopener noreferrer">Installing the Flume Client</a></span>.</li><li id="mrs_01_1061__l5081f7a101754664b448c74312c8c4cf">The network environment of the cluster is secure.</li><li id="mrs_01_1061__l9dd282823a824afaac279e0ca570ae72">The system administrator has understood service requirements and prepared Kafka administrator <strong id="mrs_01_1061__a9a30eaf6867e4eb7a7780b2581d5f8c1">flume_kafka</strong>.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1061__sdfa5f91951c84fe79e83714e653a3fc2"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1061__oa5c9a24d30a74f52b7b2c6db2c241100"><li id="mrs_01_1061__l02501c0cfe9a4fbd809f2c7d4566a41a"><span>Set Flume parameters.</span><p><div class="p" id="mrs_01_1061__p3626173914490">Use the Flume configuration tool on Manager to configure the Flume role client parameters and generate a configuration file.<ol type="a" id="mrs_01_1061__o7fab656c6fe7470a98344503e8634778"><li id="mrs_01_1061__lbab9f3739a694163aba3fc51104ea36f">Log in to FusionInsight Manager. Choose <strong id="mrs_01_1061__b4470164095314">Cluster</strong> > <strong id="mrs_01_1061__b130244759102134">Services</strong> > <strong id="mrs_01_1061__b813114279102134">Flume</strong> > <strong id="mrs_01_1061__b301622253102134">Configuration Tool</strong>.</li><li id="mrs_01_1061__l2776809973c144238c5478b5a7194c20">Set <strong id="mrs_01_1061__b1218125216517">Agent Name</strong> to <strong id="mrs_01_1061__b118318524513">client</strong>. Select and drag the source, channel, and sink to be used to the GUI on the right, and connect them.<p id="mrs_01_1061__a75859d8820cd4d458bdb0dc6264ef699">Use SpoolDir Source, Memory Channel, and Kafka Sink.</p>
|
|
</li><li id="mrs_01_1061__le4bfc77b17f84934b3958c6bcc05d907">Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to <a href="#mrs_01_1061__table1162101394616">Table 1</a> based on the actual environment.<div class="note" id="mrs_01_1061__nb4cd07b6914946a4b7627ff8c19f259e"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1061__u5a4b58996f8f4212bb25a699e4fe0ab3"><li id="mrs_01_1061__l5b6660b5c0194373a62a1b5bab02af92">If you want to continue using the <strong id="mrs_01_1061__b2074172613216">properties.propretites</strong> file by modifying it, log in to FusionInsight Manager, choose <strong id="mrs_01_1061__b675320411935">Cluster</strong> > <strong id="mrs_01_1061__b71919371368">Services</strong>. On the page that is displayed, choose <strong id="mrs_01_1061__b199271050161216">Flume</strong>. On the displayed page, click the <strong id="mrs_01_1061__b13156103719144">Configuration Tool</strong> tab, click <strong id="mrs_01_1061__b6694542145">Import</strong>, import the file, and modify the configuration items related to non-encrypted transmission.</li><li id="mrs_01_1061__l0f621521cc9f4894ab3685beb8615426">It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.</li></ul>
|
|
</div></div>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1061__table1162101394616"></a><a name="table1162101394616"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1061__table1162101394616" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters to be modified for the Flume role client</caption><thead align="left"><tr id="mrs_01_1061__row9162121304616"><th align="left" class="cellrowborder" valign="top" width="27.63%" id="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.1"><p id="mrs_01_1061__p316281313464">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="40.339999999999996%" id="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.2"><p id="mrs_01_1061__p181621513184610">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="32.029999999999994%" id="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.3"><p id="mrs_01_1061__p141621513134617">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1061__row1716281344610"><td class="cellrowborder" valign="top" width="27.63%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1061__p1416219137464">Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.339999999999996%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1061__p2162141318462">The value must be unique and cannot be left blank.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32.029999999999994%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1061__p5162513104611">test</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1061__row19162713164610"><td class="cellrowborder" valign="top" width="27.63%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1061__p81623131466">spoolDir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.339999999999996%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1061__p316231324610">Specifies the directory where the file to be collected resides. This parameter cannot be left blank. The directory needs to exist and have the write, read, and execute permissions on the flume running user.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32.029999999999994%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1061__p516241374620">/srv/BigData/hadoop/data1/zb</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1061__row1116221311464"><td class="cellrowborder" valign="top" width="27.63%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1061__p61627136466">trackerDir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.339999999999996%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1061__p10162101317465">Specifies the path for storing the metadata of files collected by Flume.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32.029999999999994%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1061__p17162101311464">/srv/BigData/hadoop/data1/tracker</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1061__row416213134467"><td class="cellrowborder" valign="top" width="27.63%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1061__p11162111320467">batchSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.339999999999996%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1061__p11162151354611">Specifies the number of events that Flume sends in a batch (number of data pieces). A larger value indicates higher performance and lower timeliness.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32.029999999999994%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1061__p14163213114610">61200</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1061__row216341316461"><td class="cellrowborder" valign="top" width="27.63%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1061__p10163101318461">kafka.topics</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.339999999999996%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1061__p141631913194619">Specifies the list of subscribed Kafka topics, which are separated by commas (,). This parameter cannot be left blank.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32.029999999999994%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1061__p131631213144615">test1</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1061__row11631513164617"><td class="cellrowborder" valign="top" width="27.63%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1061__p51631613174620">kafka.bootstrap.servers</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.339999999999996%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1061__p151631713174618">Specifies the bootstrap IP address and port list of Kafka. The default value is all Kafkabrokers in the Kafka cluster.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32.029999999999994%" headers="mcps1.3.3.2.1.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1061__p1116331314611">192.168.101.10:21007</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_1061__l14d98e844ee849a99592f46d8be65b86"><a name="mrs_01_1061__l14d98e844ee849a99592f46d8be65b86"></a><a name="l14d98e844ee849a99592f46d8be65b86"></a>Click <strong id="mrs_01_1061__b1557860543102134">Export</strong> to save the <strong id="mrs_01_1061__b1116006932102134">properties.properties</strong> configuration file to the local server.</li></ol>
|
|
</div>
|
|
</p></li><li id="mrs_01_1061__li451912490551"><span>Upload the configuration file.</span><p><p id="mrs_01_1061__p15504195085518">Upload the file exported in <a href="#mrs_01_1061__l14d98e844ee849a99592f46d8be65b86">1.d</a> to the <em id="mrs_01_1061__i4924175816416">Flume client installation directory</em><strong id="mrs_01_1061__b1823910175429">/fusioninsight-flume-</strong><span id="mrs_01_1061__text0410535195415"><em id="mrs_01_1061__i174101235175418">Flume component version number</em></span><strong id="mrs_01_1061__b6984113415429">/conf</strong> directory of the cluster.</p>
|
|
</p></li></ol><ol start="3" id="mrs_01_1061__o822164196a4a487ab1025392c4fae8d6"><li id="mrs_01_1061__l846e2d5f007e4480912a32a5702a5855"><span>Verify log transmission.</span><p><ol type="a" id="mrs_01_1061__oc6ccc3660891409ebef5e7e8bb925965"><li id="mrs_01_1061__l8fc0b7d2b78d4890a51e7b9edaeab3b7">Log in to the Kafka client.<p id="mrs_01_1061__a60b9ddf8c4a04b30b94c9d45fa62374d"><a name="mrs_01_1061__l8fc0b7d2b78d4890a51e7b9edaeab3b7"></a><a name="l8fc0b7d2b78d4890a51e7b9edaeab3b7"></a><strong id="mrs_01_1061__en-us_topic_0060039167_b190688314321">cd </strong> <em id="mrs_01_1061__i8780114694319">Kafka client installation directory</em><strong id="mrs_01_1061__a3a813fe608ad4df98c1ce6fb8bed3ecb">/Kafka/kafka</strong></p>
|
|
<p id="mrs_01_1061__a5a0e1114105349f3a5827313be1a5e69"><strong id="mrs_01_1061__b524242443102134">kinit flume_kafka</strong> (Enter the password.)</p>
|
|
</li><li id="mrs_01_1061__l8a510158b48749c0b773c7f8032c2a05">Read data from a Kafka topic.<p id="mrs_01_1061__ac037886069764cdc801c3a5ceff831ab"><a name="mrs_01_1061__l8a510158b48749c0b773c7f8032c2a05"></a><a name="l8a510158b48749c0b773c7f8032c2a05"></a><strong id="mrs_01_1061__b560744742102134">bin/kafka-console-consumer.sh --topic</strong><em id="mrs_01_1061__i1908452769102134"> topic name</em><strong id="mrs_01_1061__b605046655102134"> --bootstrap-server</strong><em id="mrs_01_1061__i1327706676102134"> Kafka service IP address of the node where the role instance is located</em><strong id="mrs_01_1061__b214267791102134">: 21007 --consumer.config config/consumer.properties --from-beginning</strong></p>
|
|
<p id="mrs_01_1061__a1a6638a05c5246c881eb0d352140a1f0">The system displays the contents of the file to be collected.</p>
|
|
<pre class="screen" id="mrs_01_1061__s921b4139dc0b4ec18fabafbd711337dc">[root@host1 kafka]# bin/kafka-console-consumer.sh --topic test1 --bootstrap-server 192.168.101.10:21007 --consumer.config config/consumer.properties --from-beginning
|
|
Welcome to flume</pre>
|
|
</li></ol>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1059.html">Non-Encrypted Transmission</a></div>
|
|
</div>
|
|
</div>
|
|
|