forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
1117 lines
112 KiB
HTML
1117 lines
112 KiB
HTML
<a name="mrs_01_0396"></a><a name="mrs_01_0396"></a>
|
|
|
|
<h1 class="topictitle1">Flume Configuration Parameter Description</h1>
|
|
<div id="body1589421630519"><p id="mrs_01_0396__p18797440517">For versions earlier than MRS 3.x, configure Flume parameters in the <strong id="mrs_01_0396__b141671814193117">properties.properties</strong> file.</p>
|
|
<p id="mrs_01_0396__p578615408177">For MRS 3.x or later, some parameters can be configured on Manager.</p>
|
|
<div class="section" id="mrs_01_0396__sc054f7820b7a44bf8ff441600796e7a7"><h4 class="sectiontitle">Overview</h4><p id="mrs_01_0396__aee5bf059f1654c0b9d57ea6afa5f962d">This section describes how to configure the sources, channels, and sinks of Flume, and modify the configuration items of each module.</p>
|
|
<p id="mrs_01_0396__p1799992519214">For MRS 3.x or later, log in to FusionInsight Manager and choose <strong id="mrs_01_0396__b290811223309">Cluster</strong> > <strong id="mrs_01_0396__b185432512304">Services</strong> > <strong id="mrs_01_0396__b7533228113018">Flume</strong>. On the displayed page, click the <strong id="mrs_01_0396__b441634193018">Configuration Tool</strong> tab, select and drag the source, channel, and sink to be used to the GUI on the right, and double-click them to configure corresponding parameters. Parameters such as <strong id="mrs_01_0396__b78471934267">channels</strong> and <strong id="mrs_01_0396__b21031868263">type</strong> are configured only in the client configuration file <strong id="mrs_01_0396__b998713252266">properties.properties</strong>, the path of which is <em id="mrs_01_0396__i19818816210">Flume client installation directory</em><strong id="mrs_01_0396__b177161552162013">/fusioninsight-flume-</strong><span id="mrs_01_0396__text147161552102019"><em id="mrs_01_0396__i21974882117">Flume version</em></span><strong id="mrs_01_0396__b117166520200">/conf/properties.properties</strong>.</p>
|
|
<div class="note" id="mrs_01_0396__nf5dc35ffb7274dbc8c6df7e280c1a980"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0396__a4d2971903e7345ef8dcc4059a8ee7914">You must input encrypted information for some configurations. For details on how to encrypt information, see <a href="mrs_01_0395.html">Using the Encryption Tool of the Flume Client</a>.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_0396__s4ad4b577d4164449b1fb6fa42d20f60f"><h4 class="sectiontitle">Common Source Configurations</h4><ul id="mrs_01_0396__u4d8711cf89a749cabf7a113bf218f589"><li id="mrs_01_0396__l3c6e0d66effd42d3aceab1868c7ab7fb"><strong id="mrs_01_0396__afea46997e67d4036a43b3c05b1035b32">Avro Source</strong><p id="mrs_01_0396__a70d2895523a14330b952e080828ccb01">An Avro source listens to the Avro port, receives data from the external Avro client, and places data into configured channels. <a href="#mrs_01_0396__tb4f2cc56cf3945f7bba26f90f1afaa79">Table 1</a> lists common configurations.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0396__tb4f2cc56cf3945f7bba26f90f1afaa79"></a><a name="tb4f2cc56cf3945f7bba26f90f1afaa79"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__tb4f2cc56cf3945f7bba26f90f1afaa79" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Common configurations of an Avro source</caption><thead align="left"><tr id="mrs_01_0396__r95b12f9e77fe4f10932363bf4cf676cb"><th align="left" class="cellrowborder" valign="top" width="30.209999999999997%" id="mcps1.3.4.2.1.3.2.4.1.1"><p id="mrs_01_0396__a79d3283c14df4c8ab76e4594c07d91fe"><strong id="mrs_01_0396__ac38afe043f8740c2b3e42f06cb2c76ef">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.4.2.1.3.2.4.1.2"><p id="mrs_01_0396__a074fca8ccf6043aa8fa6ef79a4826302"><strong id="mrs_01_0396__a34efd686ac3044008e17bddea0994a95">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="49.79%" id="mcps1.3.4.2.1.3.2.4.1.3"><p id="mrs_01_0396__a5969528eca5d4b79951f01ab8cf79e7e"><strong id="mrs_01_0396__ae0874bcd8e6645c69e068f4083e096d5">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__r31edcca6fe76445fb77d68327851aa77"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__ae6695e92559a4c358a818184e54538a5">channels</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a11feb867447c420e98fb7cf2949d06c8"><strong id="mrs_01_0396__a8c13f02c4d6f46b6b48fc6b33c0814fb">-</strong></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a3c49c21dcd384a80b88cfd4597cc2342">Specifies the channel connected to the source. Multiple channels can be configured. Use spaces to separate them.</p>
|
|
<p id="mrs_01_0396__en-us_topic_0066459131_p938226317465">In a single proxy process, sources and sinks are connected through channels. A source instance corresponds to multiple channels, but a sink instance corresponds only to one channel.</p>
|
|
<p id="mrs_01_0396__en-us_topic_0066459131_p194537817528">The format is as follows:</p>
|
|
<p id="mrs_01_0396__a611d0f461d044069bd10cc4e354323b5"><strong id="mrs_01_0396__af8ab924e6e2f44efb108303ea7b156e3"><Agent >.sources.<Source>.channels = <channel1> <channel2> <channel3>...</strong></p>
|
|
<p id="mrs_01_0396__afa63d5dc278a4eb3a7b8ef4e3fe933f7"><strong id="mrs_01_0396__a0831e4fe4f344c7b9bf50e08c45bf5cb"><Agent >.sinks.<Sink>.channels = <channel1></strong></p>
|
|
<p id="mrs_01_0396__p466654513347">This parameter can be configured only in the <strong id="mrs_01_0396__b10952133614299">properties.properties</strong> file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r604d95b589e34326b43b8246961c13a2"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a3426e4f75fe44a6d9dfd6b8a87af0e60">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a2eea6991e0754b1790b05d8cba519cf4">avro</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__adc087181482e49dd89b67dda99d80792">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__p82b69c4f4edf432489a0a6ccaa916eee"><b>avro</b></span>. The type of each source is a fixed value.</p>
|
|
<p id="mrs_01_0396__p62578013512">This parameter can be configured only in the <strong id="mrs_01_0396__b2503124762913">properties.properties</strong> file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r2de7efa6ec874d2a9099dc57c5c749c6"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a8663055e945c4a48878844456b7eea99">bind</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__ae1c42f8fd625426b9c0a563b1d030a87">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a7c695a28eabd403d9de0f04581d448fc">Specifies the host name or IP address associated with the source.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r2db207cff94c415c9f253b4527d8286b"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a0a625d0e98674d7fafbae79c18ed4a08">port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a53a8125eba28464eb0d79d62b22f37a6">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a2b23da59be8945389e0ed52b4ecdef75">Specifies the bound port number.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__re35f373dc20d47b6a40b838e18aaa06f"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__aae0082c633b143b0962b3f544cc00685">ssl</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a1413d624ce174241882aaf9f07a369d2">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a4b7eced0ae1c4df290d1469df8b5dd9a">Specifies whether to use SSL encryption.</p>
|
|
<ul id="mrs_01_0396__u12585ab92f9840de98769a8f2b04c52f"><li id="mrs_01_0396__l760e7341007d447b8e7062aba61cddf2">true</li><li id="mrs_01_0396__l565d6a4003a04e6e8c5883918945cce9">false</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3b13ac897ac64a31afa6d6cd67617a10"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__ac5f4a2d58a734cd69b98d9409dd330f7">truststore-type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a6c8979bd4f97441b8495e40bab3912a3">JKS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__ad5ad2843816949f18cd09a0178c1a457">Specifies the Java trust store type. Set this parameter to <strong id="mrs_01_0396__b11274302510">JKS</strong> or other truststore types supported by Java.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r5e1c7ca9fef0456291a252699801563f"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__afaa4c8155ce34fe4b8f5c183155d0880">truststore</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a70c07e177f024ad98e63aee34fe34aa2">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__acbdbde1cae6342558e1672f7c9df0423">Specifies the Java trust store file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r0fdf81e55cc14f60839ba95e55219eb0"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__ada4e930e46f54f8b9c085a013e0bf1fd">truststore-password</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a3c3dbb8156fc46f9a863b5f03fafbafc">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__addc19a81a38145da8791ada30a3b5fb2">Specifies the Java trust store password.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3d274aa843a64fc981c1095fcbbce84d"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a2f85ba21081645c2833a80fc92246f58">keystore-type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a8ad6ac2b9af247f1aef8baee37839a84">JKS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__afe92623aa1534606b80ba2b5730c5c9b">Specifies the key storage type. Set this parameter to <strong id="mrs_01_0396__b1637672119511">JKS</strong> or other truststore types supported by Java.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r6cacb30f252d4150a66b934c42b7a328"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__aaf504e4a5a384bfeba6ab6f728f7b04a">keystore</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a9870cb5dcb8740b69b3c7f7324820b36">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__abbd3c10427c840bbb57113d04eea1053">Specifies the key storage file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r07a1888321d248f18b5863917160bafd"><td class="cellrowborder" valign="top" width="30.209999999999997%" headers="mcps1.3.4.2.1.3.2.4.1.1 "><p id="mrs_01_0396__aae6b65b8ad7948a2a35c87cdd65c9470">keystore-password</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a4f0c3f18eb2a4e75baaeb73fcacde8ed">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.79%" headers="mcps1.3.4.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a1c4cb8fab24d42f78422a0b703f57131">Specifies the key storage password.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_0396__la88c6661d2e54dcda88c412b2e8025b3"><strong id="mrs_01_0396__ae4867a44465d4f31a948d4c3c7d64bdb">SpoolDir Source</strong><p id="mrs_01_0396__a74b23f2574724f0daec973974198b6e5">A SpoolDir source monitors and transmits new files that have been added to directories in quasi-real-time mode. Common configurations are as follows:</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__tdcd54cc71aac48129774ab69dce3bc28" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Common configurations of a SpoolDir source</caption><thead align="left"><tr id="mrs_01_0396__rc0a7ef03beb246d9961d0ba76362669c"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.4.2.2.3.2.4.1.1"><p id="mrs_01_0396__abb31cf27f6e84c60acb0cbb29a7b385c"><strong id="mrs_01_0396__a8adfbbaebc0b435f8515edccd6155c50">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.4.2.2.3.2.4.1.2"><p id="mrs_01_0396__a189f4169f1e14db38faf0a7101b621bd"><strong id="mrs_01_0396__ab506c459c9644c408823c70bfbb664fe">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.4.2.2.3.2.4.1.3"><p id="mrs_01_0396__ad1a82f299fcd4f3c959001d0b3e5d079"><strong id="mrs_01_0396__a82bd25be4d5648ab8693842fe7fde8de">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__r925a487476084b679b465db42285b9bc"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__abe6f50be881446dda904535cb3c4a8ee">channels</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a583c71a601294c968e4f2861dd2b683c">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a49193c0767874180b18f706e7508260e">Specifies the channel connected to the source. Multiple channels can be configured.</p>
|
|
<p id="mrs_01_0396__p179381134165610">This parameter can be configured only in the <strong id="mrs_01_0396__b14621123143014">properties.properties</strong> file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r6db4a07a872142d7a355bd6011f81ce5"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a21d6a328f1e146b49efc4d72cb066435">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__aaf43fca4421a4d3c9807cb5ea12e694c">spooldir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a4db268417ca14f5ab6febf5864691642">Type, which is set to <span class="parmvalue" id="mrs_01_0396__parmvalue1797608906162252"><b>spooldir</b></span>.</p>
|
|
<p id="mrs_01_0396__p16967103625616">This parameter can be configured only in the <strong id="mrs_01_0396__b3967193016302">properties.properties</strong> file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r64530bf6c0964c4ea52c43fba6c4e789"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a7c9e410235534da385b1dbb00bd40597">monTime</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a4e98807e9c9e41efab0261275a5696d9">0 (Disabled)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__ae0e09322c15a4dd988cac86fa2bc6f9c">Specifies the thread monitoring threshold. When the update time exceeds the threshold, the source is restarted. Unit: second</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3d8beb99995741f78ed8e8b1a8ad34d6"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a7eefd728f119419ca01e559ac5f4c90c">spoolDir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a6f77ef29b3184e9aaf2fba37b54c15b0">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a8383766ae29b41f0b8acb9b4c76ebfaf">Specifies the monitoring directory.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r4020c36fb23640a4bd49c9f64668824a"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__abade680165f44481bd00f45f7bdde449">fileSuffix</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a43aa794437b34265bb8ea0d72655b8cc">.COMPLETED</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__ade3f8d5ac46544999d5ca4805af17059">Specifies the suffix added after file transmission is complete.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rcd69060d2d5a42b7a3f2ad8df4110a5b"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__ae8cfe98e76244af0897f7744e1a230ce">deletePolicy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a7be5a04556b1463eae321dda172eae05">never</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a5e463ba36b3341358812417fd4336f75">Specifies the source file deletion policy after file transmission is complete. The value can be either <span class="parmvalue" id="mrs_01_0396__parmvalue1973776866162741"><b>never</b></span> or <span class="parmvalue" id="mrs_01_0396__parmvalue207274888162741"><b>immediate</b></span>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__ra83bb5d5b78c401d8a6bc67be958bf08"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__ae2f3d448cd4a45839dd6fdb525b7d04a">ignorePattern</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a3ebb88c8bd2a44e19edd7467333c15fb">^$</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__ad8767aa3e361419e81ba38c61632d600">Specifies the regular expression of a file to be ignored.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r70fc585225304949b04f64ab443ccceb"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a399ade1e2b2e49ce819ff091e314c2fd">trackerDir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__aba8d643ee7e34585acbf2128399ff029">.flumespool</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a49999cebe13744cbaafdb4fedf0f1e34">Specifies the metadata storage path during data transmission.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3a8323bc71de44f09a80224f5bfc79d4"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a922bf55f330b41488c20d368c021502f">batchSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__ab21e3e21575a4b0f863ebad413804f87">1000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__ae47ef8b64dae43d1b8e98188996ad3ea">Specifies the source transmission granularity.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r921c0ab815764f5096ffbc7f900a7eca"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__af3c790c6b9b94e8f9072dc7e0f4a3486">decodeErrorPolicy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__ab2f64f30a25c4ea4b225eed8e609de67">FAIL</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a4bc0f407e387498d8a7aec76f05b1707">Specifies the code error policy. This parameter can be configured only in the <strong id="mrs_01_0396__b18620639153012">properties.properties</strong> file.</p>
|
|
<p id="mrs_01_0396__a077e503a47b14d85b5ce4b0e335d7bb6">The value can be <strong id="mrs_01_0396__b7133164701918">FAIL</strong>, <strong id="mrs_01_0396__b1882134910191">REPLACE</strong>, or <strong id="mrs_01_0396__b83621852101917">IGNORE</strong>.</p>
|
|
<p id="mrs_01_0396__a8163ceb19cc641f2938f0b5a0742b938"><strong id="mrs_01_0396__b166070564195">FAIL</strong>: Generate an exception and fail the parsing.</p>
|
|
<p id="mrs_01_0396__aac455ea5c34c4038aafc96b77474c949"><strong id="mrs_01_0396__b1058844115208">REPLACE</strong>: Replace the characters that cannot be identified with other characters, such as U+FFFD.</p>
|
|
<p id="mrs_01_0396__afaa0bc7b08274f1bb1d158243a0d9ce4"><strong id="mrs_01_0396__b199152557209">IGNORE</strong>: Discard character strings that cannot be parsed.</p>
|
|
<div class="note" id="mrs_01_0396__nf128abd49e424b61a331d1cbf9bca083"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_0396__ab2824af9101445d1b0040ca6d58eb0e2">If a code error occurs in the file, set <strong id="mrs_01_0396__b112377904825040">decodeErrorPolicy</strong> to <strong id="mrs_01_0396__b119909977925040">REPLACE</strong> or <strong id="mrs_01_0396__b133922835925040">IGNORE</strong>. Flume will skip the code error and continue to collect subsequent logs.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rb75118b3a9cd484c991c79d488887799"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__ac697301944934bffb730ad0331e6d8f9">deserializer</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__ac3bab9c5a9a54575bb7441b62646cf6d">LINE</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a5330c8c8f4d04437909ef3ecb8dd5763">Specifies the file parser. The value can be either <span class="parmvalue" id="mrs_01_0396__parmvalue157098364425040"><b>LINE</b></span> or <span class="parmvalue" id="mrs_01_0396__parmvalue93243443925040"><b>BufferedLine</b></span>.</p>
|
|
<ul id="mrs_01_0396__u94c4fccc4ce0464e97f915ef071157ac"><li id="mrs_01_0396__l16e334a7603e48e7af590953939f864d">When the value is set to <span class="parmname" id="mrs_01_0396__p51041a4b2a4b48ba8a637cf407e7bcb1"><b>LINE</b></span>, characters read from the file are transcoded one by one.</li><li id="mrs_01_0396__la972f761aebe4538a04ca57c74d21373">When the value is set to <span class="parmvalue" id="mrs_01_0396__p50ed58ed0f2e422096be0f74a5ae1b45"><b>BufferedLine</b></span>, one line or multiple lines of characters read from the file are transcoded in batches, which delivers better performance.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rf4cea585325c4720a8cc77f5a78b136e"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__aadcd567df75b4336b61e78537eca29f5">deserializer.maxLineLength</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__afc6c5d7abeb84915822e4df154c5fd8d">2048</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a32c66d9ffdd141b49c1abe89b530077a">Specifies the maximum length for resolution by line, ranging from 0 to 2,147,483,647.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r9b55b35a7a024e80abed9b8c08c4f8c2"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p914884165858">deserializer.maxBatchLine</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a64101eabcddf4e2fa918d8424c462543">1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a96a8654bf81e44809fcf10bee29ca5d1">Specifies the maximum number of lines for resolution by line. If multiple lines are set, <span class="parmvalue" id="mrs_01_0396__pc2f9bf28d62044899fd69d813f6b7fe8"><b>maxLineLength</b></span> must be set to a corresponding multiplier. For example, if <strong id="mrs_01_0396__b15352111892220">maxBatchLine</strong> is set to <strong id="mrs_01_0396__b19597533142212">2</strong>, <strong id="mrs_01_0396__b1653418249226">maxLineLength</strong> is set to <strong id="mrs_01_0396__b19921235172213">4096</strong> (2048 x 2).</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__ra9fe198347f644aca807b0e0e85d141c"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__ad279464d89b44206af6c0a74423a93ec">selector.type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__ae0dc7b69f8654d3483fe3ab75133d7bb">replicating</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a4a709636b86146a89c0f1b680eaa4c18">Specifies the selector type. The value can be either <span class="parmvalue" id="mrs_01_0396__parmvalue75682386417949"><b>replicating</b></span> or <span class="parmvalue" id="mrs_01_0396__parmvalue195801949817949"><b>multiplexing</b></span>.</p>
|
|
<ul id="mrs_01_0396__u7a78d9c83d2b49f4be4ecbcf3d5dc988"><li id="mrs_01_0396__l875bfcffa10c443aa14d23d7da552aa1"><span class="parmvalue" id="mrs_01_0396__pd642f62047744352ab9f76e0564b4fc4"><b>replicating</b></span> indicates that the same content is sent to each channel.</li><li id="mrs_01_0396__l74304b9d9a424282b9b18230d4ad44e3"><span class="parmvalue" id="mrs_01_0396__p42395d6736854e3aa1c80023ab77e5a1"><b>multiplexing</b></span> indicates that the content is sent only to certain channels according to the distribution rule.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r923f8ce7ad0a4cf5aaf55f01243ed0aa"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a778e3138fe404571b9efffce2e5df88f">interceptors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a328d73fcd3ca47a380b99c237b7ca656">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a2bdc73924b2e4d819304c9aac8c304d4">Specifies the interceptor. For details, see the <a href="https://flume.apache.org/FlumeUserGuide.html#flume-interceptors" target="_blank" rel="noopener noreferrer">Flume official document</a>.</p>
|
|
<p id="mrs_01_0396__p38761017155719">This parameter can be configured only in the <strong id="mrs_01_0396__b4724134833020">properties.properties</strong> file.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_0396__nfcdaf89ede4c44a981733bca45f329b8"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0396__a0c3457e7591d485d9d1ae981a9336bed">The Spooling source ignores the last line feed character of each event when data is read by line. Therefore, Flume does not calculate the data volume counters used by the last line feed character.</p>
|
|
</div></div>
|
|
</li><li id="mrs_01_0396__l9dc2d0ce9cff4417935d1f94972e7e87"><strong id="mrs_01_0396__aad83445cd2b14e778dc074bfff2a2f33">Kafka Source</strong><p id="mrs_01_0396__aace221dbf8ff457e92e956bfc6d5a08d">A Kafka source consumes data from Kafka topics. Multiple sources can consume data of the same topic, and the sources consume different partitions of the topic. Common configurations are as follows:</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__t1f7ac743f0b646f8844b243a746b6a4d" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Common configurations of a Kafka source</caption><thead align="left"><tr id="mrs_01_0396__rf5739e1e01e648adb3c7ac3c79acf300"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.4.2.3.3.2.4.1.1"><p id="mrs_01_0396__addf5b78cd82d4f15b70fe78dbd2bb595"><strong id="mrs_01_0396__a52fd5301e7e9441fb2976b87178eec75">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.4.2.3.3.2.4.1.2"><p id="mrs_01_0396__a6ba5c809799744499f380e0ba370bee2"><strong id="mrs_01_0396__a92d4975627bf4eb3a31aadd6351c4173">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.4.2.3.3.2.4.1.3"><p id="mrs_01_0396__a6561acc5a9ee41f0b7a43b37f6fcf893"><strong id="mrs_01_0396__ace7f0f90b66c4f528014dd673de30938">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__r936d999c7eaf4077a1739d84df8e216d"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a65a5c12652314886b108352dd7f6471a">channels</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__aaf48ff00ef0b44128850298d5e3f45c2">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__af020007f579d477c83a5fef4c0edae18">Specifies the channel connected to the source. Multiple channels can be configured.</p>
|
|
<p id="mrs_01_0396__p1483925185814">This parameter can be configured only in the <strong id="mrs_01_0396__b1622220599305">properties.properties</strong> file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r551f68ee2d1f418084436f905ebe1740"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__abcf0d826eac646e39551fa0f0bad7934">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__a0f483b2e2a0e4aa490e186b69575588f">org.apache.flume.source.kafka.KafkaSource</p>
|
|
<p id="mrs_01_0396__a1bc0a09a53f9494e8df02f94423490da"></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__ab4458eb89dcb4142b87566b101c95bdb">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__p4acc8544476942a3ab3ebc6a4e87ca43"><b>org.apache.flume.source.kafka.KafkaSource</b></span>.</p>
|
|
<p id="mrs_01_0396__p7997154195814">This parameter can be configured only in the <strong id="mrs_01_0396__b188686683113">properties.properties</strong> file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r33e92b363e57485586a4f1f44612d841"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__aa9568c2cd18c46e4a0567a9b2d1ff370">monTime</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p10022165858">0 (Disabled)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p811818165858">Specifies the thread monitoring threshold. When the update time exceeds the threshold, the source is restarted. Unit: second</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rce16dade0d01446fbb4f8326a37a32a9"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__ab2dc8e99ae5745999e671f3c6621c3a5">nodatatime</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__ac14ca43522cb44e3ada01e54c8b6821e">0 (Disabled)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a7105dd5d6df34759b162e7534114fedc">Specifies the alarm threshold. An alarm is triggered when the duration that Kafka does not release data to subscribers exceeds the threshold. Unit: second</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rc50d41d3adaf46f784e2eb7c393f237f"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a60378524c60844f28fa5ad2f7d2d71ac">batchSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__ac2b7fde2ccf643ddad2ab0868c0a2aa0">1000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__aac21be4000a44a69bf2ce886f64322e7">Specifies the number of events written into a channel at a time.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r88260e5cd4d7443783f66a8e0ba7b573"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a345c6b47bce1407fa423aae43a3be796">batchDurationMillis</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__afbcd8d607bba4dbba5ae3202453de613">1000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__ae269a13d1cfb401b9b80ea9c5fb87011">Specifies the maximum duration of topic data consumption at a time, expressed in milliseconds.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r4bb740cfcaf24696ae6149d3a30ac7e8"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a134b7e65766d4a92aa018705c782a7e6">keepTopicInHeader</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__ab1d8c7822be74ec9a9766961abaa4f8a">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a5da0d1f42d3c472496c31faaa269b167">Specifies whether to save topics in the event header. If topics are saved, topics configured in Kafka sinks become invalid.</p>
|
|
<ul id="mrs_01_0396__u088a8cef109b4d3a9827cc42c273777b"><li id="mrs_01_0396__l1924253414d0429fa90845b1a7ee1d7b">true</li><li id="mrs_01_0396__l6e816f5433c64d23b9fe45dd5800b467">false</li></ul>
|
|
<p id="mrs_01_0396__p20309151619019">This parameter can be configured only in the <strong id="mrs_01_0396__b18149915113114">properties.properties</strong> file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__ra29eef6c4dfb46798c818946039d13f3"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a151236c7c7ae41b4b959c1d5e4e7a8b7">keepPartitionInHeader</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__a2e4ad00d070344ee91943d0136410a59">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a31ac0057a74c415d8420ae6af43eb98d">Specifies whether to save partition IDs in the event header. If partition IDs are saved, Kafka sinks write data to the corresponding partitions.</p>
|
|
<ul id="mrs_01_0396__u489c3e99ba90479491f94a33dccfe381"><li id="mrs_01_0396__lc13b1ff5537b4a8ca8a756da8dcf11a0">true</li><li id="mrs_01_0396__l61439b2c5a454fd68035b09e9059de96">false</li></ul>
|
|
<p id="mrs_01_0396__p856714369011">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r1179794397874f3a9776a3f96b3e46e6"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a0323365ebf6146f9997f36ce5f3038c7">kafka.bootstrap.servers</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__a578a2bd29f6a423e99683500ca0c7c69">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a28d56a4673f84747810fb78348570bcf">Specifies the list of Broker addresses, which are separated by commas.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__ra7d7b51a04404857b921c4c9d72cbe22"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a012d83c28c3d46349c362554d68ddb77">kafka.consumer.group.id</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__ae6bc6f6669094d10ac329acde068fff7">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a5b4945ad4ec04df089b814d5d940e14b">Specifies the Kafka consumer group ID.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__raff3c84724ed482d8a6e749ad73b3a82"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__aca73a8fc480541c1b6ee5e583898b5f2">kafka.topics</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__a347fb3f56a674f08ae562b7106e381a5">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__ae3f84f1e7925419db0f5509d5db3829a">Specifies the list of subscribed Kafka topics, which are separated by commas (,).</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r0cfe887829734409a1b2c690d9a7999b"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a2e21c6decde042abaf0e5a57cef2151e">kafka.topics.regex</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__ad35de17590ca48c89c8946682bbb9ff7">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a80fdbf0055464e768323e99335570175">Specifies the subscribed topics that comply with regular expressions. <strong id="mrs_01_0396__b842352706173325">kafka.topics.regex</strong> has a higher priority than <strong id="mrs_01_0396__b842352706173331">kafka.topics</strong> and will overwrite <strong id="mrs_01_0396__b842352706173431">kafka.topics</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__ra49d559fb5354ef6b70167e5db2aa1b4"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a7dd1ca88cbad4ee2a32893c345f206f2">kafka.security.protocol</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__afd9c0ff0028148758c69751eb5d0c0a6">SASL_PLAINTEXT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__aa50e6215245d4f2f9ccf68dd20d52fcd">Specifies the security protocol of Kafka. The value must be set to <span class="parmvalue" id="mrs_01_0396__p31a1d721f69b435cbdf0049b426ccdec"><b>PLAINTEXT</b></span> for clusters in which Kerberos authentication is disabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__row44581793257"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__p194601915252">kafka.kerberos.domain.name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__p1346019182514">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__p54601698254">Specifies the value of <strong id="mrs_01_0396__b54523852516">default_realm</strong> of Kerberos in the Kafka cluster, which should be configured only for security clusters.</p>
|
|
<p id="mrs_01_0396__p5186125016111">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r07668b1271f54024a2340433ff94e6fb"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a58be15400e6f4b07818f53278e9991c8">Other Kafka Consumer Properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.3.3.2.4.1.2 "><p id="mrs_01_0396__acfbfe4ca95e147a5ae3588ab63ba990d">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a557ae6cc32c94a0c8247c3c3ce1bd5cc">Specifies other Kafka configurations. This parameter can be set to any consumption configuration supported by Kafka, and the <span class="parmvalue" id="mrs_01_0396__pf53e06fc8e7d4a2cbe3f92e1a8acc037"><b>.kafka</b></span> prefix must be added to the configuration.</p>
|
|
<p id="mrs_01_0396__p105763466116">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_0396__l26f5ba2898484132ae29a1311a3a7f3b"><strong id="mrs_01_0396__a8319a9e3b9f04c35b139dfe2c60772c0">Taildir Source</strong><p id="mrs_01_0396__ac8fe9b9c3b1b4a2ea69a95d095f267f5">A Taildir source monitors file changes in a directory and automatically reads the file content. In addition, it can transmit data in real time. <a href="#mrs_01_0396__t2c85090722c4451682fad2657a7bdc35">Table 4</a> lists common configurations.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0396__t2c85090722c4451682fad2657a7bdc35"></a><a name="t2c85090722c4451682fad2657a7bdc35"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__t2c85090722c4451682fad2657a7bdc35" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Common configurations of a Taildir source</caption><thead align="left"><tr id="mrs_01_0396__r04e6e8a6c378446da7da5086ed88d715"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.4.2.4.3.2.4.1.1"><p id="mrs_01_0396__a20af2b72d6b04711a71b34480ec06ec6"><strong id="mrs_01_0396__ae90645de27514b14a28d2efb4e4aee26">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="19.950000000000003%" id="mcps1.3.4.2.4.3.2.4.1.2"><p id="mrs_01_0396__a7d0a1297bc744fea9f884e3a99f3297b"><strong id="mrs_01_0396__ac5490e8aa9894f648abc04f4caf07d13">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50.05%" id="mcps1.3.4.2.4.3.2.4.1.3"><p id="mrs_01_0396__a31e4fd61aab24cf8bd3c071ef2a4ce41"><strong id="mrs_01_0396__a482ccb6681eb49b78c526ccfc92f8810">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__r9b778ecf252646ab8fffd7fbf280243a"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__aaf9e9494e3b246a583de27cf89d5b75f"><span id="mrs_01_0396__p0b6947e29867421b9919d5f6f5872c66">channels</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a485c777d050543f9b277d515b4e9a90f"><span id="mrs_01_0396__p7031fa9c082b42a2bf22982e967c02c4">-</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a58715099e69f4ac781c02691a9f5b1a1">Specifies the channel connected to the source. Multiple channels can be configured.</p>
|
|
<p id="mrs_01_0396__p17347152919611">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__raa6ae4b8664b4e70a8da14c32432b4d8"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__ae62ab04fdbc44fa2a7bee5693c621756"><span id="mrs_01_0396__p5bfcbbae00264eb6b6c62097e09e8f11">type</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a830edb6cf7054b12a0ba656d2c299680">taildir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__ae440c774df324f2d9516449860c463e8">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__parmvalue2068945869174211"><b>taildir</b></span>.</p>
|
|
<p id="mrs_01_0396__p102687312067">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r285c6aa7661043d19e4833ac39114843"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a3d35da52d63d46649d15a29e4871981b">filegroups</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__ac8064d8657ee4ab097c35b4857ed4ac9">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a31740878ca4043e2bad1534597c151f3">Specifies the group name of a collection file directory. Group names are separated by spaces.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r5bb6544114e446c49983b6e615e5e078"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__aba7e8edcefac43a69175e2ee2fd072cc">filegroups.<filegroupName>.parentDir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__ae2cfeb26bbae45018c5fe10ea82b8ec8">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a9815719cf3404a5bb79e7adea2cbc3f2">Specifies the parent directory. The value must be an absolute path.</p>
|
|
<p id="mrs_01_0396__p208119248811">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3708ff735c1e4cda8f7d80817ad2768d"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a869abac9343c4887bf0b1d31f9cdfcea">filegroups.<filegroupName>.filePattern</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__ad2fa7cabbd42470a9207f9cbcd115944">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a929285d3c75e4213bd8d1055e4fffd0a">Specifies the relative file path of the file group's parent directory. Directories can be included and regular expressions are supported. It must be used together with <strong id="mrs_01_0396__b50158061425041">parentDir</strong>.</p>
|
|
<p id="mrs_01_0396__p2104430182">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r1245fc762022410ea3e05399bfdf30fa"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p440251165858">positionFile</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a75a7cb945c4e44e89575db75f73b955e">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__acfe29b9b8d4e48d28e4bb05a212b1f62">Specifies the metadata storage path during data transmission.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__ra4c8ebc16e774b859b2fd8ac7e5bc789"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a3528007b997b4a7d87aead7846e6e2fd">headers.<filegroupName>.<headerKey></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__af614e1e22e7b42f785b556ad6331f29c">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a39c7431b12d948b391aafb5fe261f478">Specifies the key-value of an event when data of a group is being collected.</p>
|
|
<p id="mrs_01_0396__p98502021694">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rb89a7fdf6df1407082d59b563f9008c2"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a2d7e3d8a4eb14cab9dc77c030cd61a76">byteOffsetHeader</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a895aaff860a5491dbcf4d37f8bae8f88">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a167bc9b1f64f4b45b2c247f502569d10">Specifies whether each event header should contain the location information about the event in the source file. The location information is saved in the <span class="parmname" id="mrs_01_0396__p6e660086a567481f9cb7660f10cf19ad"><b>byteoffset</b></span> variable.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r8e7344f01f364d508df3c09aa1f96a88"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a36ec1439b51e4a13a0b799e1547b5ba6">skipToEnd</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__acdeab4d5592a4e4f94768847ff856370">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a26e143bf898646c1a0743a2ea1bc4560">Specifies whether Flume can locate the latest location of a file and read the latest data after restart.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rb2a099c6ab4d45bf8a72b17beefc24c7"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a1c3bcf31c1944aa3a6b6ac00359a939e">idleTimeout</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__abd724a40f53146d8b3866f7a17721e89">120000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a049fa63ad3324427b7868908883f243e">Specifies the idle duration during file reading, expressed in milliseconds. If the file data is not changed in this idle period, the source closes the file. If data is written into this file after it is closed, the source opens the file and reads data.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r4282ca0909a140f2ac9bb24f8420ee96"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a25fb3e740649401ba201c65065735f40">writePosInterval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a4dd2984b49e2423494329fca05cbd68f">3000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a2a77a90f8cd241e788774a5dfa16cfe4">Specifies the interval for writing metadata to a file, expressed in milliseconds.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r6336d2c0900a4b8cb6da9b41fd4b74fb"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a46b9d3c0153346a2a43328599d8861ad">batchSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a41d48491883f4335bf43e39eacfbc85d">1000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a2c62944cdb9a48ce8bf3565941e2155f">Specifies the number of events written to the channel in batches.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rf9fdd68ae4f34cf08af5a1d2a62964af"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.4.3.2.4.1.1 "><p id="mrs_01_0396__aa4eaa806840f49ac89974f2464f006fb"><span id="mrs_01_0396__p3649447c030b418b8caba0e974cc5bd6">monTime</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.4.2.4.3.2.4.1.2 "><p id="mrs_01_0396__af2d5eb39ed9b4be7833262d716b9ea7d"><span id="mrs_01_0396__pb9b1a91154a94322b95dea7242ef6547">0 (Disabled)</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.05%" headers="mcps1.3.4.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a0dd1c41079c54f46bf73836e4434242d">Specifies the thread monitoring threshold. When the update time exceeds the threshold, the source is restarted. Unit: second</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_0396__l09badd53b1364a2ba2641e40f9ea6fd2"><strong id="mrs_01_0396__a9b7ae6cf50ea4f269e1930c5495a2fc5">Http Source</strong><p id="mrs_01_0396__a4a14170d471144c4a0895f459c757a36">An HTTP source receives data from an external HTTP client and sends the data to the configured channels. <a href="#mrs_01_0396__t033eef1276424185b1cfd10a7d4e024f">Table 5</a> lists common configurations.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0396__t033eef1276424185b1cfd10a7d4e024f"></a><a name="t033eef1276424185b1cfd10a7d4e024f"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__t033eef1276424185b1cfd10a7d4e024f" frame="border" border="1" rules="all"><caption><b>Table 5 </b>Common configurations of an HTTP source</caption><thead align="left"><tr id="mrs_01_0396__rbfd2b8ae09464420a872393daca8a1a0"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.4.2.5.3.2.4.1.1"><p id="mrs_01_0396__a5899ba527015427cb3a770191efde138"><strong id="mrs_01_0396__ae987d4b4ae2c424ea02473fd3fcebd04">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.4.2.5.3.2.4.1.2"><p id="mrs_01_0396__ae5d4f302edd4472c8beeb9e228f340ea"><strong id="mrs_01_0396__ade302d53a6fd44fc88788382826e53f7">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.4.2.5.3.2.4.1.3"><p id="mrs_01_0396__a25f09b3888ea4810a6423b31462431c9"><strong id="mrs_01_0396__ac1c0a90961eb4e87bc9e47815c6d34c4">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__rb2f4c2fcd26e436c893cd9fd34ff4eda"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.5.3.2.4.1.1 "><p id="mrs_01_0396__abca43b42e44b4e3a86fa5cbd7b373603">channels</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.5.3.2.4.1.2 "><p id="mrs_01_0396__ac7b945b508b04b2688649f97b3ee602d">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.5.3.2.4.1.3 "><p id="mrs_01_0396__a256de339f97440f09419a69a4deed709">Specifies the channel connected to the source. Multiple channels can be configured. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__re7931ec8d3f74615b961fcc49a1849b7"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.5.3.2.4.1.1 "><p id="mrs_01_0396__a6825dcc16fba4d34a22566eefe8592db">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.5.3.2.4.1.2 "><p id="mrs_01_0396__ac53c48f837b549beaf1633ebe35e48e6">http</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.5.3.2.4.1.3 "><p id="mrs_01_0396__a150440c2de204164a9911d02404f72e0">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__parmvalue3668670989133"><b>http</b></span>. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r6bbb3e1c166d4333876a6fa6d655d5fc"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.5.3.2.4.1.1 "><p id="mrs_01_0396__a05bf50afd2ae4872a246b4c331f92f03">bind</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.5.3.2.4.1.2 "><p id="mrs_01_0396__a00a0938b2a434d878974ee2281a1ae15">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.5.3.2.4.1.3 "><p id="mrs_01_0396__a99071feaf6ed4158a356dcb686508b58">Specifies the name or IP address of the bound host.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rf1ca49dad4084aeda8eb34c740431ed0"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.5.3.2.4.1.1 "><p id="mrs_01_0396__a3ebfb6d139934cc7be875ff24f65140e">port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.5.3.2.4.1.2 "><p id="mrs_01_0396__a268563dfc02e4effb67f2aaec1e689dc">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.5.3.2.4.1.3 "><p id="mrs_01_0396__a6896178fab0840a08ab04c3408110abf">Specifies the bound port.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rb1617a8ca5f247f59b67b0d0d9fa4272"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.5.3.2.4.1.1 "><p id="mrs_01_0396__a388a24688e2c467e98a5ea8cb9141e16">handler</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.5.3.2.4.1.2 "><p id="mrs_01_0396__aba1158ce78144ee6bdc57039e49b7749">org.apache.flume.source.http.JSONHandler</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.5.3.2.4.1.3 "><p id="mrs_01_0396__af9131cd16db84c8abe357c24527bca90">Specifies the message parsing method of an HTTP request. The following methods are supported:</p>
|
|
<ul id="mrs_01_0396__u00adfbb8387b484683bb7cf15f06a64f"><li id="mrs_01_0396__l9b9dd93c583949109510686c72d0579b"><span class="parmvalue" id="mrs_01_0396__pfa8ba599d4034d19bc5611c8000970ec"><b>org.apache.flume.source.http.JSONHandler</b></span>: JSON</li><li id="mrs_01_0396__l7fd04f4984a04c33b7040a5e59cdb022"><span class="parmvalue" id="mrs_01_0396__pd85fe29a10e84e52b175b075317ce4b0"><b>org.apache.flume.sink.solr.morphline.BlobHandler</b></span>: BLOB</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r40b879a3564e40bcb4aad3b457e743bf"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.5.3.2.4.1.1 "><p id="mrs_01_0396__a4d3f859e11ae413b817aa7f194551edc">handler.*</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.5.3.2.4.1.2 "><p id="mrs_01_0396__a0ab182b5779a479c98bf09fb61ea1d98">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.5.3.2.4.1.3 "><p id="mrs_01_0396__a30cf681c5ca34dc9bb7d3edea49b364f">Specifies handler parameters.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3541737adb98421a97465575f809592c"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.5.3.2.4.1.1 "><p id="mrs_01_0396__a8cb930a1fa1f473b99c8687f7e0c74e1">enableSSL</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.5.3.2.4.1.2 "><p id="mrs_01_0396__a4319b837af1443fa95ff221328bcd199">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.5.3.2.4.1.3 "><p id="mrs_01_0396__a8f54807f6c5f4f5796d4e87afd88956c">Specifies whether SSL is enabled in HTTP.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__reaa15e33b1844fad9ecec0fa12afed3e"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.5.3.2.4.1.1 "><p id="mrs_01_0396__aa6fa25e5dceb473c9779b865e2c76853">keystore</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.5.3.2.4.1.2 "><p id="mrs_01_0396__a89e4f69ffeda4c1c80aa9c6453ece143">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.5.3.2.4.1.3 "><p id="mrs_01_0396__a8dd16915e1ef4b17b66b8678c576f99b">Specifies the keystore path set after SSL is enabled in HTTP.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r732edf88e3e047579c702dca2282b2b9"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.5.3.2.4.1.1 "><p id="mrs_01_0396__ad9914f5144104b09aa173bd1df748c71">keystorePassword</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.5.3.2.4.1.2 "><p id="mrs_01_0396__ab299af5bb0314eedac956c309555a5ab">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.5.3.2.4.1.3 "><p id="mrs_01_0396__a5b0ab9d977af4d798534e7a1254b59cc">Specifies the keystore password set after SSL is enabled in HTTP.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_0396__s252653025d9a4363a2d6f53fa9c1c20d"><h4 class="sectiontitle">Common Channel Configurations</h4><ul id="mrs_01_0396__uee652512defc4096add16324f43179c6"><li id="mrs_01_0396__lad82e502aeae4a4798a09e7b033e3db3"><strong id="mrs_01_0396__en-us_topic_0066459131_b220268241758">Memory Channel</strong><p id="mrs_01_0396__en-us_topic_0066459131_p640236961758">A memory channel uses memory as the cache. Events are stored in memory queues. <a href="#mrs_01_0396__tc1421df5bc6c415ca490e671ea935f85">Table 6</a> lists common configurations.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0396__tc1421df5bc6c415ca490e671ea935f85"></a><a name="tc1421df5bc6c415ca490e671ea935f85"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__tc1421df5bc6c415ca490e671ea935f85" frame="border" border="1" rules="all"><caption><b>Table 6 </b>Common configurations of a memory channel</caption><thead align="left"><tr id="mrs_01_0396__ra791bc5ce68641fc9257a2a04e53342a"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.5.2.1.3.2.4.1.1"><p id="mrs_01_0396__en-us_topic_0066459131_p194642081758"><strong id="mrs_01_0396__en-us_topic_0066459131_b245323111758">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.2.1.3.2.4.1.2"><p id="mrs_01_0396__en-us_topic_0066459131_p294379441758"><strong id="mrs_01_0396__en-us_topic_0066459131_b330970441758">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.5.2.1.3.2.4.1.3"><p id="mrs_01_0396__en-us_topic_0066459131_p525339371758"><strong id="mrs_01_0396__en-us_topic_0066459131_b356632651758">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__r2197ee0b8023489c95c9de610c23887e"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.1.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p40372161758">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.1.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p585790501758">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.1.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p472825921758">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__pf2d9c3344bf443129fcd4fc2ab0ad90f"><b>memory</b></span>. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__ree76cea1cb4b4bc0a2099ad384d0d855"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.1.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p421630201758">capacity</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.1.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p597614891758">10000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.1.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p88424631758">Specifies the maximum number of events cached in a channel.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rce4169d8c0ee437190c23511cc60dbe5"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.1.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p37047091758">transactionCapacity</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.1.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p316460311758">1000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.1.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p131917091758">Specifies the maximum number of events accessed each time.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r06153a55cf2942c48cc5e5b7d31cee03"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.1.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p201886971758">channelfullcount</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.1.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p246717781758">10</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.1.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p522569711758">Specifies the channel full count. When the count reaches the threshold, an alarm is reported.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_0396__l11a4f63a76cb4033b39eb5bb7390cd9b"><strong id="mrs_01_0396__en-us_topic_0066459131_b652127661758">File Channel</strong><p id="mrs_01_0396__en-us_topic_0066459131_p500439871758">A file channel uses local disks as the cache. Events are stored in the folder specified by <span class="parmname" id="mrs_01_0396__p5206882b97cb45a8b8e19de96c60f3b2"><b>dataDirs</b></span>. <a href="#mrs_01_0396__td180d6190e86420d8779010b90877938">Table 7</a> lists common configurations.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0396__td180d6190e86420d8779010b90877938"></a><a name="td180d6190e86420d8779010b90877938"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__td180d6190e86420d8779010b90877938" frame="border" border="1" rules="all"><caption><b>Table 7 </b>Common configurations of a file channel</caption><thead align="left"><tr id="mrs_01_0396__rba03767d0a94493f8d32691885da14cb"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.5.2.2.3.2.4.1.1"><p id="mrs_01_0396__en-us_topic_0066459131_p498158311758"><strong id="mrs_01_0396__en-us_topic_0066459131_b428177941758">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.2.2.3.2.4.1.2"><p id="mrs_01_0396__en-us_topic_0066459131_p98456381758"><strong id="mrs_01_0396__en-us_topic_0066459131_b85505001758">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.5.2.2.3.2.4.1.3"><p id="mrs_01_0396__en-us_topic_0066459131_p639308151758"><strong id="mrs_01_0396__en-us_topic_0066459131_b592992071758">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__re81019e665f54a70ad25c0f0772a6d38"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.2.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p196810201758">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p506588201758">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p97237401758">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__p0a5d7d000b214e89a67d1228896dd2da"><b>file</b></span>. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r9f939ffdd17540b48e5d05566b7bc5b9"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.2.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p421758891758">checkpointDir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p608038491758">${BIGDATA_DATA_HOME}/flume/checkpoint</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p389699911758">Specifies the checkpoint storage directory.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3a88c82306eb402ca801c928f821e725"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.2.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p220743101758">dataDirs</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p431886571758">${BIGDATA_DATA_HOME}/flume/data</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p271583691758">Specifies the data cache directory. Multiple directories can be configured to improve performance. The directories are separated by commas (,).</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__re8eb79d0858f4b569bd24cf68fb18a66"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.2.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p13367481758">maxFileSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p411677971758">2146435071</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p462572661758">Specifies the maximum size of a single cache file, expressed in bytes.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r7807c7528a024500ad553bb5c5494066"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.2.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p328977111758">minimumRequiredSpace</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p474689261758">524288000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p197778221758">Specifies the minimum idle space in the cache, expressed in bytes.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r7981be9bee8b46678107cadebb35288c"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.2.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p567358561758">capacity</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p322015841758">1000000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p581914841758">Specifies the maximum number of events cached in a channel.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rdc01741f99234fbcb46c299303c50c00"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.2.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p87903811758">transactionCapacity</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p409322801758">10000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p271803961758">Specifies the maximum number of events accessed each time.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r2769cd7d5774439284efc3622f36a286"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.2.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p173938391758">channelfullcount</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p667237271758">10</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p359127871758">Specifies the channel full count. When the count reaches the threshold, an alarm is reported.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_0396__l0a403f840e63476a972a45b52d1c8460"><strong id="mrs_01_0396__en-us_topic_0066459131_b379825501758">Kafka Channel</strong><div class="p" id="mrs_01_0396__en-us_topic_0066459131_p337651051758">A Kafka channel uses a Kafka cluster as the cache. Kafka provides high availability and multiple copies to prevent data from being immediately consumed by sinks when Flume or Kafka Broker crashes. <a href="#mrs_01_0396__ta58e4ea5e98446418e498b81cf0c75b7">Table 10 Common configurations of a Kafka channel</a> lists common configurations.
|
|
<div class="tablenoborder"><a name="mrs_01_0396__ta58e4ea5e98446418e498b81cf0c75b7"></a><a name="ta58e4ea5e98446418e498b81cf0c75b7"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__ta58e4ea5e98446418e498b81cf0c75b7" frame="border" border="1" rules="all"><caption><b>Table 8 </b>Common configurations of a Kafka channel</caption><thead align="left"><tr id="mrs_01_0396__rff4c69fc6ac048a3b6106d552dab78a4"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.5.2.3.2.2.2.4.1.1"><p id="mrs_01_0396__en-us_topic_0066459131_p105045311758"><strong id="mrs_01_0396__a61555af3d5db41c3a25e5f9712d831b2">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.2.3.2.2.2.4.1.2"><p id="mrs_01_0396__en-us_topic_0066459131_p73927981758"><strong id="mrs_01_0396__a8433a23373cd4422a387a594213956f7">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.5.2.3.2.2.2.4.1.3"><p id="mrs_01_0396__en-us_topic_0066459131_p206406681758"><strong id="mrs_01_0396__acee38f6d383743328a29455c204b7022">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__rb209da814106468ba942b148c701ce13"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.3.2.2.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p648505311758">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.3.2.2.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p184016431758">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.3.2.2.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p601339601758">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__pa302307c05d1434094a68f30595f6e8d"><b>org.apache.flume.channel.kafka.KafkaChannel</b></span>.</p>
|
|
<p id="mrs_01_0396__p2906432172213">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r93c1fc7e14974e9eafe6c2dafc088962"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.3.2.2.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p155690131758">kafka.bootstrap.servers</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.3.2.2.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p531305751758">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.3.2.2.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p86093461758">Specifies the list of Brokers in the Kafka cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r70c08faa517445f7ab40169a365a998f"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.3.2.2.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p350890541758">kafka.topic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.3.2.2.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p236411081758">flume-channel</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.3.2.2.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p358816321758">Specifies the Kafka topic used by the channel to cache data.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rdc7cec1d498d49b6a7ab575ced85c934"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.3.2.2.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p523617011758">kafka.consumer.group.id</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.3.2.2.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p134394111758">flume</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.3.2.2.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p148504811758">Specifies the Kafka consumer group ID.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r0f6e374ca98947e69ece2e7fe7db7e71"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.3.2.2.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p214736031758">parseAsFlumeEvent</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.3.2.2.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p616402731758">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.3.2.2.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p268062201758">Specifies whether data is parsed into Flume events.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r536ea53053c2460096ef10b81b75a5c8"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.3.2.2.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p130550941758">migrateZookeeperOffsets</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.3.2.2.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p508296631758">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.3.2.2.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p235620001758">Specifies whether to search for offsets in ZooKeeper and submit them to Kafka when there is no offset in Kafka.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r92a9c1a1288d4ef6a2c194f130ece84b"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.3.2.2.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p639379881758">kafka.consumer.auto.offset.reset</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.3.2.2.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p115945781758">latest</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.3.2.2.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p667456361758">Consumes data from the specified location when there is no offset.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r6a4c9130ed67427e9ed6db5f479a821f"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.3.2.2.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p36428141758">kafka.producer.security.protocol</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.3.2.2.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p266325551758">SASL_PLAINTEXT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.3.2.2.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p97533361758">Specifies the Kafka producer security protocol.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r7a34758ad20540b9964d6b1bb2fe4e86"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.5.2.3.2.2.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p637514191758">kafka.consumer.security.protocol</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.3.2.2.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p635912801758">SASL_PLAINTEXT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.2.3.2.2.2.4.1.3 "><p id="mrs_01_0396__a74697ecb1be742968c394b9bac8c950f">Specifies the Kafka consumer security protocol.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_0396__s7d014458bd014b1891c713d57d369f2f"><h4 class="sectiontitle">Common Sink Configurations</h4><ul id="mrs_01_0396__ue5e55088269f40ddb1d080809c4591f3"><li id="mrs_01_0396__lf88331d23ef24af8aab6842019d7437e"><strong id="mrs_01_0396__aac60ce2e77c542d1be36dc574aa52cb7">HDFS Sink</strong><p id="mrs_01_0396__a29f2bc7a07614cff8e2d76562df3b1c7">An HDFS sink writes data into HDFS. <a href="#mrs_01_0396__t3f4509459f734167afdd0cb20857d2ef">Table 9</a> lists common configurations.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0396__t3f4509459f734167afdd0cb20857d2ef"></a><a name="t3f4509459f734167afdd0cb20857d2ef"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__t3f4509459f734167afdd0cb20857d2ef" frame="border" border="1" rules="all"><caption><b>Table 9 </b>Common configurations of an HDFS sink</caption><thead align="left"><tr id="mrs_01_0396__r428a381b672745aab7c2ea64ea8babd5"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.6.2.1.3.2.4.1.1"><p id="mrs_01_0396__a8d2bca0510a541b3a5ccb21798bff2b6"><strong id="mrs_01_0396__a150cb57dda5f4e1d8009e004848c318a">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.6.2.1.3.2.4.1.2"><p id="mrs_01_0396__acc7f6e47f19c44d3a7b7419ff82c8ea9"><strong id="mrs_01_0396__a9648039eec824b63bb3c04fba6030c3f">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.6.2.1.3.2.4.1.3"><p id="mrs_01_0396__en-us_topic_0066459131_p328508017533"><strong id="mrs_01_0396__a88dae19dc91a4b2c9793e3500716d779">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__r6f4b81287cdf4a60978dd11293789afd"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a11db38e8f11e44aeb05a01182c40c055">channel</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a582103f12263427da0905257bd2697dc"><strong id="mrs_01_0396__ac01029686b15427aa8b605de731bfacf">-</strong></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p619013817533">Specifies the channel connected to the sink. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r0b8083b767f14f068faf94198eecc389"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__af3e90a3bf11d4335a8cb9f2033984bd6">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a4a73455e81734c9190ebbbce1decd067">hdfs</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a265167e9f50f4a399f5dc9926354bb77">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__p9018bb8360ee4dc9b82baa044d34ec28"><b>hdfs</b></span>. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rb6cfe4a1765e498cac531919cc241103"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a6a38b53182834d65a3ab52cc50acbaa0">monTime</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p843901117533">0 (Disabled)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__aa4a8646530a24ba69c552822844aebd0">Specifies the thread monitoring threshold. When the update time exceeds the threshold, the sink is restarted. Unit: second</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r1c72034e03d740838cfa8ecf35ae1185"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__ae65ca687db7643e38536252f8a349a61">hdfs.path</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a13d14639b59747c98b003acabf1e32c1">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a00deeca3d3244cf68e9fe7ae968414ce">Specifies the HDFS path.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r07fc2301dfd5446086b9cf6071fd9014"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a1eca519a1fe8438fb2e3409a8d181d6c">hdfs.inUseSuffix</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__ae09cfb564fed4af1800e0a6bf5094316">.tmp</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a7ed839e807c0423996251f3c8e06fac0">Specifies the suffix of the HDFS file to which data is being written.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rab8fc474da824422812d2f37c7d5f408"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__af809d471a241484aa6f1595025978cb6">hdfs.rollInterval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p880104717533">30</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a7f0f6f452a574b95a916b31628c220b6">Specifies the interval for file rolling, expressed in seconds.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__re2b855b21d1247db932da00aa16ff143"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p201758517533">hdfs.rollSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__ab0daa8c0cdb84cf29a0e5aa6ed504d0f">1024</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a0d21426e67d84437b35f2f1f02d1de14">Specifies the size for file rolling, expressed in bytes.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3daec9017b184ece8994115463f9e739"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a7df4d1dd875b4afdb965e00b3296a2bb">hdfs.rollCount</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__ab5859a61079449b493d5039f5b6a4737">10</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a40b74d2a8eb04d9ea41ae99b097fb55a">Specifies the number of events for file rolling.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r115413f2353c4106a84eeb3811f9be60"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__acd88f8cbcc8f48a88ffaf0b39c3fe25e">hdfs.idleTimeout</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p981461617533">0</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a6394e611a30143499f441fa5502d4936">Specifies the timeout interval for closing idle files automatically, expressed in seconds.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r68b30643028b4c9293ca8e33fe69295b"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a77d305269b3948c598889cfe46213f47">hdfs.batchSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a2ee9988a5566422ba6a20544038c8f1e">1000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a7d6ebd7da6bb4bcc8ada3b290fb1d121">Specifies the number of events written into HDFS at a time.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r1466b090ba4c46fdb8ca140c7c63cac3"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a382a357d4a034556a78b9262b85d9868">hdfs.kerberosPrincipal</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__ab6938a937d0d42509dd97c46c7ec3f0e">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a3156f686d01a45a890ae6fd7b1cf3f3a">Specifies the Kerberos username for HDFS authentication. This parameter is not required for a cluster in which Kerberos authentication is disabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r2bf541a598f54a6abaacb9257ec67321"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a36c43bfaf9b44106a668ef172ca5c140">hdfs.kerberosKeytab</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__ad4030b99937c4946becab0a05ad41fb9">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__adae96cd9d118455584e07be0d513e3c0">Specifies the Kerberos keytab of HDFS authentication. This parameter is not required for a cluster in which Kerberos authentication is disabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rc715718c37434f8a973ea6dec2274397"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a8f4edbee92dc45d1aa67023b2a25ba60">hdfs.fileCloseByEndEvent</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a001a4facae8740f299d5e7ca1993f8e9">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__ad1be897c0b434c158fbe44a1ff78a590">Specifies whether to close the file when the last event is received.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rbfb26ebc81584714b8fa2f0ecbc15749"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a530f21bfe9f144538c3712205ae8c38d">hdfs.batchCallTimeout</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__ad075b831ade249a39b6615e69aab68bb">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__ac53f40e7d18349eba7240db41055d46a">Specifies the timeout control duration each time events are written into HDFS, expressed in milliseconds.</p>
|
|
<p id="mrs_01_0396__aeae7224c87ec46969831bc8f0c18fed1">If this parameter is not specified, the timeout duration is controlled when each event is written into HDFS. When the value of <span class="parmname" id="mrs_01_0396__p28e8f45199794ea09d1fc4cae28d2474"><b>hdfs.batchSize</b></span> is greater than 0, configure this parameter to improve the performance of writing data into HDFS.</p>
|
|
<div class="note" id="mrs_01_0396__nd9c23ee414754ff580abc33dd73a4bf0"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_0396__ac5be970c6595411393f32ecad2a3b6b6">The value of <span class="parmname" id="mrs_01_0396__parmname23250825610294"><b>hdfs.batchCallTimeout</b></span> depends on <span class="parmname" id="mrs_01_0396__parmname1286027129102915"><b>hdfs.batchSize</b></span>. A greater <span class="parmname" id="mrs_01_0396__parmname673810159102940"><b>hdfs.batchSize</b></span> requires a larger <span class="parmname" id="mrs_01_0396__parmname756367162102956"><b>hdfs.batchCallTimeout</b></span>. If the value of <span class="parmname" id="mrs_01_0396__parmname57894028310323"><b>hdfs.batchCallTimeout</b></span> is too small, writing events to HDFS may fail.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r1b60be2a793c4b6aa8cd56cf37038bab"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.1.3.2.4.1.1 "><p id="mrs_01_0396__a68d545c4ef3c4f71bf3e9995a2080a9f">serializer.appendNewline</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.1.3.2.4.1.2 "><p id="mrs_01_0396__a9c53ede997cb4f8983d341b28673e235">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.1.3.2.4.1.3 "><p id="mrs_01_0396__a824e416390154521b07055764a28279e">Specifies whether to add a line feed character (<strong id="mrs_01_0396__b93472962225041">\n</strong>) after an event is written to HDFS. If a line feed character is added, the data volume counters used by the line feed character will not be calculated by HDFS sinks.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_0396__lf84c68de22de4be6b82e42f22bc05477"><strong id="mrs_01_0396__en-us_topic_0066459131_b661552117533">Avro Sink</strong><p id="mrs_01_0396__a3e337b94515a418eb5f11215da0a0f13">An Avro sink converts events into Avro events and sends them to the monitoring ports of the hosts. <a href="#mrs_01_0396__tcf9863ee677d41a6882b71987541fa33">Table 10</a> lists common configurations.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0396__tcf9863ee677d41a6882b71987541fa33"></a><a name="tcf9863ee677d41a6882b71987541fa33"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__tcf9863ee677d41a6882b71987541fa33" frame="border" border="1" rules="all"><caption><b>Table 10 </b>Common configurations of an Avro sink</caption><thead align="left"><tr id="mrs_01_0396__re19fc9b53efa43969c188df75afc7df6"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.6.2.2.3.2.4.1.1"><p id="mrs_01_0396__en-us_topic_0066459131_p660555417533"><strong id="mrs_01_0396__a4b3ef20be65c4f7ebf1fd6714a5f6dea">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.6.2.2.3.2.4.1.2"><p id="mrs_01_0396__acfed75dc45a742b08b9ce5abbc177936"><strong id="mrs_01_0396__a13c9e1819de04f8abda5c649c4c55f67">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.6.2.2.3.2.4.1.3"><p id="mrs_01_0396__a60dbbde284224956a74d97d57add9430"><strong id="mrs_01_0396__a830ef800b3a04beead4b9935f305b027">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__r6e296b1504eb45fe827576692235a400"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a59bd61b8c29c414280fe5a6e1546d55d">channel</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__aeb897062962f4311905bb80da552798d"><strong id="mrs_01_0396__a888ca45b6ab2472bacc8d77ad224f7a0">-</strong></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__ab0acbcd6fd62453a95e23b8d68c3e67c">Specifies the channel connected to the sink. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r789d9fd3aa0f46d4ba39710ffc024aea"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a05cb3e099e57474faf4ebe8fed4255ef">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__aa8e7c3480f394f92a6d5a7072bca25c6">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__af8a448089f394d3b9215cd3fcf8a9b2c">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__p260ea6b33ad54455840d7ded67b33782"><b>avro</b></span>. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r1fb98adb58c14054830c2a3764b53548"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__ae2fe815312de4ae6bcb2a528093a4729">hostname</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p35597717533">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a06c85d581bff44fbaca045f059d8edf2">Specifies the name or IP address of the bound host.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r0c1a3a0f05004faa8ecee29c84a5a77e"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__aeed48f0613464ed09459021a7c072dc1">port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p924698617533">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__aba450cb5a8ba470eab62752c9459f69d">Specifies the monitoring port.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r62e564bdeea247f58174047cb98b0ffd"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a4db78151f4f04c2489c7f47d5c895df3">batch-size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a7bc09ed79a584ecca7df52513cea12a5">1000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__af45550ce5d9e4f1fbba4964df3d5a69b">Specifies the number of events sent in a batch.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rede163131eec4250ae961c5d0b0aebce"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__ab0edd0f1bbdd412ea38b096ea3e77cb2">ssl</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p334117217533">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p219951417533">Specifies whether to use SSL encryption.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r6439ad5ed5f944aa95641e90a05fe1f0"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a641dabd3c6b44853992c63b1cc33d918">truststore-type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__ad409d4fde0414f5391e1f4716c559649">JKS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__acede364f20694d38ac0810a4f422f219">Specifies the Java trust store type.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r1ba554f453694aa9973e9053c824cdd7"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__a3c4857121cf6423da51ead2a3c093c88">truststore</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a0a3be538d86b4833baee1ce331812c85">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p196860717533">Specifies the Java trust store file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r5bf8d80ddc454c5d80a4dd0c70c30a81"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__af1895eab45904dcc9787cb52a5eaceaf">truststore-password</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a38db08ecc8654854b9a040aa2ca0359f">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__adf82dda138354588b5ce521a0b244601">Specifies the Java trust store password.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r1b03bc431e7f4c7e867ffa8c4a1ada50"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__accc23caf3bc543728bec1e35ee865314">keystore-type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p476166117533">JKS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__a9caa5e4f54d04b8cbb700486a839131f">Specifies the key storage type.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3487c784e9784e81a44a951f169c6320"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__aba9d1f2a3fbc4b2298a4fe754deffa03">keystore</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p831478117533">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p240867317533">Specifies the key storage file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r57c2243216a548d89e171586a613e83c"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.2.3.2.4.1.1 "><p id="mrs_01_0396__aee3b5126dddc4869a68d65a8f21e5b40">keystore-password</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.2.3.2.4.1.2 "><p id="mrs_01_0396__a4c21117bae7e46bba3e23c3848f779f7">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.2.3.2.4.1.3 "><p id="mrs_01_0396__aca9b7d4fabc346a58257274b94875590">Specifies the key storage password.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_0396__l15ac95de9d434e76bd01227b2f59f6e3"><strong id="mrs_01_0396__a72fe1e54882547a19a2b081b8cfa08a5">HBase Sink</strong><p id="mrs_01_0396__ae19cf7551ad04a5dae08d41e80228664">An HBase sink writes data into HBase. <a href="#mrs_01_0396__tf429beac69444e93a744abfe1d0fb744">Table 11</a> lists common configurations.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0396__tf429beac69444e93a744abfe1d0fb744"></a><a name="tf429beac69444e93a744abfe1d0fb744"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__tf429beac69444e93a744abfe1d0fb744" frame="border" border="1" rules="all"><caption><b>Table 11 </b>Common configurations of an HBase sink</caption><thead align="left"><tr id="mrs_01_0396__ra970fcbddbeb4ec5bc23c81b686b33da"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.6.2.3.3.2.4.1.1"><p id="mrs_01_0396__a769b4777d8634bb0869894c50860b3d4"><strong id="mrs_01_0396__a6d37fa300354467b83dbea427bfd65eb">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.6.2.3.3.2.4.1.2"><p id="mrs_01_0396__a84061bc2d39a4d5ab068eb59bd84f457"><strong id="mrs_01_0396__a1bff2d21e37b485a9c5716f48441601f">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.6.2.3.3.2.4.1.3"><p id="mrs_01_0396__en-us_topic_0066459131_p763900917533"><strong id="mrs_01_0396__en-us_topic_0066459131_b84877817533">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__ra2df0f21842740cf80551de7e30423f0"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a1c1fda0a54de4529ba72ad4883ed02f1">channel</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.3.3.2.4.1.2 "><p id="mrs_01_0396__a970e1ea742344ab09ec7bcd370f6715a"><strong id="mrs_01_0396__a4858c02556d6474c86bcad898d6d2778">-</strong></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.3.3.2.4.1.3 "><p id="mrs_01_0396__abb556afd58584cf6b04a9186890ade9e">Specifies the channel connected to the sink. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rdcf7ac3bb8484b9fa7c2e51c12a66737"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.3.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p568160617533">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.3.3.2.4.1.2 "><p id="mrs_01_0396__a61fcc3ab01ea477c9cd0fd2e00ce4602">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a51dfd4d22d464108b62552aed4341f39">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__p233eebfa8ca548b8be56b4345498a1b8"><b>hbase</b></span>. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r86767b1614fa4a8187eb36e6e7bdb4c6"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.3.3.2.4.1.1 "><p id="mrs_01_0396__a86f24d691409443f9593c7b6d16ac3d3">table</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.3.3.2.4.1.2 "><p id="mrs_01_0396__a18dbe4ead6e6432089a4d42e8da37fb6">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.3.3.2.4.1.3 "><p id="mrs_01_0396__aa1476d1591734f379748110926c61e82">Specifies the HBase table name.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rf50e908f3e0f4fa5aba6530a14d09b6b"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.3.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p99517817533">monTime</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.3.3.2.4.1.2 "><p id="mrs_01_0396__aa416818b6376422487336b4a22fdbdda">0 (Disabled)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a5e0c05f647bb454f92d4890169cdd727">Specifies the thread monitoring threshold. When the update time exceeds the threshold, the sink is restarted. Unit: second</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r138bdb7aab1b4f67ab5da7a33cb9d0c1"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.3.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p802703117533">columnFamily</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.3.3.2.4.1.2 "><p id="mrs_01_0396__a369296d6439c417fb6e6e26773e2285f">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a734fb6dd6a5b4f40abf70ad9b58bf5d2">Specifies the HBase column family.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r07ba877866b345249138be459c2308e3"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.3.3.2.4.1.1 "><p id="mrs_01_0396__add882f1e15864683a4ef46fee43299d1">batchSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.3.3.2.4.1.2 "><p id="mrs_01_0396__abcf4f8ae555c48659f0127b1328acf06">1000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.3.3.2.4.1.3 "><p id="mrs_01_0396__a4ad9de126988457c9080ca51350efdfc">Specifies the number of events written into HBase at a time.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3305a24ac40f496bbc0bfa12743fe915"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.3.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p980622217533">kerberosPrincipal</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.3.3.2.4.1.2 "><p id="mrs_01_0396__a6fd92d2fc9ef4ee9a3bce453273a7cd2">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.3.3.2.4.1.3 "><p id="mrs_01_0396__ac1945205e4384c40b327e0f33d17876b">Specifies the Kerberos username for HBase authentication. This parameter is not required for a cluster in which Kerberos authentication is disabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r32bcca9c56484448ba64170bdeddb062"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.3.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p379596517533">kerberosKeytab</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.3.3.2.4.1.2 "><p id="mrs_01_0396__ab0f6be33ecc744ed90dd6eb4925381e4">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.3.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p794234417533">Specifies the Kerberos keytab of HBase authentication. This parameter is not required for a cluster in which Kerberos authentication is disabled.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_0396__lc086ce274dea42d4af978cad9f242d82"><strong id="mrs_01_0396__en-us_topic_0066459131_b844000217533">Kafka Sink</strong><p id="mrs_01_0396__en-us_topic_0066459131_p885115617533">A Kafka sink writes data into Kafka. <a href="#mrs_01_0396__tf898876f2a2f45629655554005c3f0a8">Table 12</a> lists common configurations.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0396__tf898876f2a2f45629655554005c3f0a8"></a><a name="tf898876f2a2f45629655554005c3f0a8"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0396__tf898876f2a2f45629655554005c3f0a8" frame="border" border="1" rules="all"><caption><b>Table 12 </b>Common configurations of a Kafka sink</caption><thead align="left"><tr id="mrs_01_0396__r83705dd984674f549a81a591a3ab58f3"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.6.2.4.3.2.4.1.1"><p id="mrs_01_0396__en-us_topic_0066459131_p565891217533"><strong id="mrs_01_0396__en-us_topic_0066459131_b808530817533">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.6.2.4.3.2.4.1.2"><p id="mrs_01_0396__ad21340b406434c1889a8fc44e2f65e68"><strong id="mrs_01_0396__a79d80603c4394a3b893f8b989164481f">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.6.2.4.3.2.4.1.3"><p id="mrs_01_0396__a3e797ff13cfd4e29bcef95a39df33046"><strong id="mrs_01_0396__ae59bfb1ae60e4e048c7ee7337076f6eb">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0396__ra277720243374d178ac6c8a574cd5a2b"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.4.3.2.4.1.1 "><p id="mrs_01_0396__ae22f17dd53e44391b123e10a2da0658d">channel</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a1624b267ba71494db8be70b05219f307"><strong id="mrs_01_0396__a11ceb1dbc6c94b14b56b4a85f0c1a2f5">-</strong></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.4.3.2.4.1.3 "><p id="mrs_01_0396__afd83d2ee21994e0880545cccec045f38">Specifies the channel connected to the sink. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r1543a8f29d484108a61adb4411777f0e"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.4.3.2.4.1.1 "><p id="mrs_01_0396__aa744821c495f45c1b8a9bcb810aee740">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a1099972b687544b887c1ab69c3a9d0bc">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a59530aa82dda48e5b990c5e7c660d87e">Specifies the type, which is set to <span class="parmvalue" id="mrs_01_0396__pff8b3b82581945c297957c7d4b60f7fc"><b>org.apache.flume.sink.kafka.KafkaSink</b></span>.</p>
|
|
<p id="mrs_01_0396__p9658234282">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__rb14230c362534e7482c8fabe4ae18e94"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a23a9c697afcd4f3093c9ec9c56afba86">kafka.bootstrap.servers</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.4.3.2.4.1.2 "><p id="mrs_01_0396__aea3856345b5d4248bddb18c5dfa78395">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a12061b8ddfa34d8a92ee14bccf860a16">Specifies the list of Kafka Brokers, which are separated by commas.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r3b9be3b7d10546f6856c1de26b5d1e49"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a0dcb476a77b648a2a8e5bb8f2a9ac98a">monTime</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a8d0ba495bab74d3a84eeb43e0b1b8f9a">0 (Disabled)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.4.3.2.4.1.3 "><p id="mrs_01_0396__ad3314d2a3182424fbef136a62ba5cd54">Specifies the thread monitoring threshold. When the update time exceeds the threshold, the sink is restarted. Unit: second</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r667870d07e6a4c33adf350bb627a0447"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a9ed9d60b2fc148e7886f97cd531b3540">kafka.topic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.4.3.2.4.1.2 "><p id="mrs_01_0396__ac6eeebae5265487f92da32511f6cd4be">default-flume-topic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a8a420ec747cc4b778230cbb76980c371">Specifies the topic where data is written.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r7f33a19dea5c4d239ffddd452edc98cd"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a5767e97d5e564a8a96cce83791eea409">flumeBatchSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a7381dd3e46b14861a7c2fa20874a33b6">1000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.4.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p740582917533">Specifies the number of events written into Kafka at a time.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__ra7f11dfe7e3c4d00ac8f7008da20f790"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.4.3.2.4.1.1 "><p id="mrs_01_0396__ab3ebf881143d4ed391b85e82c6e019d1">kafka.security.protocol</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.4.3.2.4.1.2 "><p id="mrs_01_0396__a0908b24fc0334751b1988352f6e04b85">SASL_PLAINTEXT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.4.3.2.4.1.3 "><p id="mrs_01_0396__aa25e39815a0943e69fe4ae734da16fae">Specifies the security protocol of Kafka. The value must be set to <span class="parmvalue" id="mrs_01_0396__p62252d84cac248309ac8bd2a4fe00ad3"><b>PLAINTEXT</b></span> for clusters in which Kerberos authentication is disabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r68d48aea470f4ac2991c30f53a5436ad"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.4.3.2.4.1.1 "><p id="mrs_01_0396__en-us_topic_0066459131_p309358951508">kafka.kerberos.domain.name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.4.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p227795341508">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.4.3.2.4.1.3 "><p id="mrs_01_0396__en-us_topic_0066459131_p332029861508">Specifies the Kafka domain name. This parameter is mandatory for a security cluster. This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0396__r4a507fb8c4d14ad9b8f4c5b1a6878215"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.2.4.3.2.4.1.1 "><p id="mrs_01_0396__a1e0caf77f9b94372b0038a03fa6f1b89">Other Kafka Producer Properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.2.4.3.2.4.1.2 "><p id="mrs_01_0396__en-us_topic_0066459131_p897233717533">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.2.4.3.2.4.1.3 "><p id="mrs_01_0396__a8bfe25d2b4024f6ab9a97861fd8fdf7a">Specifies other Kafka configurations. This parameter can be set to any production configuration supported by Kafka, and the <span class="parmvalue" id="mrs_01_0396__p02b388c0a37c472da309e8cb3970a748"><b>.kafka</b></span> prefix must be added to the configuration.</p>
|
|
<p id="mrs_01_0396__p2674194282916">This parameter can be set only in the properties.properties file.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0390.html">Using Flume</a></div>
|
|
</div>
|
|
</div>
|
|
|