forked from docs/doc-exports
Su, Xiaomeng
76a5b1ee83
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
168 lines
21 KiB
HTML
168 lines
21 KiB
HTML
<a name="dli_08_0254"></a><a name="dli_08_0254"></a>
|
|
|
|
<h1 class="topictitle1">MRS Kafka Sink Stream</h1>
|
|
<div id="body1574393117318"><div class="section" id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_section1593192619539"><h4 class="sectiontitle">Function</h4><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p17940172645314">DLI exports the output data of the Flink job to Kafka.</p>
|
|
<p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p166117014354">Apache Kafka is a fast, scalable, and fault-tolerant distributed message publishing and subscription system. It delivers high throughput and built-in partitions and provides data replicas and fault tolerance. Apache Kafka is applicable to scenarios of handling massive messages. Kafka clusters are deployed and hosted on MRS that is powered on Apache Kafka.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_section988111409128"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0254__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_ul19567726133916"><li id="dli_08_0254__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_li729542114118">If the Kafka server listens on the port using hostname, you need to add the mapping between the hostname and IP address of the Kafka Broker node to the DLI queue. Contact the Kafka service deployment personnel to obtain the hostname and IP address of the Kafka Broker node. For details about how to add an IP-domain mapping, see <strong id="dli_08_0254__dli_08_0238_b1189481225013">Enhanced Datasource Connections</strong> > <strong id="dli_08_0254__dli_08_0238_b689517126505">Modifying the Host Information</strong> in the <em id="dli_08_0254__dli_08_0238_i1589518121509">Data Lake Insight User Guide</em>.</li><li id="dli_08_0254__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_li2741672136">Kafka is an offline cluster. You need to use the enhanced datasource connection function to connect Flink jobs to Kafka. You can also set security group rules as required.<p id="dli_08_0254__dli_08_0238_p94091618103916"><a name="dli_08_0254__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_li2741672136"></a><a name="dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_li2741672136"></a>For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_08_0254__dli_08_0238_dli_08_0249_en-us_topic_0132966325_filepath13726182511815"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_08_0254__dli_08_0238_dli_08_0249_cite534516e564224b13bad1de4471da3139181527">Data Lake Insight User Guide</cite></i>.</p>
|
|
<p id="dli_08_0254__dli_08_0238_p17953337436">For details about how to configure security group rules, see <span class="filepath" id="dli_08_0254__dli_08_0238_dli_08_0249_filepath78235157113"><b>Security Group</b></span> in the <em id="dli_08_0254__dli_08_0238_dli_08_0249_i082519153111">Virtual Private Cloud User Guide</em>.</p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_section078317474405"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0254__screen9610204914256"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">stream_id</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"kafka"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_bootstrap_servers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_topic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"json"</span>
|
|
<span class="p">)</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
<p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p89673266530"></p>
|
|
</div>
|
|
<div class="section" id="dli_08_0254__section1767810414417"><h4 class="sectiontitle">Keywords</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_table2972926145319" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_row29913265537"><th align="left" class="cellrowborder" valign="top" width="20.82%" id="mcps1.3.4.2.2.4.1.1"><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p17998132611534">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="8.44%" id="mcps1.3.4.2.2.4.1.2"><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p12310279537">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="70.74000000000001%" id="mcps1.3.4.2.2.4.1.3"><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p10982717536">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_row816152795316"><td class="cellrowborder" valign="top" width="20.82%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p202362713532">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.44%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p6281227165310">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.74000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p143216277530">Output channel type. <strong id="dli_08_0254__b84235270692937">kafka</strong> indicates that data is exported to Kafka.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_row0343279539"><td class="cellrowborder" valign="top" width="20.82%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p13427275537">kafka_bootstrap_servers</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.44%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p34712277532">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.74000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p20521827195312">Port that connects DLI to Kafka. Use enhanced datasource connections to connect DLI queues with Kafka clusters.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_row955727195316"><td class="cellrowborder" valign="top" width="20.82%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p156112719538">kafka_topic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.44%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p1654277535">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.74000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p10706276532">Kafka topic into which DLI writes data.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_row871152765314"><td class="cellrowborder" valign="top" width="20.82%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p10960151965212">encode</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.44%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p68111276531">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.74000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p128552710539">Encoding format. Currently, <span class="parmvalue" id="dli_08_0254__parmvalue1235384611614"><b>json</b></span> and <span class="parmvalue" id="dli_08_0254__parmvalue25108506164"><b>user_defined</b></span> are supported.</p>
|
|
<p id="dli_08_0254__p13963848144415"><span class="parmname" id="dli_08_0254__parmname1845215320197"><b>encode_class_name</b></span> and <span class="parmname" id="dli_08_0254__parmname7457133111912"><b>encode_class_parameter</b></span> must be specified if this parameter is set to <span class="parmvalue" id="dli_08_0254__parmvalue14571238194"><b>user_defined</b></span>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0254__row92374013456"><td class="cellrowborder" valign="top" width="20.82%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0254__en-us_topic_0111562554_p135020211578">encode_class_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.44%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0254__en-us_topic_0111562554_p19502102115716">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.74000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0254__en-us_topic_0111562554_p205020213578">If <strong id="dli_08_0254__b230719248194">encode</strong> is set to <strong id="dli_08_0254__b1231212248193">user_defined</strong>, you need to set this parameter to the name of the user-defined decoding class (including the complete package path). The class must inherit the <strong id="dli_08_0254__b173121924101913">DeserializationSchema</strong> class.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0254__row1108154512456"><td class="cellrowborder" valign="top" width="20.82%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0254__en-us_topic_0111562554_p550214275712">encode_class_parameter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.44%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0254__en-us_topic_0111562554_p195031628572">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.74000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0254__en-us_topic_0111562554_p25031129570">If <strong id="dli_08_0254__b18351428201913">encode</strong> is set to <strong id="dli_08_0254__b18840132814191">user_defined</strong>, you can set this parameter to specify the input parameter of the user-defined decoding class. Only one parameter of the string type is supported.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0254__row2430123213207"><td class="cellrowborder" valign="top" width="20.82%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0254__p1814431414555">krb_auth</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.44%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0254__p131442148552">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.74000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0254__p198893535164">Authentication name for creating a datasource connection authentication. This parameter is mandatory when Kerberos authentication is enabled. If Kerberos authentication is not enabled for the created MRS cluster, ensure that the <span class="filepath" id="dli_08_0254__filepath872010341054"><b>/etc/hosts</b></span> information of the master node in the MRS cluster is added to the host file of the DLI queue.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0254__en-us_topic_0113887277_row1385875510369"><td class="cellrowborder" valign="top" width="20.82%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0254__en-us_topic_0113887277_p168591655103618">kafka_properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.44%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0254__en-us_topic_0113887277_p10859145518369">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.74000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0254__en-us_topic_0113887277_p16859105516368">This parameter is used to configure the native attributes of Kafka. The format is <strong id="dli_08_0254__b5496442183311">key1=value1;key2=value2</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0254__row7504181310491"><td class="cellrowborder" valign="top" width="20.82%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0254__p1564718693412">kafka_certificate_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.44%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0254__p2648766342">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.74000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0254__p8648761341">Specifies the name of the datasource authentication information. This parameter is valid only when the datasource authentication type is set to <strong id="dli_08_0254__b1094411182579">Kafka_SSL</strong>.</p>
|
|
<div class="note" id="dli_08_0254__note2261630114619"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0254__ul137206345480"><li id="dli_08_0254__li2720123464812">If this parameter is specified, the service loads only the specified file and password under the authentication. The system automatically sets this parameter to <strong id="dli_08_0254__b956424419569">kafka_properties</strong>.</li><li id="dli_08_0254__li18720143413484">Other configuration information required for Kafka SSL authentication needs to be manually configured in the <strong id="dli_08_0254__b14737155418566">kafka_properties</strong> attribute.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0254__section93411348194111"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0254__p13161125434113">None</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_section069818538408"><h4 class="sectiontitle">Example</h4><p id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_p4119112712532">Output data to Kafka.</p>
|
|
<ul id="dli_08_0254__ul76141922192815"><li id="dli_08_0254__li3614172272817">Example 1:<div class="codecoloring" codetype="Sql" id="dli_08_0254__en-us_topic_0113887277_en-us_topic_0111562554_screen9122162775318"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">kafka_sink</span><span class="w"> </span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="n">STRING</span><span class="p">)</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="o">=</span><span class="ss">"kafka"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_bootstrap_servers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"ip1:port1,ip2:port2"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_topic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"testsink"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"json"</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_08_0254__li1618343419283">Example 2:<div class="codecoloring" codetype="Sql" id="dli_08_0254__screen489585832811"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">kafka_sink</span><span class="w"> </span><span class="p">(</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="n">a1</span><span class="w"> </span><span class="n">string</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">a2</span><span class="w"> </span><span class="n">string</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">a3</span><span class="w"> </span><span class="n">string</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">a4</span><span class="w"> </span><span class="nb">INT</span>
|
|
<span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">//</span><span class="w"> </span><span class="k">Output</span><span class="w"> </span><span class="n">Field</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="o">=</span><span class="ss">"kafka"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_bootstrap_servers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"192.x.x.x:9093, 192.x.x.x:9093, 192.x.x.x:9093"</span><span class="p">,</span>
|
|
<span class="n">kafka_topic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"testflink"</span><span class="p">,</span><span class="w"> </span><span class="o">//</span><span class="w"> </span><span class="n">Written</span><span class="w"> </span><span class="n">topic</span>
|
|
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"csv"</span><span class="p">,</span><span class="w"> </span><span class="o">//</span><span class="w"> </span><span class="k">Encoding</span><span class="w"> </span><span class="n">format</span><span class="p">,</span><span class="w"> </span><span class="n">which</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">JSON</span><span class="w"> </span><span class="k">or</span><span class="w"> </span><span class="n">CSV</span><span class="p">.</span>
|
|
<span class="w"> </span><span class="n">kafka_certificate_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"Flink"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_properties_delimiter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">","</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_properties</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"</span><span class="n">xxx</span><span class="err">\</span><span class="ss">" password=\"</span><span class="n">xxx</span><span class="err">\</span><span class="ss">";,sasl.mechanism=PLAIN,security.protocol=SASL_SSL"</span>
|
|
<span class="w"> </span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0240.html">Creating a Sink Stream</a></div>
|
|
</div>
|
|
</div>
|
|
|