vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0398.html
Su, Xiaomeng 04d4597cf3 dli_sqlreference_0511_version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2023-11-02 14:34:08 +00:00

496 lines
51 KiB
HTML
Raw Blame History

<a name="dli_08_0398"></a><a name="dli_08_0398"></a>
<h1 class="topictitle1">Kafka Result Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0398__en-us_topic_0000001262655734_dli_08_0256_en-us_topic_0132788972_section108631122164917"><h4 class="sectiontitle">Function</h4><p id="dli_08_0398__en-us_topic_0000001262655734_p661816320442">DLI outputs the Flink job output data to Kafka through the Kafka result table.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p2618163211448">Apache Kafka is a fast, scalable, and fault-tolerant distributed message publishing and subscription system. It delivers high throughput and built-in partitions and provides data replicas and fault tolerance. Apache Kafka is applicable to scenarios of handling massive messages.</p>
</div>
<div class="section" id="dli_08_0398__en-us_topic_0000001262655734_dli_08_0256_en-us_topic_0132788972_section2579142713429"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0398__en-us_topic_0000001262655734_ul578517503176"><li id="dli_08_0398__en-us_topic_0000001262655734_li2785750141711">You have created a Kafka cluster.</li><li id="dli_08_0398__en-us_topic_0000001262655734_li1627119813515">An enhanced datasource connection has been created for DLI to connect to Kafka clusters, so that jobs can run on the dedicated queue of DLI and you can set the security group rules as required.
</li></ul>
</div>
<div class="section" id="dli_08_0398__en-us_topic_0000001262655734_section1173351352211"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0398__en-us_topic_0000001262655734_ul197731050557"><li id="dli_08_0398__en-us_topic_0000001262655734_li6647155118407">When creating a Flink OpenSource SQL job, you need to set <strong id="dli_08_0398__en-us_topic_0000001262655734_b12586415192014">Flink Version</strong> to <strong id="dli_08_0398__en-us_topic_0000001262655734_b2587131510202">1.12</strong> on the <strong id="dli_08_0398__en-us_topic_0000001262655734_b1258751552012">Running Parameters</strong> tab of the job editing page, select <strong id="dli_08_0398__en-us_topic_0000001262655734_b17588615132019">Save Job Log</strong>, and set the OBS bucket for saving job logs.</li><li id="dli_08_0398__en-us_topic_0000001262655734_li396415223476">For details about how to use data types, see section <a href="dli_08_0407.html#dli_08_0407">Format</a>.</li></ul>
</div>
<div class="section" id="dli_08_0398__en-us_topic_0000001262655734_dli_08_0256_en-us_topic_0132788972_section3126105364419"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0398__en-us_topic_0000001262655734_dli_08_0256_screen1461215294716"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">kafkaSink</span><span class="p">(</span>
<span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="p">,</span><span class="w"> </span><span class="p">...)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">ENFORCED</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'kafka'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'topic'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'properties.bootstrap.servers'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'format'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0398__en-us_topic_0000001262655734_dli_08_0256_section4299113491"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0398__en-us_topic_0000001262655734_table977517125819" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0398__en-us_topic_0000001262655734_row37759121888"><th align="left" class="cellrowborder" valign="top" width="20.002000200020003%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0398__en-us_topic_0000001262655734_p1361712418461">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.450945094509452%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0398__en-us_topic_0000001262655734_p176171424114615">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="12.13121312131213%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0398__en-us_topic_0000001262655734_p20935421305">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="14.02140214021402%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0398__en-us_topic_0000001262655734_p184921935103118">Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="44.39443944394439%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0398__en-us_topic_0000001262655734_p261712247467">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0398__en-us_topic_0000001262655734_row677511121684"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p477501214812">connector</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p167750121185">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p5298145623719">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p2077520121583">string</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p12863164218916">Connector to be used. Set this parameter to <strong id="dli_08_0398__en-us_topic_0000001262655734_b1929717115211">kafka</strong>.</p>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row1977581217811"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p87751212889">topic</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p16775161211818">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p0775121214813">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p107755121382">string</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p377517129815">Topic name of the Kafka result table.</p>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row67751512689"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p1077519121088">properties.bootstrap.servers</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p57751112987">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p12775161214811">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p277531215814">string</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p10775912581">Kafka broker address. The value is in the format of <strong id="dli_08_0398__en-us_topic_0000001262655734_b9809141520">host:port,host:port,host:port</strong>. Multiple <strong id="dli_08_0398__en-us_topic_0000001262655734_b524142213526">host:port</strong> pairs are separated with commas (,).</p>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row37751612380"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p77751812381">format</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p197757125811">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p197751112882">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p3775212485">string</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p17391532174016">Format used by the Flink Kafka connector to serialize Kafka messages. Either this parameter or the <strong id="dli_08_0398__en-us_topic_0000001262655734_b16783277554">value.format</strong> parameter is required.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p10140201324017">The following formats are supported:</p>
<ul id="dli_08_0398__en-us_topic_0000001262655734_ul5245162971319"><li id="dli_08_0398__en-us_topic_0000001262655734_li624592913136">csv</li><li id="dli_08_0398__en-us_topic_0000001262655734_li1972514331136">json</li><li id="dli_08_0398__en-us_topic_0000001262655734_li32653741313">avro</li></ul>
<p id="dli_08_0398__en-us_topic_0000001262655734_p1386923574012">Refer to <a href="dli_08_0407.html#dli_08_0407">Format</a> for more details and format parameters.</p>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row2571536523"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p17554731112214">topic-pattern</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p355483132220">No</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p128993513389">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p455493116222">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p415223719318">Regular expression for matching the Kafka topic name.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p1797018378534">Only one of <strong id="dli_08_0398__en-us_topic_0000001262655734_b87831345105611">topic</strong> and <strong id="dli_08_0398__en-us_topic_0000001262655734_b1278413450568">topic-pattern</strong> can be specified.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p16915112317">Example: 'topic.*'</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p1915151183110">'(topic-c|topic-d)'</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p791581153114">'(topic-a|topic-b|topic-\\d*)'</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p10915171193115">'(topic-a|topic-b|topic-[0-9]*)'</p>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row169241352328"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p1755413314229">properties.*</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p115541531182212">No</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p35547318225">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p10554113119225">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p97929517327">This parameter can set and pass arbitrary Kafka configurations.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p773119105325">Note:</p>
<ul id="dli_08_0398__en-us_topic_0000001262655734_ul0491787522"><li id="dli_08_0398__en-us_topic_0000001262655734_li199931630195813">Suffix names must match the configuration key defined in <a href="https://kafka.apache.org/documentation/#configuration" target="_blank" rel="noopener noreferrer">Apache Kafka</a>.<p id="dli_08_0398__en-us_topic_0000001262655734_p948323185810">For example, you can disable automatic topic creation via <strong id="dli_08_0398__en-us_topic_0000001262655734_b7883743165816">'properties.</strong><strong id="dli_08_0398__en-us_topic_0000001262655734_b17889743155815">allow.auto.create.topics</strong><strong id="dli_08_0398__en-us_topic_0000001262655734_b78901143155818">' = 'false'</strong>.</p>
</li><li id="dli_08_0398__en-us_topic_0000001262655734_li194440530015">Some configurations are not supported, for example, <strong id="dli_08_0398__en-us_topic_0000001262655734_b1984716480582">'key.deserializer'</strong> and <strong id="dli_08_0398__en-us_topic_0000001262655734_b1684714816588">'value.deserializer'</strong>.</li></ul>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row453621322415"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p143321611172620">key.format</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p134901013182611">No</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p4668191610260">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p269571032719">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p116462027182719">Format used to deserialize and serialize the key part of Kafka messages.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p3646227172716">Note:</p>
<ul id="dli_08_0398__en-us_topic_0000001262655734_ul17646627102717"><li id="dli_08_0398__en-us_topic_0000001262655734_li26466278275">If a key format is defined, the <strong id="dli_08_0398__en-us_topic_0000001262655734_b19868795914">key.fields</strong> parameter is required as well. Otherwise, the Kafka records will have an empty key.</li><li id="dli_08_0398__en-us_topic_0000001262655734_li12646327152714">Possible values are:<p id="dli_08_0398__en-us_topic_0000001262655734_p2646127182718"><a name="dli_08_0398__en-us_topic_0000001262655734_li12646327152714"></a><a name="en-us_topic_0000001262655734_li12646327152714"></a>csv</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p164662752718">json</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p14646527182718">avro</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p1364692712273">debezium-json</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p76461927192711">canal-json</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p13647727132714">maxwell-json</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p106472277273">avro-confluent</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p4647182742716">raw</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p1341952616301">Refer to <a href="dli_08_0407.html#dli_08_0407">Format</a> for more details and format parameters.</p>
</li></ul>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row87271510172413"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p45551331122214">key.fields</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p145552031142215"></p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p1955515318222">No</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p1955503122213"></p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p5555103112210">[]</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p155563192211">List&lt;String&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p19922554193214">Defines the columns in the table as the list of keys. This parameter must be configured in pair with <strong id="dli_08_0398__en-us_topic_0000001262655734_b11776193515591">key.format</strong>.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p2639195952913">This parameter is left empty by default. Therefore, no key is defined.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p863975917298">The format is like <strong id="dli_08_0398__en-us_topic_0000001262655734_b15151114910597">field1;field2</strong>.</p>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row11602370297"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p1555519310225">key.fields-prefix</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p125552031152220">No</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p47941413143815">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p20555231112219">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p1955593111223">Defines a custom prefix for all fields of the key format to avoid name clashes with fields of the value format.</p>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row12919104013294"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p10555163112220">value.format</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p1055593118223">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p2555231192220">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p7555193116229">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p8314133141511">Format used to deserialize and serialize the value part of Kafka messages.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p7555431122212">Note:</p>
<ul id="dli_08_0398__en-us_topic_0000001262655734_ul166571238111518"><li id="dli_08_0398__en-us_topic_0000001262655734_li667718114163">Either this parameter or the <strong id="dli_08_0398__en-us_topic_0000001262655734_b154993310017">format</strong> parameter is required. If two parameters are configured, a conflict occurs.</li><li id="dli_08_0398__en-us_topic_0000001262655734_li1641151847">Refer to <a href="dli_08_0407.html#dli_08_0407">Format</a> for more details and format parameters.</li></ul>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row161581545162911"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p142611629113120">value.fields-include</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p1526172913312">No</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p13555133172214">ALL</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p134898438282">Enum</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p548954310282">Possible values: [ALL, EXCEPT_KEY]</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p14555193120228">Whether to contain the key field when parsing the message body.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p83061413192819">Possible values are:</p>
<ul id="dli_08_0398__en-us_topic_0000001262655734_ul133062136285"><li id="dli_08_0398__en-us_topic_0000001262655734_li63061713192810"><strong id="dli_08_0398__en-us_topic_0000001262655734_b1261310133218">ALL</strong> (default): All defined fields are included in the value of Kafka messages.</li><li id="dli_08_0398__en-us_topic_0000001262655734_li1430611132287"><strong id="dli_08_0398__en-us_topic_0000001262655734_b2066012519214">EXCEPT_KEY</strong>: All the fields except those defined by <strong id="dli_08_0398__en-us_topic_0000001262655734_b1866712251321">key.fields</strong> are included in the value of Kafka messages.</li></ul>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row187417221360"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p14165158181310">sink.partitioner</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p17165145851315">No</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p162722018203814">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p61659584136">string</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p416585861316">Mapping from Flink's partitions into Kafka's partitions. Valid values are as follows:</p>
<ul id="dli_08_0398__en-us_topic_0000001262655734_ul10617348191417"><li id="dli_08_0398__en-us_topic_0000001262655734_li2618048171412"><strong id="dli_08_0398__en-us_topic_0000001262655734_b1171063061013">fixed</strong> (default): Each Flink partition ends up in at most one Kafka partition.</li><li id="dli_08_0398__en-us_topic_0000001262655734_li675113191517"><strong id="dli_08_0398__en-us_topic_0000001262655734_b10876154341011">round-robin</strong>: A Flink partition is distributed to Kafka partitions in a round-robin manner.</li><li id="dli_08_0398__en-us_topic_0000001262655734_li1097371131515"><strong id="dli_08_0398__en-us_topic_0000001262655734_b1954553110156">Custom FlinkKafkaPartitioner subclass</strong>: If <strong id="dli_08_0398__en-us_topic_0000001262655734_b149619071613">fixed</strong> and <strong id="dli_08_0398__en-us_topic_0000001262655734_b1917112141618">round-robin</strong> do not meet your requirements, you can create subclass <strong id="dli_08_0398__en-us_topic_0000001262655734_b9106131413160">FlinkKafkaPartitioner</strong> to customize the partition mapping, for example, <strong id="dli_08_0398__en-us_topic_0000001262655734_b668922220179">org.mycompany.MyPartitioner</strong>.</li></ul>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row566562418610"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p18666152418611">sink.semantic</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p7666124263">No</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p10666824760">at-least-once</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p666615245619">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p86661824768">Defines the delivery semantic for the Kafka sink.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p95381240796">Valid values are as follows:</p>
<ul id="dli_08_0398__en-us_topic_0000001262655734_ul359812471193"><li id="dli_08_0398__en-us_topic_0000001262655734_li7598147497">at-least-once</li><li id="dli_08_0398__en-us_topic_0000001262655734_li17264145417910">exactly-once</li><li id="dli_08_0398__en-us_topic_0000001262655734_li1246411582090">none</li></ul>
</td>
</tr>
<tr id="dli_08_0398__en-us_topic_0000001262655734_row12198104010611"><td class="cellrowborder" valign="top" width="20.002000200020003%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0398__en-us_topic_0000001262655734_p51982040669">sink.parallelism</p>
</td>
<td class="cellrowborder" valign="top" width="9.450945094509452%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0398__en-us_topic_0000001262655734_p619914406615">No</p>
</td>
<td class="cellrowborder" valign="top" width="12.13121312131213%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0398__en-us_topic_0000001262655734_p101998409615">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.02140214021402%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0398__en-us_topic_0000001262655734_p13199194011610">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="44.39443944394439%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0398__en-us_topic_0000001262655734_p61991840164">Defines the parallelism of the Kafka sink operator.</p>
<p id="dli_08_0398__en-us_topic_0000001262655734_p565142051818">By default, the parallelism is determined by the framework using the same parallelism of the upstream chained operator.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0398__en-us_topic_0000001262655734_dli_08_0256_en-us_topic_0132788972_section19670175612466"><h4 class="sectiontitle">Example (SASL_SSL Disabled for the Kafka Cluster)</h4><p id="dli_08_0398__en-us_topic_0000001262655734_p1326131081119">In this example, data is read from a Kafka topic and written to another using a Kafka result table.</p>
<ol id="dli_08_0398__en-us_topic_0000001262655734_ol227413918186"><li id="dli_08_0398__en-us_topic_0000001262655734_li12741739181820">Create an enhanced datasource connection in the VPC and subnet where Kafka locates, and bind the connection to the required Flink elastic resource pool.</li><li id="dli_08_0398__en-us_topic_0000001262655734_li1427423912182">Set Kafka security groups and add inbound rules to allow access from the Flink queue. Test the connectivity using the Kafka address. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</li><li id="dli_08_0398__en-us_topic_0000001262655734_li227423912182">Create a Flink OpenSource SQL job. Enter the following job script and submit the job.<div class="p" id="dli_08_0398__en-us_topic_0000001262655734_p39352912333"><a name="dli_08_0398__en-us_topic_0000001262655734_li227423912182"></a><a name="en-us_topic_0000001262655734_li227423912182"></a>When you create a job, set <strong id="dli_08_0398__en-us_topic_0000001262655734_b1899611468194">Flink Version</strong> to <strong id="dli_08_0398__en-us_topic_0000001262655734_b1199774671912">1.12</strong> on the <strong id="dli_08_0398__en-us_topic_0000001262655734_b18997114631917">Running Parameters</strong> tab. Select <strong id="dli_08_0398__en-us_topic_0000001262655734_b19998946141918">Save Job Log</strong>, and specify the OBS bucket for saving job logs. <strong id="dli_08_0398__en-us_topic_0000001262655734_b1158014814196">Change the values of the parameters in bold as needed in the following script.</strong><pre class="screen" id="dli_08_0398__en-us_topic_0000001262655734_screen550720113134">CREATE TABLE kafkaSource (
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = '<em id="dli_08_0398__en-us_topic_0000001262655734_i4543155265320"><strong id="dli_08_0398__en-us_topic_0000001262655734_b1754316528535">Kafka</strong></em><em id="dli_08_0398__en-us_topic_0000001262655734_i15341759195319"><strong id="dli_08_0398__en-us_topic_0000001262655734_b5227105913538">Topic</strong></em>',
'properties.bootstrap.servers' = '<em id="dli_08_0398__en-us_topic_0000001262655734_i9319103215313"><strong id="dli_08_0398__en-us_topic_0000001262655734_b0322115133213">KafkaAddress1:KafkaPort,KafkaAddress2:KafkaPort</strong></em>',
'properties.group.id' = '<em id="dli_08_0398__en-us_topic_0000001262655734_i14535174011311"><strong id="dli_08_0398__en-us_topic_0000001262655734_b1466610178329">GroupId</strong></em>',
'scan.startup.mode' = 'latest-offset',
"format" = "json"
);
CREATE TABLE kafkaSink (
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = '<em id="dli_08_0398__en-us_topic_0000001262655734_i158031562095"><strong id="dli_08_0398__en-us_topic_0000001262655734_b1680313611916">KafkaSink</strong></em><em id="dli_08_0398__en-us_topic_0000001262655734_i6803186895"><strong id="dli_08_0398__en-us_topic_0000001262655734_b28039615911">Topic</strong></em>',
'properties.bootstrap.servers' = '<em id="dli_08_0398__en-us_topic_0000001262655734_i16651521292"><strong id="dli_08_0398__en-us_topic_0000001262655734_b66510212910">KafkaAddress1:KafkaPort,KafkaAddress2:KafkaPort</strong></em>',
"format" = "json"
);
insert into kafkaSink select * from kafkaSource;</pre>
</div>
</li><li id="dli_08_0398__en-us_topic_0000001262655734_li827413910185">Connect to the Kafka cluster and insert the following test data into the source topic in Kafka:<pre class="screen" id="dli_08_0398__en-us_topic_0000001262655734_screen1643116512133">{"order_id":"202103241000000001","order_channel":"webShop","order_time":"2021-03-24 10:00:00","pay_amount":100.0,"real_pay":100.0,"pay_time":"2021-03-24 10:02:03","user_id":"0001","user_name":"Alice","area_id":"330106"}
{"order_id":"202103241606060001","order_channel":"appShop","order_time":"2021-03-24 16:06:06","pay_amount":200.0,"real_pay":180.0,"pay_time":"2021-03-24 16:10:06","user_id":"0001","user_name":"Alice","area_id":"330106"}</pre>
</li><li id="dli_08_0398__en-us_topic_0000001262655734_li202745396183">Connect to the Kafka cluster and read data from the sink topic of Kafka.<pre class="screen" id="dli_08_0398__en-us_topic_0000001262655734_screen11212991415">{"order_id":"202103241000000001","order_channel":"webShop","order_time":"2021-03-24 10:00:00","pay_amount":100.0,"real_pay":100.0,"pay_time":"2021-03-24 10:02:03","user_id":"0001","user_name":"Alice","area_id":"330106"}
{"order_id":"202103241606060001","order_channel":"appShop","order_time":"2021-03-24 16:06:06","pay_amount":200.0,"real_pay":180.0,"pay_time":"2021-03-24 16:10:06","user_id":"0001","user_name":"Alice","area_id":"330106"}</pre>
</li></ol>
</div>
<div class="section" id="dli_08_0398__en-us_topic_0000001262655734_section1376121165311"><h4 class="sectiontitle">Example (SASL_SSL Enabled for the Kafka Cluster)</h4><ul id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_ul165787314155"><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li157853171516"><strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1615142615405">Example 1: Enable SASL_SSL authentication for the DMS cluster.</strong><p id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_p48839353813">Create a Kafka cluster for DMS, enable SASL_SSL, download the SSL certificate, and upload the downloaded certificate <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b727121674119">client.jks</strong> to an OBS bucket.</p>
<pre class="screen" id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_screen691531701511">CREATE TABLE ordersSource (
order_id string,
order_channel string,
order_time timestamp(3),
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = 'xx',
'properties.bootstrap.servers' = 'xx:9093,xx:9093,xx:9093',
'properties.group.id' = 'GroupId',
'scan.startup.mode' = 'latest-offset',
'properties.connector.auth.open' = 'true',
'properties.ssl.truststore.location' = 'obs://xx/xx.jks', -- Location where the user uploads the certificate to
'properties.sasl.mechanism' = 'PLAIN', -- Value format: SASL_PLAINTEXT
'properties.security.protocol' = 'SASL_SSL',
'properties.sasl.jaas.config' = 'org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xx\" password=\"xx\";', -- Account and password set when the Kafka cluster is created
"format" = "json"
);
CREATE TABLE ordersSink (
order_id string,
order_channel string,
order_time timestamp(3),
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = 'xx',
'properties.bootstrap.servers' = 'xx:9093,xx:9093,xx:9093',
'properties.connector.auth.open' = 'true',
'properties.ssl.truststore.location' = 'obs://xx/xx.jks',
'properties.sasl.mechanism' = 'PLAIN',
'properties.security.protocol' = 'SASL_SSL',
'properties.sasl.jaas.config' = 'org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xx\" password=\"xx\";',
"format" = "json"
);
insert into ordersSink select * from ordersSource;</pre>
</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li107204111517"><strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b153566314430">Example 2: Enable Kafka SASL_SSL authentication for the MRS cluster.</strong><ul id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_ul0537181312174"><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li11610111071719">Enable Kerberos authentication for the MRS cluster.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li1261921017172">Click the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1624616617165">Components</strong> tab and click <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b196895461618">Kafka</strong>. In the displayed page, click the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b7886122919170">Service Configuration</strong> tab, locate the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1117255781713">security.protocol</strong>, and set it to <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b4822184151812">SASL_SSL</strong>.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li06912478184">Log in to the FusionInsight Manager of the MRS cluster and download the user credential. Choose <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b999511506233">System</strong> &gt; <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1297675202315">Permission</strong> &gt; <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1057815532312">User</strong>. Locate the row that contains the target user, choose <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b9521732152415">More</strong> &gt; <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b65063351244">Download Authentication Credential</strong>.<p id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_p333645161810">Obtain the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1872285013246">truststore.jks</strong> file using the authentication credential and store the credential and <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b18744182712255">truststore.jks</strong> file in OBS.</p>
</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li061981014175">If "Message stream modified (41)" is displayed, the JDK version may be incorrect. Change the JDK version in the sample code to a version earlier than 8u_242 or delete the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b118581656102610">renew_lifetime = 0m</strong> configuration item from the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b4858145692613">krb5.conf</strong> configuration file.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li10619121014179">Set the port to the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b12616371276">sasl_ssl.port</strong> configured in the Kafka service configuration.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li861916104173">In the following statements, set <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b32437492279">security.protocol</strong> to <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b31069517275">SASL_SSL</strong>.</li></ul>
<pre class="screen" id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_screen14604182619228">CREATE TABLE ordersSource (
order_id string,
order_channel string,
order_time timestamp(3),
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = 'xx',
'properties.bootstrap.servers' = 'xx:21009,xx:21009',
'properties.group.id' = 'GroupId',
'scan.startup.mode' = 'latest-offset',
'properties.sasl.kerberos.service.name' = 'kafka',
'properties.connector.auth.open' = 'true',
'properties.connector.kerberos.principal' = 'xx', --Username
'properties.connector.kerberos.krb5' = 'obs://xx/krb5.conf',
'properties.connector.kerberos.keytab' = 'obs://xx/user.keytab',
'properties.security.protocol' = 'SASL_SSL',
'properties.ssl.truststore.location' = 'obs://xx/truststore.jks',
'properties.ssl.truststore.password' = 'xx', -- Password set for generating <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b415514783818">truststore.jks</strong>
'properties.sasl.mechanism' = 'GSSAPI',
"format" = "json"
);
CREATE TABLE ordersSink (
order_id string,
order_channel string,
order_time timestamp(3),
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = 'xx',
'properties.bootstrap.servers' = 'xx:21009,xx:21009',
'properties.sasl.kerberos.service.name' = 'kafka',
'properties.connector.auth.open' = 'true',
'properties.connector.kerberos.principal' = 'xx',
'properties.connector.kerberos.krb5' = 'obs://xx/krb5.conf',
'properties.connector.kerberos.keytab' = 'obs://xx/user.keytab',
'properties.ssl.truststore.location' = 'obs://xx/truststore.jks',
'properties.ssl.truststore.password' = 'xx',
'properties.security.protocol' = 'SASL_SSL',
'properties.sasl.mechanism' = 'GSSAPI',
"format" = "json"
);
insert into ordersSink select * from ordersSource;</pre>
</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li3317747102218"><strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b315652615317">Example 3: Enable Kerberos SASL_PAINTEXT authentication for the MRS cluster</strong><ul id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_ul184471113132319"><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li144471513152311">Enable Kerberos authentication for the MRS cluster.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li10447161315237">Click the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b6590105635618">Components</strong> tab and click <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b7590856195611">Kafka</strong>. In the displayed page, click the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b859185619564">Service Configuration</strong> tab, locate the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b185911256115614">security.protocol</strong>, and set it to <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b15911856145610">SASL_PLAINTEXT</strong>.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li544713139232">Log in to the FusionInsight Manager of the MRS cluster and download the user credential. Choose <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b122614158577">System</strong> &gt; <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b11261191510574">Permission</strong> &gt; <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b12261315145714">User</strong>. Locate the row that contains the target user, choose <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b15262171519579">More</strong> &gt; <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b112621415105712">Download Authentication Credential</strong>. Upload the credential to OBS.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li744713131237">If error message "Message stream modified (41)" is displayed, the JDK version may be incorrect. Change the JDK version in the sample code to a version earlier than 8u_242 or delete the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1095743618578">renew_lifetime = 0m</strong> configuration item from the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1295713616570">krb5.conf</strong> configuration file.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li544751382315">Set the port to the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b154639199256">sasl.port</strong> configured in the Kafka service configuration.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li55706303232">In the following statements, set <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b614482182912">security.protocol</strong> to <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b8144142122913">SASL_PLAINTEXT</strong>.</li></ul>
<pre class="screen" id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_screen17997112152617">CREATE TABLE ordersSources (
order_id string,
order_channel string,
order_time timestamp(3),
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = 'xx',
'properties.bootstrap.servers' = 'xx:21007,xx:21007',
'properties.group.id' = 'GroupId',
'scan.startup.mode' = 'latest-offset',
'properties.sasl.kerberos.service.name' = 'kafka',
'properties.connector.auth.open' = 'true',
'properties.connector.kerberos.principal' = 'xx',
'properties.connector.kerberos.krb5' = 'obs://xx/krb5.conf',
'properties.connector.kerberos.keytab' = 'obs://xx/user.keytab',
'properties.security.protocol' = 'SASL_PLAINTEXT',
'properties.sasl.mechanism' = 'GSSAPI',
"format" = "json"
);
CREATE TABLE ordersSink (
order_id string,
order_channel string,
order_time timestamp(3),
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = 'xx',
'properties.bootstrap.servers' = 'xx:21007,xx:21007',
'properties.sasl.kerberos.service.name' = 'kafka',
'properties.connector.auth.open' = 'true',
'properties.connector.kerberos.principal' = 'xx',
'properties.connector.kerberos.krb5' = 'obs://xx/krb5.conf',
'properties.connector.kerberos.keytab' = 'obs://xx/user.keytab',
'properties.security.protocol' = 'SASL_PLAINTEXT',
'properties.sasl.mechanism' = 'GSSAPI',
"format" = "json"
);
insert into ordersSink select * from ordersSource;</pre>
</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li9872336192610"><strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b34356665818">Example 4: Use SSL for the MRS cluster</strong><ul id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_ul56598200271"><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li196591420192719">Do not enable Kerberos authentication for the MRS cluster.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li8781164792714">Log in to the FusionInsight Manager of the MRS cluster and download the user credential. Choose <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b266414402613">System</strong> &gt; <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b266418405616">Permission</strong> &gt; <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1066420409617">User</strong>. Locate the row that contains the target user, choose <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1366412401611">More</strong> &gt; <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b4665144012612">Download Authentication Credential</strong>.<p id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_p626025219286">Obtain the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b15842547365">truststore.jks</strong> file using the authentication credential and store the credential and <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b178428471760">truststore.jks</strong> file in OBS.</p>
</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li56593204272">Set the port to the <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b163213314713">ssl.port</strong> configured in the Kafka service configuration.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li14659920202712">In the following statements, set <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b16346417874">security.protocol</strong> to <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b173464171177">SSL</strong>.</li><li id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_li66597202270">Set <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1474518124815">ssl.mode.enable</strong> to <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b154236319337">true</strong>.<pre class="screen" id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_screen842494402916">CREATE TABLE ordersSource (
order_id string,
order_channel string,
order_time timestamp(3),
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = 'xx',
'properties.bootstrap.servers' = 'xx:9093,xx:9093,xx:9093',
'properties.group.id' = 'GroupId',
'scan.startup.mode' = 'latest-offset',
'properties.connector.auth.open' = 'true',
'properties.ssl.truststore.location' = 'obs://xx/truststore.jks',
'properties.ssl.truststore.password' = 'xx', -- Password set for generating <strong id="dli_08_0398__en-us_topic_0000001262655734_en-us_topic_0000001310095781_b1530114386913">truststore.jks</strong>
'properties.security.protocol' = 'SSL',
"format" = "json"
);
CREATE TABLE ordersSink (
order_id string,
order_channel string,
order_time timestamp(3),
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'print'
);
insert into ordersSink select * from ordersSource;</pre>
</li></ul>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0391.html">Creating Result Tables</a></div>
</div>
</div>
View Git Blame Copy Permalink