vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0301.html
Su, Xiaomeng 04d4597cf3 dli_sqlreference_0511_version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2023-11-02 14:34:08 +00:00

252 lines
28 KiB
HTML
Raw Blame History

<a name="dli_08_0301"></a><a name="dli_08_0301"></a>
<h1 class="topictitle1">Kafka Source Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_dli_08_0237_en-us_topic_0111555123_section17358104193813"><h4 class="sectiontitle">Function</h4><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_dli_08_0239_en-us_topic_0111499973_p18963184212118">Create a source stream to obtain data from Kafka as input data for jobs.</p>
<p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_dli_08_0239_en-us_topic_0111499973_p433435316341">Apache Kafka is a fast, scalable, and fault-tolerant distributed message publishing and subscription system. It delivers high throughput and built-in partitions and provides data replicas and fault tolerance. Apache Kafka is applicable to scenarios of handling massive messages.</p>
</div>
<div class="section" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_dli_08_0237_section1144171011544"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_dli_08_0237_p1038754112542">Kafka is an offline cluster. You have built an enhanced datasource connection to connect Flink jobs to Kafka. You have set security group rules as required.</p>
</div>
<div class="section" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_section13191959213"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p247314912217">SASL_SSL cannot be enabled for the interconnected Kafka cluster.</p>
</div>
<div class="section" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_dli_08_0237_en-us_topic_0111555123_section1832173783817"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_dli_08_0237_screen16634186122414"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">kafkaSource</span><span class="p">(</span>
<span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="p">,</span><span class="w"> </span><span class="p">...)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">ENFORCED</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">WATERMARK</span><span class="w"> </span><span class="k">FOR</span><span class="w"> </span><span class="n">rowtime_column_name</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">watermark</span><span class="o">-</span><span class="n">strategy_expression</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'kafka'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.version'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.topic'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.properties.bootstrap.servers'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.properties.group.id'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.startup-mode'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'format.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_section13754915155510"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_table9699145182314" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row8700645142312"><th align="left" class="cellrowborder" valign="top" width="23.2023202320232%" id="mcps1.3.5.2.2.4.1.1"><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p570074517233">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="7.9007900790079%" id="mcps1.3.5.2.2.4.1.2"><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p127001445132317">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="68.89688968896888%" id="mcps1.3.5.2.2.4.1.3"><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1170018458238">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row117001545112312"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p97001745122316">connector.type</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1770014518232">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1470014522317">Connector type. Set this parameter to <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b644495411810">kafka</strong>.</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row1370016459231"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p6700114572315">connector.version</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p16700144582313">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p6700174511232">Kafka version. The value can be '<strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b327071812597">0.10'</strong> or <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b15544102075917">'0.11'</strong>, which corresponds to Kafka 2.11 to 2.4.0 and other historical versions, respectively.</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row170034520236"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p77001945152315">format.type</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p470024502310">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1170034516235">Data deserialization format. The value can be <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b5741144115194">csv</strong>, <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b7421843171917">json</strong>, or <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b267354471920">avro</strong>.</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row10174103611235"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1919231813244">format.field-delimiter</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p17192151882416">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p31922184244">Attribute delimiter. You can customize the attribute delimiter only when the encoding format is CSV. The default delimiter is a comma (,).</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row13700164582310"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p18700184514239">connector.topic</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p17002457236">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p270074562311">Kafka topic name. Either this parameter or <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b16614125841915">connector.topic-pattern</strong> is used.</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row15011844104518"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p15501194411458">connector.topic-pattern</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p12501144184519">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p18502164417457">Regular expression for matching the Kafka topic name. Either this parameter or <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b1637242372111">connector.topic</strong> is used.</p>
<p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p109825913515">Example:</p>
<p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p591562625211">'topic.*'</p>
<p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1549183485212">'(topic-c|topic-d)'</p>
<p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p562915211509">'(topic-a|topic-b|topic-\\d*)'</p>
<p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p139521105526">'(topic-a|topic-b|topic-[0-9]*)'</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row11700114512233"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1970094522311">connector.properties.bootstrap.servers</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p270014516239">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p117006459232">Kafka broker addresses. Use commas (,) to separated them.</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row14700174532318"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p07001945152310">connector.properties.group.id</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p16700134522314">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1870015454237">Consumer group name</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row147001745122314"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p167001645182314">connector.startup-mode</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p77001945152314">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1870034513231">Consumer startup mode. The value can be <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b9363171811291">earliest-offset</strong>, <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b9572152113291">latest-offset</strong>, <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b12721625142915">group-offsets</strong>, <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b12272628102915">specific-offsets</strong> or <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b9870163220294">timestamp</strong>. The default value is <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b13470164116291">group-offsets</strong>.</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row11700114511236"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p177000451231">connector.specific-offsets</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p7700164515236">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p670010457234">Consumption offset. This parameter is mandatory when <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b1566116341579">startup-mode</strong> is <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b53555407575">specific-offsets</strong>. The value is in the 'partition:0,offset:42;partition:1,offset:300' format.</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row177001845142316"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p9700045132316">connector.startup-timestamp-millis</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1070024510238">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p2070012459233">Consumption start timestamp. This parameter is mandatory when <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b207971255917">startup-mode</strong> is <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b201051220185918">timestamp</strong>.</p>
</td>
</tr>
<tr id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_row19700845182318"><td class="cellrowborder" valign="top" width="23.2023202320232%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p2700134516233">connector.properties.*</p>
</td>
<td class="cellrowborder" valign="top" width="7.9007900790079%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p87008456238">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.89688968896888%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p2700104542319">Native Kafka property</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_dli_08_0237_section11421328152716"><h4 class="sectiontitle">Example</h4><ul id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_ul172685117610"><li id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_li972614511569">Create table <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b1249103729">kafkaSource</strong> and read data encoded in CSV format from Kafka.<div class="codecoloring" codetype="Sql" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_screen1496541416910"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">kafkaSource</span><span class="p">(</span>
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_speed</span><span class="w"> </span><span class="nb">INT</span><span class="p">)</span>
<span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'kafka'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.version'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'0.11'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.topic'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'test-topic'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.properties.bootstrap.servers'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xx.xx.xx.xx:9092'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.properties.group.id'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'test-group'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.startup-mode'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'latest-offset'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'format.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'csv'</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_li19651814198">Create table <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b01049019615">kafkaSource</strong> and read data in non-nested JSON strings from Kafka.<div class="p" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p17591120171516">Assume that the non-nested JSON strings are as follows:<pre class="screen" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_screen5591702155">{"car_id": 312, "car_owner": "wang", "car_brand": "tang"}
{"car_id": 313, "car_owner": "li", "car_brand": "lin"}
{"car_id": 314, "car_owner": "zhao", "car_brand": "han"}</pre>
</div>
<div class="p" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p165911506157">You can create the table as follows:<div class="codecoloring" codetype="Sql" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_screen165913015155"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">kafkaSource</span><span class="p">(</span>
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span>
<span class="p">)</span>
<span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'kafka'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.version'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'0.11'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.topic'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'test-topic'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.properties.bootstrap.servers'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xx.xx.xx.xx:9092'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.properties.group.id'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'test-group'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.startup-mode'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'latest-offset'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'format.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'json'</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
</li><li id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_li559150171519">Create table <strong id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_b176837641211">kafkaSource</strong> and read the nested JSON data from Kafka.<p id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p1869545515153">Assume that the JSON data is as follows:</p>
<pre class="screen" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_screen1760454191617">{
"id":"1",
"type":"online",
"data":{
"patient_id":1234,
"name":"bob1234",
"age":"Bob",
"gmt_create":"Bob",
"gmt_modify":"Bob"
}
}</pre>
<div class="p" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_p63441310131616">You can create the table as follows:<pre class="screen" id="dli_08_0301__en-us_topic_0000001165951915_en-us_topic_0000001081419144_screen0832181771611">CREATE table kafkaSource(
id STRING,
type STRING,
data ROW(
patient_id STRING,
name STRING,
age STRING,
gmt_create STRING,
gmt_modify STRING)
)
with (
'connector.type' = 'kafka',
'connector.version' = '0.11',
'connector.topic' = 'test-topic',
'connector.properties.bootstrap.servers' = 'xx.xx.xx.xx:9092',
'connector.properties.group.id' = 'test-group',
'connector.startup-mode' = 'latest-offset',
'format.type' = 'json'
);</pre>
</div>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0300.html">Creating a Source Table</a></div>
</div>
</div>
View Git Blame Copy Permalink