forked from docs/doc-exports
Su, Xiaomeng
04d4597cf3
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
232 lines
21 KiB
HTML
232 lines
21 KiB
HTML
<a name="dli_08_0408"></a><a name="dli_08_0408"></a>
|
|
|
|
<h1 class="topictitle1">Avro</h1>
|
|
<div id="body8662426"><div class="section" id="dli_08_0408__en-us_topic_0000001309855881_section1699332117215"><h4 class="sectiontitle">Function</h4><p id="dli_08_0408__en-us_topic_0000001309855881_p457512386218">Apache Avro is supported for you to read and write Avro data based on an Avro schema with Flink. The Avro schema is derived from the table schema.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0408__en-us_topic_0000001309855881_section0152105362113"><h4 class="sectiontitle">Supported Connectors</h4><ul id="dli_08_0408__en-us_topic_0000001309855881_ul7219843143018"><li id="dli_08_0408__en-us_topic_0000001309855881_li421954373011">Kafka</li><li id="dli_08_0408__en-us_topic_0000001309855881_li3158115212300">Upsert Kafka</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0408__en-us_topic_0000001309855881_section7549418211"><h4 class="sectiontitle">Parameters</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0408__en-us_topic_0000001309855881_table6232172462311" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter</caption><thead align="left"><tr id="dli_08_0408__en-us_topic_0000001309855881_row18232132462319"><th align="left" class="cellrowborder" valign="top" width="10.83%" id="mcps1.3.3.2.2.6.1.1"><p id="dli_08_0408__en-us_topic_0000001309855881_p2232142432313">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="29.17%" id="mcps1.3.3.2.2.6.1.2"><p id="dli_08_0408__en-us_topic_0000001309855881_p323210243235">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.3.2.2.6.1.3"><p id="dli_08_0408__en-us_topic_0000001309855881_p323232415231">Default value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.3.2.2.6.1.4"><p id="dli_08_0408__en-us_topic_0000001309855881_p14232132416238">Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.3.2.2.6.1.5"><p id="dli_08_0408__en-us_topic_0000001309855881_p42321224172315">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0408__en-us_topic_0000001309855881_row17232182412231"><td class="cellrowborder" valign="top" width="10.83%" headers="mcps1.3.3.2.2.6.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p54831039182715">format</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="29.17%" headers="mcps1.3.3.2.2.6.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p9483639172712">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.3.2.2.6.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p11483103911278">None</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.3.2.2.6.1.4 "><p id="dli_08_0408__en-us_topic_0000001309855881_p4483039152719">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.3.2.2.6.1.5 "><p id="dli_08_0408__en-us_topic_0000001309855881_p2483103915272">Format to be used. Set the value to <strong id="dli_08_0408__en-us_topic_0000001309855881_b1067516101393">avro</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row1023212410233"><td class="cellrowborder" valign="top" width="10.83%" headers="mcps1.3.3.2.2.6.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p648333942714">avro.codec</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="29.17%" headers="mcps1.3.3.2.2.6.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p16483183918279">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.3.2.2.6.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p154834393274">None</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.3.2.2.6.1.4 "><p id="dli_08_0408__en-us_topic_0000001309855881_p1848363902711">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.3.2.2.6.1.5 "><p id="dli_08_0408__en-us_topic_0000001309855881_p12483143992710">Avro compression codec for the file system only The codec is disabled by default. Available values are <strong id="dli_08_0408__en-us_topic_0000001309855881_b255525817404">deflate</strong>, <strong id="dli_08_0408__en-us_topic_0000001309855881_b35441210174114">snappy</strong>, <strong id="dli_08_0408__en-us_topic_0000001309855881_b151697129417">bzip2</strong>, and <strong id="dli_08_0408__en-us_topic_0000001309855881_b17209111414419">xz</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0408__en-us_topic_0000001309855881_section6691185734518"><h4 class="sectiontitle">Data Type Mapping</h4><p id="dli_08_0408__en-us_topic_0000001309855881_p12512158104613">Currently, the Avro schema is derived from the table schema and cannot be explicitly defined. The following table lists mappings between Flink to Avro types.</p>
|
|
<p id="dli_08_0408__en-us_topic_0000001309855881_p1247461719478">In addition to the following types, Flink supports reading/writing nullable types. Flink maps nullable types to Avro <strong id="dli_08_0408__en-us_topic_0000001309855881_b868418311532">union(something, null)</strong>, where <strong id="dli_08_0408__en-us_topic_0000001309855881_b97297525462">something</strong> is an Avro type converted from Flink type.</p>
|
|
<p id="dli_08_0408__en-us_topic_0000001309855881_p4474101794711">You can refer to <a href="https://avro.apache.org/docs/current/spec.html" target="_blank" rel="noopener noreferrer">Apache Avro 1.11.0 Specification</a> for more information about Avro types.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0408__en-us_topic_0000001309855881_table127011632174812" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Data Type Mapping</caption><thead align="left"><tr id="dli_08_0408__en-us_topic_0000001309855881_row6701153244811"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.5.2.4.1.1"><p id="dli_08_0408__en-us_topic_0000001309855881_p3701203218486">Flink SQL Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.5.2.4.1.2"><p id="dli_08_0408__en-us_topic_0000001309855881_p2070153214817">Avro Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.5.2.4.1.3"><p id="dli_08_0408__en-us_topic_0000001309855881_p137011632114817">Avro Logical Type</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0408__en-us_topic_0000001309855881_row19701173224810"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p370112322488">CHAR / VARCHAR / STRING</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p270183214484">string</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p77011932174814">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row147011332174815"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p137011332154819">BOOLEAN</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p370112325486">boolean</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p770110325482">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row17011232104817"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p17011532104816">BINARY / VARBINARY</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p3701532134810">bytes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p070183217489">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row67013329481"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p97011322488">DECIMAL</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p1670183213488">fixed</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p15701532154815">decimal</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row19701153284818"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p177015322486">TINYINT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p5701173210482">int</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p1770163254810">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row4701173224815"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p9701123214484">SMALLINT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p11701103214819">int</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p11701332154814">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row107018327484"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p17017323481">INT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p1701133234816">int</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p187011132114817">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row18701332144820"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p4701173216484">BIGINT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p370133254820">long</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p127011632144818">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row177011432134819"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p470173224817">FLOAT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p4701163215481">float</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p117018325484">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row670133224818"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p3701173274814">DOUBLE</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p1170123211486">double</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p13701732174819">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row570103210486"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p87014325483">DATE</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p15701103214489">int</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p147010322486">date</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row147012032104818"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p20701103264818">TIME</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p1270163224810">int</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p18701123210482">time-millis</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row770123214811"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p27016329485">TIMESTAMP</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p20701173264814">long</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p1370103244811">timestamp-millis</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row6701532114817"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p2701173214483">ARRAY</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p19701532164817">array</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p1270116324482">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row127015324484"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p0701163216480">MAP (keys must be of the string, char, or varchar type.)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p16701143244811">map</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p177018324480">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row1570163264815"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p1070163217486">MULTISET (elements must be of the string, char, or varchar type.)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p13702832194820">map</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p12702133244815">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0408__en-us_topic_0000001309855881_row137021432184818"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.1 "><p id="dli_08_0408__en-us_topic_0000001309855881_p10702232124819">ROW</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.2 "><p id="dli_08_0408__en-us_topic_0000001309855881_p8702203219487">record</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.5.2.4.1.3 "><p id="dli_08_0408__en-us_topic_0000001309855881_p47024324482">-</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0408__en-us_topic_0000001309855881_section1451265719219"><h4 class="sectiontitle">Example</h4><p id="dli_08_0408__en-us_topic_0000001309855881_p10979854205910">Read data from Kafka, deserialize the data to the Avro format, and outputs the data to print.</p>
|
|
<ol id="dli_08_0408__en-us_topic_0000001309855881_ol1276162418571"><li id="dli_08_0408__en-us_topic_0000001309855881_li04031578234"><span>Create a datasource connection for access to the VPC and subnet where Kafka locates and bind the connection to the queue. Set a security group and inbound rule to allow access of the queue and test the connectivity of the queue using the Kafka IP address. For example, locate a general-purpose queue where the job runs and choose <strong id="dli_08_0408__en-us_topic_0000001309855881_b132971191318">More</strong> > <strong id="dli_08_0408__en-us_topic_0000001309855881_b18866634134">Test Address Connectivity</strong> in the <strong id="dli_08_0408__en-us_topic_0000001309855881_b10162181136">Operation</strong> column. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</span></li><li id="dli_08_0408__en-us_topic_0000001309855881_li1599913011242"><span>Create a Flink OpenSource SQL job and select Flink 1.12. Copy the following statement and submit the job:</span><p><pre class="screen" id="dli_08_0408__en-us_topic_0000001309855881_screen175599544570">CREATE TABLE kafkaSource (
|
|
order_id string,
|
|
order_channel string,
|
|
order_time string,
|
|
pay_amount double,
|
|
real_pay double,
|
|
pay_time string,
|
|
user_id string,
|
|
user_name string,
|
|
area_id string
|
|
) WITH (
|
|
'connector' = 'kafka',
|
|
'topic' = '<yourTopic>',
|
|
'properties.bootstrap.servers' = '<yourKafkaAddress1>:<yourKafkaPort>,<yourKafkaAddress2>:<yourKafkaPort>,<yourKafkaAddress3>:<yourKafkaPort>',
|
|
'properties.group.id' = '<yourGroupId>',
|
|
'scan.startup.mode' = 'latest-offset',
|
|
"format" = "avro"
|
|
);
|
|
|
|
CREATE TABLE printSink (
|
|
order_id string,
|
|
order_channel string,
|
|
order_time string,
|
|
pay_amount double,
|
|
real_pay double,
|
|
pay_time string,
|
|
user_id string,
|
|
user_name string,
|
|
area_id string
|
|
) WITH (
|
|
'connector' = 'print'
|
|
);
|
|
|
|
insert into printSink select * from kafkaSource;</pre>
|
|
</p></li><li id="dli_08_0408__en-us_topic_0000001309855881_li122251732830"><span>Insert the following data to Kafka using Avro data serialization:</span><p><pre class="screen" id="dli_08_0408__en-us_topic_0000001309855881_screen144861201921">{"order_id":"202103241000000001","order_channel":"webShop","order_time":"2021-03-24 10:00:00","pay_amount":100.0,"real_pay":100.0,"pay_time":"2021-03-24 10:02:03","user_id":"0001","user_name":"Alice","area_id":"330106"}
|
|
|
|
{"order_id":"202103241606060001","order_channel":"appShop","order_time":"2021-03-24 16:06:06","pay_amount":200.0,"real_pay":180.0,"pay_time":"2021-03-24 16:10:06","user_id":"0001","user_name":"Alice","area_id":"330106"}</pre>
|
|
</p></li><li id="dli_08_0408__en-us_topic_0000001309855881_li4353143193117"><span>Perform the following operations to view the output:</span><p><ul id="dli_08_0408__en-us_topic_0000001309855881_ul173531443113114"><li id="dli_08_0408__en-us_topic_0000001309855881_li735314312316">Method 1: Locate the job and click <strong id="dli_08_0408__en-us_topic_0000001309855881_b350514134208">More</strong> > <strong id="dli_08_0408__en-us_topic_0000001309855881_b186591735192010">FlinkUI</strong>. Choose <strong id="dli_08_0408__en-us_topic_0000001309855881_b749153902010">Task Managers</strong> > <strong id="dli_08_0408__en-us_topic_0000001309855881_b3913357142015">Stdout</strong>.</li><li id="dli_08_0408__en-us_topic_0000001309855881_li33538437315">Method 2: If you allow DLI to save job logs in OBS, view the output in the <strong id="dli_08_0408__en-us_topic_0000001309855881_b493345910212">taskmanager.out</strong> file.<pre class="screen" id="dli_08_0408__en-us_topic_0000001309855881_screen399124820519">+I(202103241000000001,webShop,2021-03-2410:00:00,100.0,100.0,2021-03-2410:02:03,0001,Alice,330106)
|
|
+I(202103241606060001,appShop,2021-03-2416:06:06,200.0,180.0,2021-03-2416:10:06,0001,Alice,330106)</pre>
|
|
</li></ul>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0407.html">Format</a></div>
|
|
</div>
|
|
</div>
|
|
|