forked from docs/doc-exports
Hasko, Vladimir
cfc48b3aed
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-committed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
144 lines
37 KiB
HTML
144 lines
37 KiB
HTML
<a name="dli_09_0009"></a><a name="dli_09_0009"></a>
|
|
|
|
<h1 class="topictitle1">Reading Data from Kafka and Writing Data to RDS</h1>
|
|
<div id="body8662426"><div class="notice" id="dli_09_0009__en-us_topic_0000001318262117_note04712015123019"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="dli_09_0009__en-us_topic_0000001318262117_p15471191523013">This guide provides reference for Flink 1.12 only.</p>
|
|
</div></div>
|
|
<div class="section" id="dli_09_0009__en-us_topic_0000001318262117_section10920163411416"><h4 class="sectiontitle">Description</h4><p id="dli_09_0009__en-us_topic_0000001318262117_p128093392048">In this example, we aim to query information about top three most-clicked offerings in each hour from a set of real-time click data. Offerings' real-time click data will be sent to Kafka as the input source, and then the analysis result of Kafka data is to be output to RDS.</p>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p17817311367">For example, enter the following sample data:</p>
|
|
<pre class="screen" id="dli_09_0009__en-us_topic_0000001318262117_screen14982105910615">{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 08:01:00", "product_id":"0002", "product_name":"name1"}
|
|
{"user_id":"0002", "user_name":"Bob", "event_time":"2021-03-24 08:02:00", "product_id":"0002", "product_name":"name1"}
|
|
{"user_id":"0002", "user_name":"Bob", "event_time":"2021-03-24 08:06:00", "product_id":"0004", "product_name":"name2"}
|
|
{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 08:10:00", "product_id":"0003", "product_name":"name3"}
|
|
{"user_id":"0003", "user_name":"Cindy", "event_time":"2021-03-24 08:15:00", "product_id":"0005", "product_name":"name4"}
|
|
{"user_id":"0003", "user_name":"Cindy", "event_time":"2021-03-24 08:16:00", "product_id":"0005", "product_name":"name4"}
|
|
{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 08:56:00", "product_id":"0004", "product_name":"name2"}
|
|
{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 09:05:00", "product_id":"0005", "product_name":"name4"}
|
|
{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 09:10:00", "product_id":"0006", "product_name":"name5"}
|
|
{"user_id":"0002", "user_name":"Bob", "event_time":"2021-03-24 09:13:00", "product_id":"0006", "product_name":"name5"}</pre>
|
|
<div class="p" id="dli_09_0009__en-us_topic_0000001318262117_p13373161717716">Expected output:<pre class="screen" id="dli_09_0009__en-us_topic_0000001318262117_screen82511349972">2021-03-24 08:00:00 - 2021-03-24 08:59:59,0002,name1,2
|
|
2021-03-24 08:00:00 - 2021-03-24 08:59:59,0004,name2,2
|
|
2021-03-24 08:00:00 - 2021-03-24 08:59:59,0005,name4,2
|
|
2021-03-24 09:00:00 - 2021-03-24 09:59:59,0006,name5,2
|
|
2021-03-24 09:00:00 - 2021-03-24 09:59:59,0005,name4,1</pre>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_09_0009__en-us_topic_0000001318262117_section99670144912"><h4 class="sectiontitle">Prerequisites</h4><ol id="dli_09_0009__en-us_topic_0000001318262117_ol188213811126"><li id="dli_09_0009__en-us_topic_0000001318262117_li58321354155514">You have created a DMS for Kafka instance.<div class="caution" id="dli_09_0009__en-us_topic_0000001318262117_note144838216371"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><p id="dli_09_0009__en-us_topic_0000001318262117_p1483321113713">When you create the instance, do not enable <strong id="dli_09_0009__en-us_topic_0000001318262117_b11539164814224">Kafka SASL_SSL</strong>.</p>
|
|
</div></div>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li85801749103820">You have created an RDS for MySQL DB instance.<p id="dli_09_0009__en-us_topic_0000001318262117_p19557185015387"><a name="dli_09_0009__en-us_topic_0000001318262117_li85801749103820"></a><a name="en-us_topic_0000001318262117_li85801749103820"></a>In this example, the RDS for MySQL database version is 8.0.</p>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0009__en-us_topic_0000001318262117_section12587518310"><h4 class="sectiontitle">Overall Development Process</h4><div class="p" id="dli_09_0009__en-us_topic_0000001318262117_p7741285314">Overall Process<div class="fignone" id="dli_09_0009__en-us_topic_0000001318262117_fig1691441652"><span class="figcap"><b>Figure 1 </b>Job development process</span><br><span><img id="dli_09_0009__en-us_topic_0000001318262117_image691841454" src="en-us_image_0000001318102237.png"></span></div>
|
|
</div>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p16413127863"><a href="#dli_09_0009__en-us_topic_0000001318262117_section792923214216">Step 1: Create a Queue</a></p>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p0567192986"><a href="#dli_09_0009__en-us_topic_0000001318262117_section78516116518">Step 2: Create a Kafka Topic</a></p>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p1259352616818"><a href="#dli_09_0009__en-us_topic_0000001318262117_section1627154113018">Step 3: Create an RDS Database and Table</a></p>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p11783144819818"><a href="#dli_09_0009__en-us_topic_0000001318262117_section074025752119">Step 4: Create an Enhanced Datasource Connection</a></p>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p102618381914"><a href="#dli_09_0009__en-us_topic_0000001318262117_section12448959174212">Step 5: Run a Job</a></p>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p491452441010"><a href="#dli_09_0009__en-us_topic_0000001318262117_section4387527162418">Step 6: Send Data and Query Results</a></p>
|
|
</div>
|
|
<div class="section" id="dli_09_0009__en-us_topic_0000001318262117_section792923214216"><a name="dli_09_0009__en-us_topic_0000001318262117_section792923214216"></a><a name="en-us_topic_0000001318262117_section792923214216"></a><h4 class="sectiontitle">Step 1: Create a Queue</h4><ol id="dli_09_0009__en-us_topic_0000001318262117_ol193907145108"><li id="dli_09_0009__en-us_topic_0000001318262117_li3390161431020">Log in to the DLI console. In the navigation pane on the left, choose <strong id="dli_09_0009__en-us_topic_0000001318262117_b111463578328">Resources</strong> > <strong id="dli_09_0009__en-us_topic_0000001318262117_b0259106113316">Queue Management</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li5390171401016">On the displayed page, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b152514236337">Buy Queue</strong> in the upper right corner.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li133901314191011">On the <strong id="dli_09_0009__en-us_topic_0000001318262117_b69651138103312">Buy Queue</strong> page, set queue parameters as follows:<ul id="dli_09_0009__en-us_topic_0000001318262117_ul161521581016"><li id="dli_09_0009__en-us_topic_0000001318262117_li5992112615541"><strong id="dli_09_0009__en-us_topic_0000001318262117_b12726191202619">Billing Mode</strong>: . </li><li id="dli_09_0009__en-us_topic_0000001318262117_li17566251558"><strong id="dli_09_0009__en-us_topic_0000001318262117_b0251102482712">Region</strong> and <strong id="dli_09_0009__en-us_topic_0000001318262117_b2940228162712">Project</strong>: Retain the default values.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li9378124110319"><strong id="dli_09_0009__en-us_topic_0000001318262117_b9123113418279">Name</strong>: Enter a queue name.<div class="note" id="dli_09_0009__en-us_topic_0000001318262117_note1523218284569"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0009__en-us_topic_0000001318262117_en-us_topic_0069078607_en-us_topic_0069077926_p61185513">The queue name can contain only digits, letters, and underscores (_), but cannot contain only digits or start with an underscore (_). The name must contain 1 to 128 characters.</p>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p6288122915199"><strong id="dli_09_0009__en-us_topic_0000001318262117_b354813179295">The queue name is case-insensitive. Uppercase letters will be automatically converted to lowercase letters.</strong></p>
|
|
</div></div>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li1761814682119"><strong id="dli_09_0009__en-us_topic_0000001318262117_b54221238123018">Type</strong>: Select <strong id="dli_09_0009__en-us_topic_0000001318262117_b17317167315">For general purpose</strong>. Select the <strong id="dli_09_0009__en-us_topic_0000001318262117_b186804179336">Dedicated Resource Mode</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li360481635014"><strong id="dli_09_0009__en-us_topic_0000001318262117_b3858742103315">AZ Mode</strong> and <strong id="dli_09_0009__en-us_topic_0000001318262117_b32811437354">Specifications</strong>: Retain the default values.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li32421882515"><strong id="dli_09_0009__en-us_topic_0000001318262117_b178350339357">Enterprise Project</strong>: Select <strong id="dli_09_0009__en-us_topic_0000001318262117_b4891541173515">default</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li6862026155115"><strong id="dli_09_0009__en-us_topic_0000001318262117_b1945816485355">Advanced Settings</strong>: Select <strong id="dli_09_0009__en-us_topic_0000001318262117_b1676513547356">Custom</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li615215819015"><strong id="dli_09_0009__en-us_topic_0000001318262117_b204415915366">CIDR Block</strong>: Specify the queue network segment. For example, <strong id="dli_09_0009__en-us_topic_0000001318262117_b276513519361">10.0.0.0/16</strong>.<div class="caution" id="dli_09_0009__en-us_topic_0000001318262117_note243428112912"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><p id="dli_09_0009__en-us_topic_0000001318262117_p114344818296">The CIDR block of a queue cannot overlap with the CIDR blocks of DMS Kafka and RDS for MySQL DB instances. Otherwise, datasource connections will fail to be created.</p>
|
|
</div></div>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li1066019193813">Set other parameters as required.</li></ul>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li83901314181011">Click <strong id="dli_09_0009__en-us_topic_0000001318262117_b6383838163718">Buy</strong>. Confirm the configuration and click <strong id="dli_09_0009__en-us_topic_0000001318262117_b133841738173711">Submit</strong>.</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0009__en-us_topic_0000001318262117_section78516116518"><a name="dli_09_0009__en-us_topic_0000001318262117_section78516116518"></a><a name="en-us_topic_0000001318262117_section78516116518"></a><h4 class="sectiontitle">Step 2: Create a Kafka Topic</h4><ol id="dli_09_0009__en-us_topic_0000001318262117_ol68922017114"><li id="dli_09_0009__en-us_topic_0000001318262117_li48921006112">On the Kafka management console, click an instance name on the <strong id="dli_09_0009__en-us_topic_0000001318262117_b61998289391">DMS for Kafka</strong> page. Basic information of the Kafka instance is displayed.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li6892202113">Choose <strong id="dli_09_0009__en-us_topic_0000001318262117_b813214154118">Topics</strong>. On the displayed page, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b4926153010413">Create Topic</strong>. Configure the following parameters:<ul id="dli_09_0009__en-us_topic_0000001318262117_ul114131915412"><li id="dli_09_0009__en-us_topic_0000001318262117_li105301717185414">Topic Name For this example, enter <strong id="dli_09_0009__en-us_topic_0000001318262117_b9107103104215">testkafkatopic</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li20948104275411"><strong id="dli_09_0009__en-us_topic_0000001318262117_b2764164124219">Partitions</strong>: Set the value to <strong id="dli_09_0009__en-us_topic_0000001318262117_b1176444154217">1</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li15662134885416"><strong id="dli_09_0009__en-us_topic_0000001318262117_b172819487425">Replicas</strong>: Set the value to <strong id="dli_09_0009__en-us_topic_0000001318262117_b1856544215423">1</strong>.</li></ul>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p11749490558">Retain default values for other parameters.</p>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0009__en-us_topic_0000001318262117_section1627154113018"><a name="dli_09_0009__en-us_topic_0000001318262117_section1627154113018"></a><a name="en-us_topic_0000001318262117_section1627154113018"></a><h4 class="sectiontitle">Step 3: Create an RDS Database and Table</h4><ol id="dli_09_0009__en-us_topic_0000001318262117_ol622615816111"><li id="dli_09_0009__en-us_topic_0000001318262117_li16226198013">Log in to the RDS console. On the displayed page, locate the target MySQL DB instance and choose <strong id="dli_09_0009__en-us_topic_0000001318262117_b1971017183432">More</strong> > <strong id="dli_09_0009__en-us_topic_0000001318262117_b18710161814436">Log In</strong> in the <strong id="dli_09_0009__en-us_topic_0000001318262117_b271031874316">Operation</strong> column.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li1371112481236">On the displayed login dialog box, enter the username and password and click <strong id="dli_09_0009__en-us_topic_0000001318262117_b174219415439">Log In</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li2298131648">On the <strong id="dli_09_0009__en-us_topic_0000001318262117_b19852155704511">Databases</strong> page, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b4852145794515">Create Database</strong>. In the displayed dialog box, enter <strong id="dli_09_0009__en-us_topic_0000001318262117_b168524578457">testrdsdb</strong> as the database name and retain default values of rest parameters. Then, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b19852105717454">OK</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li75524472088">In the <strong id="dli_09_0009__en-us_topic_0000001318262117_b161311794614">Operation</strong> column of row where the created database locates, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b1114181754620">SQL Window</strong> and enter the following statement to create a table:<pre class="screen" id="dli_09_0009__en-us_topic_0000001318262117_screen15200175112817">CREATE TABLE clicktop (
|
|
`range_time` VARCHAR(64) NOT NULL,
|
|
`product_id` VARCHAR(32) NOT NULL,
|
|
`product_name` VARCHAR(32),
|
|
`event_count` VARCHAR(32),
|
|
PRIMARY KEY (`range_time`,`product_id`)
|
|
) ENGINE = InnoDB
|
|
DEFAULT CHARACTER SET = utf8mb4;</pre>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0009__en-us_topic_0000001318262117_section074025752119"><a name="dli_09_0009__en-us_topic_0000001318262117_section074025752119"></a><a name="en-us_topic_0000001318262117_section074025752119"></a><h4 class="sectiontitle">Step 4: Create an Enhanced Datasource Connection</h4><ul id="dli_09_0009__en-us_topic_0000001318262117_ul1231663714228"><li id="dli_09_0009__en-us_topic_0000001318262117_li193161137122213"><strong id="dli_09_0009__en-us_topic_0000001318262117_b667754142417">Connecting DLI to Kafka</strong><ol id="dli_09_0009__en-us_topic_0000001318262117_ol24611049949"><li id="dli_09_0009__en-us_topic_0000001318262117_li71971337017">On the Kafka management console, click an instance name on the <strong id="dli_09_0009__en-us_topic_0000001318262117_b97241144479">DMS for Kafka</strong> page. Basic information of the Kafka instance is displayed.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li19197133109">In the <strong id="dli_09_0009__en-us_topic_0000001318262117_b766232414711">Connection</strong> pane, obtain the <strong id="dli_09_0009__en-us_topic_0000001318262117_b366252410475">Instance Address (Private Network)</strong>. In the <strong id="dli_09_0009__en-us_topic_0000001318262117_b366242418471">Network</strong> pane, obtain the VPC and subnet of the instance.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li557658113910">Click the security group name in the <strong id="dli_09_0009__en-us_topic_0000001318262117_b7386164917177">Network</strong> pane. On the displayed page, click the <strong id="dli_09_0009__en-us_topic_0000001318262117_b1277862115205">Inbound Rules</strong> tab and add a rule to allow access from DLI queues. For example, if the CIDR block of the queue is 10.0.0.0/16, set <strong id="dli_09_0009__en-us_topic_0000001318262117_b11651754113014">Priority</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b6647155733014">1</strong>, <strong id="dli_09_0009__en-us_topic_0000001318262117_b11928187153112">Action</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b178375415313">Allow</strong>, <strong id="dli_09_0009__en-us_topic_0000001318262117_b13871181253116">Protocol</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b89278145318">TCP</strong>, <strong id="dli_09_0009__en-us_topic_0000001318262117_b134147238324">Type</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b11521274321">IPv4</strong>, <strong id="dli_09_0009__en-us_topic_0000001318262117_b11649192943218">Source</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b1888395517329">10.0.0.0/16</strong>, and click <strong id="dli_09_0009__en-us_topic_0000001318262117_b9683205743219">OK</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li10751165711389">Log in to the DLI management console. In the navigation pane on the left, choose <strong id="dli_09_0009__en-us_topic_0000001318262117_b1554934413334">Datasource Connections</strong>. On the displayed page, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b11549544123318">Create</strong> in the <strong id="dli_09_0009__en-us_topic_0000001318262117_b15505445337">Enhanced</strong> tab.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li10469451946">In the displayed dialog box, set the following parameters:<ul id="dli_09_0009__en-us_topic_0000001318262117_ul1032935016521"><li id="dli_09_0009__en-us_topic_0000001318262117_li1332914503526"><strong id="dli_09_0009__en-us_topic_0000001318262117_b4333867346">Connection Name</strong>: Enter a name for the enhanced datasource connection. For this example, enter <strong id="dli_09_0009__en-us_topic_0000001318262117_b65681920173414">dli_kafka</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li432905017524"><strong id="dli_09_0009__en-us_topic_0000001318262117_b8176121133515">Resource Pool</strong>: Select the name of the queue created in <a href="#dli_09_0009__en-us_topic_0000001318262117_section792923214216">Step 1: Create a Queue</a>. (Queues that are not added to a resource pool are displayed in this list.)</li><li id="dli_09_0009__en-us_topic_0000001318262117_li1932945012529"><strong id="dli_09_0009__en-us_topic_0000001318262117_b014273011351">VPC</strong>: Select the VPC of the Kafka instance.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li5329750115213"><strong id="dli_09_0009__en-us_topic_0000001318262117_b1554144853511">Subnet</strong>: Select the subnet of Kafka instance.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li16590071163">Set other parameters as you need.</li></ul>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p3174856105519">Click <strong id="dli_09_0009__en-us_topic_0000001318262117_b1468314157369">OK</strong>. Click the name of the created datasource connection to view its status. You can perform subsequent steps only after the connection status changes to <strong id="dli_09_0009__en-us_topic_0000001318262117_b6401619173619">Active</strong>.</p>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li18197531400">Choose <strong id="dli_09_0009__en-us_topic_0000001318262117_b189181243113620">Resources</strong> > <strong id="dli_09_0009__en-us_topic_0000001318262117_b7918164311361">Queue Management</strong> from the navigation pane, locate the queue you created in <a href="#dli_09_0009__en-us_topic_0000001318262117_section792923214216">Step 1: Create a Queue</a>. In the <strong id="dli_09_0009__en-us_topic_0000001318262117_b091884393612">Operation</strong> column, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b1791804353618">More</strong> > <strong id="dli_09_0009__en-us_topic_0000001318262117_b1391819434367">Test Address Connectivity</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li171971831506">In the displayed dialog box, enter <em id="dli_09_0009__en-us_topic_0000001318262117_i1980113603819">Kafka instance address (private network)</em><strong id="dli_09_0009__en-us_topic_0000001318262117_b38025643818">:</strong><em id="dli_09_0009__en-us_topic_0000001318262117_i1680246173820">port</em> in the <strong id="dli_09_0009__en-us_topic_0000001318262117_b68021617388">Address</strong> box and click <strong id="dli_09_0009__en-us_topic_0000001318262117_b88021460380">Test</strong> to check whether the instance is reachable.</li></ol>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li474495213221"><strong id="dli_09_0009__en-us_topic_0000001318262117_b18110175854320">Connecting DLI to RDS</strong><ol id="dli_09_0009__en-us_topic_0000001318262117_ol17815135442310"><li id="dli_09_0009__en-us_topic_0000001318262117_li14815454112312">Go to the RDS console, click the name of the target RDS DB instance on the <strong id="dli_09_0009__en-us_topic_0000001318262117_b112076520455">Instances</strong> page. Basic information of the instance is displayed.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li19666016361"><a name="dli_09_0009__en-us_topic_0000001318262117_li19666016361"></a><a name="en-us_topic_0000001318262117_li19666016361"></a>In the <strong id="dli_09_0009__en-us_topic_0000001318262117_b129616261456">Connection Information</strong> pane, obtain the floating IP address, database port, VPC, and subnet.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li4233434123610">Click the security group name. On the displayed page, click the <strong id="dli_09_0009__en-us_topic_0000001318262117_b11950162294614">Inbound Rules</strong> tab and add a rule to allow access from DLI queues. For example, if the CIDR block of the queue is 10.0.0.0/16, set <strong id="dli_09_0009__en-us_topic_0000001318262117_b51971344164619">Priority</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b1519814414616">1</strong>, <strong id="dli_09_0009__en-us_topic_0000001318262117_b2198184411460">Action</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b14198104413466">Allow</strong>, <strong id="dli_09_0009__en-us_topic_0000001318262117_b1619816449468">Protocol</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b519894414612">TCP</strong>, <strong id="dli_09_0009__en-us_topic_0000001318262117_b1719817441462">Type</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b171999443461">IPv4</strong>, <strong id="dli_09_0009__en-us_topic_0000001318262117_b11199114420460">Source</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b01990441467">10.0.0.0/16</strong>, and click <strong id="dli_09_0009__en-us_topic_0000001318262117_b15199344204618">OK</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li14803182620216">Check whether the Kafka instance and RDS DB instance are in the same VPC and subnet.<ol type="a" id="dli_09_0009__en-us_topic_0000001318262117_ol184431423137"><li id="dli_09_0009__en-us_topic_0000001318262117_li957111932">If they are, go to <a href="#dli_09_0009__en-us_topic_0000001318262117_li9816175412318">7</a>. You do not need to create an enhanced datasource connection again.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li1086652519411">If they are not, go to <a href="#dli_09_0009__en-us_topic_0000001318262117_li11976319011">5</a>. Create an enhanced datasource connection to connect DLI to the subnet where the RDS DB instance locates.</li></ol>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li11976319011"><a name="dli_09_0009__en-us_topic_0000001318262117_li11976319011"></a><a name="en-us_topic_0000001318262117_li11976319011"></a>Log in to the DLI management console. In the navigation pane on the left, choose <strong id="dli_09_0009__en-us_topic_0000001318262117_b59359925419">Datasource Connections</strong>. On the displayed page, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b129356935418">Create</strong> in the <strong id="dli_09_0009__en-us_topic_0000001318262117_b49351912546">Enhanced</strong> tab.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li198151354192319">In the displayed dialog box, set the following parameters:<ul id="dli_09_0009__en-us_topic_0000001318262117_ul17815125415233"><li id="dli_09_0009__en-us_topic_0000001318262117_li1181518543233"><strong id="dli_09_0009__en-us_topic_0000001318262117_b12215344547">Connection Name</strong>: Enter a name of the enhanced datasource connection For this example, enter <strong id="dli_09_0009__en-us_topic_0000001318262117_b168953399544">dli_rds</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li681518542232"><strong id="dli_09_0009__en-us_topic_0000001318262117_b414414546491">Resource Pool</strong>: Select the name of the queue created in <a href="#dli_09_0009__en-us_topic_0000001318262117_section792923214216">Step 1: Create a Queue</a>. (Queues that are not added to a resource pool are displayed in this list.)</li><li id="dli_09_0009__en-us_topic_0000001318262117_li58162542235"><strong id="dli_09_0009__en-us_topic_0000001318262117_b19141353125416">VPC</strong>: Select the VPC of the RDS DB instance.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li11816105432317"><strong id="dli_09_0009__en-us_topic_0000001318262117_b146401218105514">Subnet</strong>: Select the subnet of RDS DB instance.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li781675472310">Set other parameters as you need.</li></ul>
|
|
<p id="dli_09_0009__en-us_topic_0000001318262117_p1081617549235">Click <strong id="dli_09_0009__en-us_topic_0000001318262117_b468082725511">OK</strong>. Click the name of the created datasource connection to view its status. You can perform subsequent steps only after the connection status changes to <strong id="dli_09_0009__en-us_topic_0000001318262117_b42101229115519">Active</strong>.</p>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li9816175412318"><a name="dli_09_0009__en-us_topic_0000001318262117_li9816175412318"></a><a name="en-us_topic_0000001318262117_li9816175412318"></a>Choose <strong id="dli_09_0009__en-us_topic_0000001318262117_b1629463120555">Resources</strong> > <strong id="dli_09_0009__en-us_topic_0000001318262117_b9294113116553">Queue Management</strong> from the navigation pane, locate the queue you created in <a href="#dli_09_0009__en-us_topic_0000001318262117_section792923214216">Step 1: Create a Queue</a>. In the <strong id="dli_09_0009__en-us_topic_0000001318262117_b62941313555">Operation</strong> column, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b15294133119559">More</strong> > <strong id="dli_09_0009__en-us_topic_0000001318262117_b142944317555">Test Address Connectivity</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li7816454162319">In the displayed dialog box, enter <em id="dli_09_0009__en-us_topic_0000001318262117_i202134135515">floating IP address</em><strong id="dli_09_0009__en-us_topic_0000001318262117_b10218345557">:</strong><em id="dli_09_0009__en-us_topic_0000001318262117_i1321634155515">database port</em> of the RDS DB instance you have obtained in <a href="#dli_09_0009__en-us_topic_0000001318262117_li19666016361">2</a> in the <strong id="dli_09_0009__en-us_topic_0000001318262117_b53103445519">Address</strong> box and click <strong id="dli_09_0009__en-us_topic_0000001318262117_b63143414554">Test</strong> to check whether the database is reachable.</li></ol>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_09_0009__en-us_topic_0000001318262117_section12448959174212"><a name="dli_09_0009__en-us_topic_0000001318262117_section12448959174212"></a><a name="en-us_topic_0000001318262117_section12448959174212"></a><h4 class="sectiontitle">Step 5: Run a Job</h4><ol id="dli_09_0009__en-us_topic_0000001318262117_ol1313811362437"><li id="dli_09_0009__en-us_topic_0000001318262117_li219661215114">On the DLI management console, choose <strong id="dli_09_0009__en-us_topic_0000001318262117_b18974175112564">Job Management</strong> > <strong id="dli_09_0009__en-us_topic_0000001318262117_b6974195111569">Flink Jobs</strong>. On the <strong id="dli_09_0009__en-us_topic_0000001318262117_b1497465117563">Flink Jobs</strong> page, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b17974551155615">Create Job</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li18197181225114">In the <strong id="dli_09_0009__en-us_topic_0000001318262117_b066914214577">Create Job</strong> dialog box, set <strong id="dli_09_0009__en-us_topic_0000001318262117_b166911275719">Type</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b76704219570">Flink OpenSource SQL</strong> and <strong id="dli_09_0009__en-us_topic_0000001318262117_b1767012215572">Name</strong> to <strong id="dli_09_0009__en-us_topic_0000001318262117_b106701211579">FlinkKafkaRds</strong>. Click <strong id="dli_09_0009__en-us_topic_0000001318262117_b1445141435815">OK</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li119731295112">On the job editing page, set the following parameters and retain the default values of other parameters.<ul id="dli_09_0009__en-us_topic_0000001318262117_ul1970291612112"><li id="dli_09_0009__en-us_topic_0000001318262117_li6702216152118"><strong id="dli_09_0009__en-us_topic_0000001318262117_b721163114590">Queue</strong>: Select the queue created in <a href="#dli_09_0009__en-us_topic_0000001318262117_section792923214216">Step 1: Create a Queue</a>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li1629221563615"><strong id="dli_09_0009__en-us_topic_0000001318262117_b68931629105920">Flink Version</strong>: Select <strong id="dli_09_0009__en-us_topic_0000001318262117_b14927402593">1.12</strong>.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li84401118192110"><strong id="dli_09_0009__en-us_topic_0000001318262117_b10532241115912">Save Job Log</strong>: Enable this function.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li12684193718212"><strong id="dli_09_0009__en-us_topic_0000001318262117_b1386020122014">OBS Bucket</strong>: Select an OBS bucket for storing job logs and grant access permissions of the OBS bucket as prompted.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li1382275713215"><strong id="dli_09_0009__en-us_topic_0000001318262117_b322620281701">Enable Checkpointing</strong>: Enable this function.</li><li id="dli_09_0009__en-us_topic_0000001318262117_li1479717252318">Enter a SQL statement in the editing pane. The following is an example. Modify the parameters in bold as you need.<div class="note" id="dli_09_0009__en-us_topic_0000001318262117_note16434951143717"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0009__en-us_topic_0000001318262117_p3472334185">In this example, the syntax version of Flink OpenSource SQL is 1.12. In this example, the data source is Kafka and the result data is written to RDS.</p>
|
|
</div></div>
|
|
<pre class="screen" id="dli_09_0009__en-us_topic_0000001318262117_screen453112325017">create table click_product(
|
|
user_id string, --ID of the user
|
|
user_name string, --Username
|
|
event_time string, --Click time
|
|
product_id string, --Offering ID
|
|
product_name string --Offering name
|
|
) with (
|
|
"connector" = "kafka",
|
|
"properties.bootstrap.servers" = " <em id="dli_09_0009__en-us_topic_0000001318262117_i1222631295017"><strong id="dli_09_0009__en-us_topic_0000001318262117_b179264118503">10.128.0.120:9092,10.128.0.89:9092,10.128.0.83:9092</strong></em> ",-- Internal network address and port number of the Kafka instance
|
|
"properties.group.id" = "click",
|
|
"topic" = " <strong id="dli_09_0009__en-us_topic_0000001318262117_b1629671195414">testkafkatopic</strong> ",--Name of the created Kafka topic
|
|
"format" = "json",
|
|
"scan.startup.mode" = "latest-offset"
|
|
);
|
|
|
|
--Result table
|
|
create table top_product (
|
|
range_time string, --Calculated time range
|
|
product_id string, --Offering ID
|
|
product_name string --Offering name
|
|
event_count bigint, --Number of clicks
|
|
primary key (range_time, product_id) not enforced
|
|
) with (
|
|
"connector" = "jdbc",
|
|
"url" = "jdbc:mysql://<em id="dli_09_0009__en-us_topic_0000001318262117_i16452122735019"><strong id="dli_09_0009__en-us_topic_0000001318262117_b44331926165019">192.168.12.148:3306</strong></em>/<strong id="dli_09_0009__en-us_topic_0000001318262117_b1480315413545">testrdsdb</strong> ",--testrdsdb indicates the name of the created RDS database. Replace the IP address and port number with those of the RDS DB instance.
|
|
"table-name" = "clicktop",
|
|
"pwd_auth_name"="<em id="dli_09_0009__en-us_topic_0000001318262117_i41515154612"><strong id="dli_09_0009__en-us_topic_0000001318262117_b81518164610">xxxxx</strong></em>", -- Name of the datasource authentication of the password type created on DLI. If datasource authentication is used, you do not need to set the username and password for the job.
|
|
"sink.buffer-flush.max-rows" = "1000",
|
|
"sink.buffer-flush.interval" = "1s"
|
|
);
|
|
|
|
create view current_event_view
|
|
as
|
|
select product_id, product_name, count(1) as click_count, concat(substring(event_time, 1, 13), ":00:00") as min_event_time, concat(substring(event_time, 1, 13), ":59:59") as max_event_time
|
|
from click_product group by substring (event_time, 1, 13), product_id, product_name;
|
|
|
|
insert into top_product
|
|
select
|
|
concat(min_event_time, " - ", max_event_time) as range_time,
|
|
product_id,
|
|
product_name,
|
|
click_count
|
|
from (
|
|
select *,
|
|
row_number() over (partition by min_event_time order by click_count desc) as row_num
|
|
from current_event_view
|
|
)
|
|
where row_num <= 3</pre>
|
|
</li></ul>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li219761217517">Click <strong id="dli_09_0009__en-us_topic_0000001318262117_b54225117111">Check Semantic</strong> and ensure that the SQL statement passes the check. Click <strong id="dli_09_0009__en-us_topic_0000001318262117_b24449631113">Save</strong>. Click <strong id="dli_09_0009__en-us_topic_0000001318262117_b139921810181112">Start</strong>, confirm the job parameters, and click <strong id="dli_09_0009__en-us_topic_0000001318262117_b999231016117">Start Now</strong> to execute the job. Wait until the job status changes to <strong id="dli_09_0009__en-us_topic_0000001318262117_b161304316126">Running</strong>.</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0009__en-us_topic_0000001318262117_section4387527162418"><a name="dli_09_0009__en-us_topic_0000001318262117_section4387527162418"></a><a name="en-us_topic_0000001318262117_section4387527162418"></a><h4 class="sectiontitle">Step 6: Send Data and Query Results</h4><ol id="dli_09_0009__en-us_topic_0000001318262117_ol0558165272410"><li id="dli_09_0009__en-us_topic_0000001318262117_li1942135894212">Use the Kafka client to send data to topics created in <a href="#dli_09_0009__en-us_topic_0000001318262117_section78516116518">Step 2: Create a Kafka Topic</a> to simulate real-time data streams.<p id="dli_09_0009__en-us_topic_0000001318262117_p34915378425">The sample data is as follows:</p>
|
|
<pre class="screen" id="dli_09_0009__en-us_topic_0000001318262117_screen114918373424">{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 08:01:00", "product_id":"0002", "product_name":"name1"}
|
|
{"user_id":"0002", "user_name":"Bob", "event_time":"2021-03-24 08:02:00", "product_id":"0002", "product_name":"name1"}
|
|
{"user_id":"0002", "user_name":"Bob", "event_time":"2021-03-24 08:06:00", "product_id":"0004", "product_name":"name2"}
|
|
{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 08:10:00", "product_id":"0003", "product_name":"name3"}
|
|
{"user_id":"0003", "user_name":"Cindy", "event_time":"2021-03-24 08:15:00", "product_id":"0005", "product_name":"name4"}
|
|
{"user_id":"0003", "user_name":"Cindy", "event_time":"2021-03-24 08:16:00", "product_id":"0005", "product_name":"name4"}
|
|
{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 08:56:00", "product_id":"0004", "product_name":"name2"}
|
|
{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 09:05:00", "product_id":"0005", "product_name":"name4"}
|
|
{"user_id":"0001", "user_name":"Alice", "event_time":"2021-03-24 09:10:00", "product_id":"0006", "product_name":"name5"}
|
|
{"user_id":"0002", "user_name":"Bob", "event_time":"2021-03-24 09:13:00", "product_id":"0006", "product_name":"name5"}</pre>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li71634315812">Log in to the RDS console, click the name of the RDS DB instance. On the displayed page, click the name of the created database, for example, <strong id="dli_09_0009__en-us_topic_0000001318262117_b177851950142617">testrdsdb</strong>, and click <strong id="dli_09_0009__en-us_topic_0000001318262117_b4791185042612">Query SQL Statements</strong> in the <strong id="dli_09_0009__en-us_topic_0000001318262117_b12791185082614">Operation</strong> column of the row that containing the <strong id="dli_09_0009__en-us_topic_0000001318262117_b16791550192614">clicktop</strong> table.<pre class="screen" id="dli_09_0009__en-us_topic_0000001318262117_screen1035620187317">select * from `clicktop`;</pre>
|
|
</li><li id="dli_09_0009__en-us_topic_0000001318262117_li930754410587">On the displayed page, click <strong id="dli_09_0009__en-us_topic_0000001318262117_b159501234162718">Execute SQL</strong>. Check whether data has been written into the RDS table.</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0006.html">Flink OpenSource SQL Jobs</a></div>
|
|
</div>
|
|
</div>
|
|
|