yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
a41a4e0331fbc0beff63bb28fa59228cc9a751df
doc-exports/docs/dws/dev/dws_04_1212.html
luhuayi 177cd61a57 DWS DEVG 910.211 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: luhuayi <luhuayi@huawei.com>
Co-committed-by: luhuayi <luhuayi@huawei.com>
2025-05-05 07:44:03 +00:00

499 lines
65 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<a name="EN-US_TOPIC_0000001845482249"></a><a name="EN-US_TOPIC_0000001845482249"></a>
<h1 class="topictitle1">Real-Time Binlog Consumption by Flink</h1>
<div id="body8662426"><div class="section" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_section1016613565439"><h4 class="sectiontitle">Precautions</h4><ul id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_ul141142080441"><li id="EN-US_TOPIC_0000001845482249__li1941727192710">Currently, only 9.1.0 and later versions support HStore and HStore Opt tables to record binlogs.</li><li id="EN-US_TOPIC_0000001845482249__li5726102762513">V3 HStore tables do not support binlogs, whereas only V3 HStore Opt tables offer binlog support. V3 is currently in trial commercial use and should be thoroughly evaluated before deployment.</li><li id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_li18114188144419">The Binlog function is only supported for HStore and HStore Opt tables in GaussDB(DWS). These tables must have primary keys and one of parameters <strong id="EN-US_TOPIC_0000001845482249__b1017217424269">enable_binlog</strong> and <strong id="EN-US_TOPIC_0000001845482249__b14172144262610">enable_binlog_timestamp</strong> must be set to <strong id="EN-US_TOPIC_0000001845482249__b11721542172610">on</strong>.</li><li id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_li1412512154417">The name of the consumed binlog table cannot contain special characters, such as periods (.) and double quotation marks (").</li><li id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_li7842717134418">If multiple tasks consume binlog data of a single table, ensure that <strong id="EN-US_TOPIC_0000001845482249__b149807548753217">binlogSlotName</strong> of each task is unique.</li><li id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_li663922213446">For maximum consumption speed, match task concurrency with the number of DNs in your GaussDB(DWS) cluster.</li><li id="EN-US_TOPIC_0000001845482249__li1968895711279">If you use the sink capability of "dws-connector-flink" in <em id="EN-US_TOPIC_0000001845482249__i122420035412">Data Warehouse Service (DWS) Tool Guide</em> to write binlog data, pay attention to the following:<ul id="EN-US_TOPIC_0000001845482249__ul193191810283"><li id="EN-US_TOPIC_0000001845482249__li1593625192813">To ensure the data write sequence on DNs, set <strong id="EN-US_TOPIC_0000001845482249__b190055213549">connectionSize</strong> to <strong id="EN-US_TOPIC_0000001845482249__b129801054175419">1</strong>.</li><li id="EN-US_TOPIC_0000001845482249__li99741353112617">If the primary key is updated on the source end or Flink is required for aggregation calculation, set <strong id="EN-US_TOPIC_0000001845482249__b579571355513">ignoreUpdateBefore</strong> to <strong id="EN-US_TOPIC_0000001845482249__b7410101515520">false</strong>. Otherwise, you are not advised to set <strong id="EN-US_TOPIC_0000001845482249__b568420211558">ignoreUpdateBefore</strong> to <strong id="EN-US_TOPIC_0000001845482249__b1697182414552">false</strong> (the default value is <strong id="EN-US_TOPIC_0000001845482249__b125822025205517">true</strong>).</li></ul>
</li></ul>
</div>
<div class="section" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_section5329125611449"><h4 class="sectiontitle">Real-Time Binlog Consumption by Flink</h4><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p14219144042813">Use DWS Connector to consume binlogs in real time. For details, see "DWS-Connector" in <em id="EN-US_TOPIC_0000001845482249__i12788203225514">Data Warehouse Service (DWS) Tool Guide</em>.</p>
<p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p16677194517">If full data has been synchronized to the target end using other synchronization tools, and only incremental synchronization is required, you can call the following system function to update the synchronization points.</p>
</div>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_screen783964404211"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">pg_catalog</span><span class="p">.</span><span class="n">pgxc_register_full_sync_point</span><span class="p">(</span><span class="s1">'table_name'</span><span class="p">,</span><span class="w"> </span><span class="s1">'slot_name'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<div class="section" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_section9680558195312"><h4 class="sectiontitle">Source Table DDL</h4><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p162374116548">The source autonomously assigns the appropriate Flink RowKind type (INSERT, DELETE, UPDATE_BEFORE, or UPDATE_AFTER) to each data row based on the operation type. This mechanism facilitates the synchronization of table data in a mirrored way, akin to the Change Data Capture (CDC) feature in MySQL and PostgreSQL databases.</p>
</div>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_screen4841644154218"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_binlog_source</span><span class="w"> </span><span class="p">(</span><span class="w"> </span>
<span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="k">c</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="k">primary</span><span class="w"> </span><span class="k">key</span><span class="p">(</span><span class="n">a</span><span class="p">)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">ENFORCED</span>
<span class="p">)</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'dws'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'url'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'jdbc:gaussdb://ip:port/gaussdb'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'binlog'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'tableName'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'test_binlog_source'</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="s1">'binlogSlotName'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'slot'</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="s1">'username'</span><span class="o">=</span><span class="s1">'xxx'</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="s1">'password'</span><span class="o">=</span><span class="s1">'xxx'</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
<div class="section" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_section15883153653314"><h4 class="sectiontitle">Binlog Parameters</h4><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p101118167405">The following table describes the parameters involved in binlog consumption.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_table47081931183511" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row57088314352"><th align="left" class="cellrowborder" valign="top" width="26.810000000000002%" id="mcps1.3.6.3.2.5.1.1"><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p697164213356">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="43.04%" id="mcps1.3.6.3.2.5.1.2"><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p199711242173515">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="15.27%" id="mcps1.3.6.3.2.5.1.3"><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p662854710276">Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="14.879999999999999%" id="mcps1.3.6.3.2.5.1.4"><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p17971104253518">Default Value</p>
</th>
</tr>
</thead>
<tbody><tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row2070916312352"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p2097164210355">binlog</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p179711425351">Specifies whether to read binlog information.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p16295472278">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p59711242103511">false</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row1570993116354"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p169711042113512">binlogSlotName</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p169710426359">Slot, which serves as an identifier. Multiple Flink tasks can simultaneously consume binlog data of the same table, so each task's <strong id="EN-US_TOPIC_0000001845482249__b5150126653217">binlogSlotName</strong> must be unique.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p46291247172714">String</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p2097234233516">Name of the Flink mapping table</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row27091031183519"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p42808118367">binlogBatchReadSize</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1828091113619">Rows of binlog data read in batches.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p166296474272">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p8280512367">5000</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row1159283511349"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p10593163511345">fullSyncBinlogBatchReadSize</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1840964383510">Rows of binlog data fully read.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p12593435183417">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p459323518346">50000</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row147109319352"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p328151103612">binlogReadTimeout</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p328112123618">Timeout for incrementally consuming binlog data, in milliseconds.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p462914752714">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1928115113362">600000</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row15710153116355"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p528191203610">fullSyncBinlogReadTimeout</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p12810103610">Timeout for fully consuming binlog data, in milliseconds.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1462914792710">Long</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p128112113360">1800000</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row77111631173516"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1628111118364">binlogSleepTime</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p3281215365">Sleep duration when no real-time binlog data is consumed, in milliseconds. The sleep duration with consecutive read failures is <strong id="EN-US_TOPIC_0000001845482249__b24801040132">binlogSleepTime</strong> * failures, up to <strong id="EN-US_TOPIC_0000001845482249__b187550535253217">binlogMaxSleepTime</strong>. The value is reset after successful data read.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p15629124714277">Long</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p11282131123620">500</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row207114312352"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p4282141203616">binlogMaxSleepTime</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p11282817362">Maximum sleep duration when no real-time binlog data is consumed, in milliseconds.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p106291347162717">Long</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p11282161123612">10000</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row6711183153519"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1928215123614">binlogMaxRetryTimes</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p328316163617">Maximum number of retries after a binlog data consumption error.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p15918935183620">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p7283131203619">1</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row10712231123520"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p126212515380">binlogRetryInterval</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p162628515382">Interval between retries after a binlog data consumption error, in milliseconds. Sleep duration during retry, which is calculated as <strong id="EN-US_TOPIC_0000001845482249__b162721301149">binlogRetryInterval</strong> * (1~<strong id="EN-US_TOPIC_0000001845482249__b9988231541">binlogMaxRetryTimes</strong>) + Random(100). The unit is millisecond.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p463024712714">Long</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1626317503813">100</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row65821528153617"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1926355163815">binlogParallelNum</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1626319593816">Number of threads for consuming binlog data. This parameter is valid only when task concurrency is less than the number of DNs in the GaussDB(DWS) cluster.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1610510385364">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p926312533817">3</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_row68741035163620"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p02632533814">connectionPoolSize</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p52631451385">Number of connections in the JDBC connection pool.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p117911739193614">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p7263105143810">5</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__row11589131817298"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__p155892018182917">needRedistribution</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__p55896189293">Determines compatibility with expansion redistribution. To ensure compatibility, upgrade the kernel to the corresponding version. If the kernel is an older version, set this parameter to <strong id="EN-US_TOPIC_0000001845482249__b183822521815">false</strong>. If set to <strong id="EN-US_TOPIC_0000001845482249__b785705516812">true</strong>, <strong id="EN-US_TOPIC_0000001845482249__b5501201591">restart-strategy</strong> of Flink cannot be set to <strong id="EN-US_TOPIC_0000001845482249__b16906526919">none</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__p758918184294">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__p5589131802919">true</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__row164332242911"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__p1364362262918">newSystemValue</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__p19643112212292">Indicates whether to use the new system field when reading binlog data. (The kernel needs to be upgraded to the corresponding version. If the kernel is an older version, set this parameter to <strong id="EN-US_TOPIC_0000001845482249__b1424142618914">false</strong>.)</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__p2064372211296">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__p864332262916">true</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__row1995220265292"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__p169521326152919">checkNodeChangeInterval</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__p59521726182916">Interval for detecting node changes. This parameter is valid only when <strong id="EN-US_TOPIC_0000001845482249__b19454134320914">needRedistribution</strong> is set to <strong id="EN-US_TOPIC_0000001845482249__b1070710443911">true</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__p49521626112910">Long</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__p17952162692916">10000</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__row12285193112914"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__p10285931122915">connectionSocketTimeout</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__p17285123112918">Timeout interval for connection processing, in milliseconds. It can also be considered as the timeout interval for executing SQL statements on the client. The default value is <strong id="EN-US_TOPIC_0000001845482249__b9806193011015">0</strong>, which means that the timeout interval is not set.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__p6285431162910">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__p13285163192910">0</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__row1320814474408"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__p820874720408">binlogIgnoreUpdateBefore</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__p13208547194018">Determines whether to filter out <strong id="EN-US_TOPIC_0000001845482249__b9287204394012">before_update</strong> records in binlogs and whether to return only primary key information for <strong id="EN-US_TOPIC_0000001845482249__b1623355124517">delete</strong> records. This parameter is supported only in 9.1.0.200 and later versions.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__p1820814784017">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__p320814710406">false</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__row2086481116416"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__p158642115415">binlogStartTime</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__p286521104111">Sets the time point from which binlogs are consumed can be set using the format <strong id="EN-US_TOPIC_0000001845482249__b15537174334116">yyyy-MM-dd hh:mm:ss</strong>. <strong id="EN-US_TOPIC_0000001845482249__b13644101444619"><a href="dws_04_1029.html#EN-US_TOPIC_0000001764650772__li125481342133319">enable_binlog_timestamp</a></strong> must be enabled for the table.</p>
<p id="EN-US_TOPIC_0000001845482249__p390274732913">This parameter is supported only in 9.1.0.200 and later versions.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__p1986511134110">String</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__p1865151164118">N/A</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001845482249__row14145172912417"><td class="cellrowborder" valign="top" width="26.810000000000002%" headers="mcps1.3.6.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001845482249__p1714515297418">binlogSyncPointSize</p>
</td>
<td class="cellrowborder" valign="top" width="43.04%" headers="mcps1.3.6.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001845482249__p41451929174119">Specifies the size of the synchronization point range for incrementally reading binlogs. This can control data flushing if the data volume is too large.</p>
<p id="EN-US_TOPIC_0000001845482249__p1209450172919">This parameter is supported only in 9.1.0.200 and later versions.</p>
</td>
<td class="cellrowborder" valign="top" width="15.27%" headers="mcps1.3.6.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001845482249__p1145162912412">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="14.879999999999999%" headers="mcps1.3.6.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001845482249__p114562913416">5000</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_section18896171812343"><h4 class="sectiontitle">Data Synchronization Example</h4><ul id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_ul128662934219"><li id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_li0287182914210">On GaussDB(DWS):<div class="note" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_note6242193704410"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p5242113734415">When creating a binlog table, set <a href="dws_04_0936.html#EN-US_TOPIC_0000001764491796__section19418514112914">enable_hstore_binlog_table</a> to true. You can run the <strong id="EN-US_TOPIC_0000001845482249__b104271747256">show enable_hstore_binlog_table</strong> command to query the binlog table.</p>
</div></div>
<p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p1653919581470">-- Source table (generating binlogs)</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_screen080063915714"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_binlog_source</span><span class="p">(</span><span class="n">a</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span><span class="w"> </span><span class="k">c</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span><span class="w"> </span><span class="k">primary</span><span class="w"> </span><span class="k">key</span><span class="p">(</span><span class="n">a</span><span class="p">))</span><span class="w"> </span><span class="k">with</span><span class="p">(</span><span class="n">orientation</span><span class="o">=</span><span class="k">column</span><span class="p">,</span><span class="w"> </span><span class="n">enable_hstore_opt</span><span class="o">=</span><span class="k">on</span><span class="p">,</span><span class="w"> </span><span class="n">enable_binlog</span><span class="o">=</span><span class="k">true</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p55391258164720">-- Target table</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_screen106006492574"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_binlog_sink</span><span class="p">(</span><span class="n">a</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span><span class="w"> </span><span class="k">c</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span><span class="w"> </span><span class="k">primary</span><span class="w"> </span><span class="k">key</span><span class="p">(</span><span class="n">a</span><span class="p">))</span><span class="w"> </span><span class="k">with</span><span class="p">(</span><span class="n">orientation</span><span class="o">=</span><span class="k">column</span><span class="p">,</span><span class="w"> </span><span class="n">enable_hstore_opt</span><span class="o">=</span><span class="k">on</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_li125901820104814">On Flink:<p id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_p185491243164517"><a name="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_li125901820104814"></a><a name="en-us_topic_0000001795377406_li125901820104814"></a>Run the following commands to perform complete data synchronization:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001845482249__en-us_topic_0000001795377406_screen1784634411421"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span></pre></div></td><td class="code"><div><pre><span></span><span class="c1">-- Create a mapping table for the source table.</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_binlog_source</span><span class="w"> </span><span class="p">(</span><span class="w"> </span>
<span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="k">c</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="k">primary</span><span class="w"> </span><span class="k">key</span><span class="p">(</span><span class="n">a</span><span class="p">)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">ENFORCED</span>
<span class="p">)</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'dws'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'url'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'jdbc:gaussdb://ip:port/gaussdb'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'binlog'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'tableName'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'test_binlog_source'</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="s1">'binlogSlotName'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'slot'</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="s1">'username'</span><span class="o">=</span><span class="s1">'xxx'</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="s1">'password'</span><span class="o">=</span><span class="s1">'xxx'</span><span class="p">);</span>
<span class="w"> </span>
<span class="c1">-- Create a mapping table for the target table:</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_binlog_sink</span><span class="w"> </span><span class="p">(</span><span class="w"> </span>
<span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="k">c</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="k">primary</span><span class="w"> </span><span class="k">key</span><span class="p">(</span><span class="n">a</span><span class="p">)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">ENFORCED</span>
<span class="p">)</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'dws'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'url'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'jdbc:gaussdb://ip:port/gaussdb'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'tableName'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'test_binlog_sink'</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="s1">'ignoreUpdateBefore'</span><span class="o">=</span><span class="s1">'false'</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="s1">'connectionSize'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'1'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'username'</span><span class="o">=</span><span class="s1">'xxx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'password'</span><span class="o">=</span><span class="s1">'xxx'</span><span class="p">);</span>
<span class="err">​</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_binlog_sink</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">test_binlog_source</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</div>
<div class="section" id="EN-US_TOPIC_0000001845482249__section184520010440"><h4 class="sectiontitle">Example of Using Java Programs</h4><p id="EN-US_TOPIC_0000001845482249__p35143312443">Create a source table and a target table.</p>
</div>
<div class="codecoloring" codetype="Java" id="EN-US_TOPIC_0000001845482249__screen181809519444"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="o">--</span><span class="w"> </span><span class="n">source</span>
<span class="n">create</span><span class="w"> </span><span class="n">table</span><span class="w"> </span><span class="nf">binlog_test_source</span><span class="p">(</span><span class="n">a</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">primary</span><span class="w"> </span><span class="nf">key</span><span class="p">(</span><span class="n">a</span><span class="p">))</span><span class="w"> </span><span class="n">with</span><span class="p">(</span><span class="n">orientation</span><span class="o">=</span><span class="n">column</span><span class="p">,</span><span class="w"> </span><span class="n">enable_hstore_opt</span><span class="o">=</span><span class="n">on</span><span class="p">,</span><span class="w"> </span><span class="n">enable_binlog</span><span class="o">=</span><span class="kc">true</span><span class="p">);</span>
<span class="o">--</span><span class="w"> </span><span class="n">sink</span>
<span class="n">create</span><span class="w"> </span><span class="n">table</span><span class="w"> </span><span class="nf">binlog_test_sink</span><span class="p">(</span><span class="n">a</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="w"> </span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">primary</span><span class="w"> </span><span class="nf">key</span><span class="p">(</span><span class="n">a</span><span class="p">))</span><span class="w"> </span><span class="n">with</span><span class="p">(</span><span class="n">orientation</span><span class="o">=</span><span class="n">column</span><span class="p">,</span><span class="w"> </span><span class="n">enable_hstore_opt</span><span class="o">=</span><span class="n">on</span><span class="p">,</span><span class="w"> </span><span class="n">enable_binlog</span><span class="o">=</span><span class="kc">true</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<p id="EN-US_TOPIC_0000001845482249__p129631621193115">Demo program:</p>
<div class="codecoloring" codetype="Java" id="EN-US_TOPIC_0000001845482249__screen834618461936"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span>
<span class="normal">41</span>
<span class="normal">42</span>
<span class="normal">43</span>
<span class="normal">44</span>
<span class="normal">45</span>
<span class="normal">46</span>
<span class="normal">47</span>
<span class="normal">48</span>
<span class="normal">49</span>
<span class="normal">50</span>
<span class="normal">51</span>
<span class="normal">52</span>
<span class="normal">53</span>
<span class="normal">54</span>
<span class="normal">55</span>
<span class="normal">56</span>
<span class="normal">57</span>
<span class="normal">58</span>
<span class="normal">59</span>
<span class="normal">60</span>
<span class="normal">61</span>
<span class="normal">62</span>
<span class="normal">63</span>
<span class="normal">64</span>
<span class="normal">65</span>
<span class="normal">66</span>
<span class="normal">67</span>
<span class="normal">68</span>
<span class="normal">69</span>
<span class="normal">70</span>
<span class="normal">71</span>
<span class="normal">72</span>
<span class="normal">73</span>
<span class="normal">74</span>
<span class="normal">75</span>
<span class="normal">76</span>
<span class="normal">77</span>
<span class="normal">78</span>
<span class="normal">79</span>
<span class="normal">80</span>
<span class="normal">81</span>
<span class="normal">82</span>
<span class="normal">83</span>
<span class="normal">84</span>
<span class="normal">85</span>
<span class="normal">86</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">BinlogDemo</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">//Name of the binlog table</span>
<span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kd">final</span><span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">BINLOG_TABLE_NAME</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;binlog_test_source&quot;</span><span class="p">;</span>
<span class="w"> </span><span class="c1">//Slot name of the binlog table</span>
<span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kd">final</span><span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">BINLOG_SLOT_NAME</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;binlog_test_slot&quot;</span><span class="p">;</span>
<span class="w"> </span><span class="c1">//Name of the table to be written</span>
<span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kd">final</span><span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">SINK_TABLE_NAME</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;binlog_test_sink&quot;</span><span class="p">;</span>
<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">String</span><span class="o">[]</span><span class="w"> </span><span class="n">args</span><span class="p">)</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">Exception</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">DwsConfig</span><span class="w"> </span><span class="n">dwsConfig</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">buildDwsConfig</span><span class="p">();</span>
<span class="w"> </span><span class="n">DwsClient</span><span class="w"> </span><span class="n">dwsClient</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">DwsClient</span><span class="p">(</span><span class="n">dwsConfig</span><span class="p">);</span>
<span class="w"> </span><span class="n">TableSchema</span><span class="w"> </span><span class="n">sourceTableSchema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dwsClient</span><span class="p">.</span><span class="na">getTableSchema</span><span class="p">(</span><span class="n">TableName</span><span class="p">.</span><span class="na">valueOf</span><span class="p">(</span><span class="n">BINLOG_TABLE_NAME</span><span class="p">));</span>
<span class="w"> </span><span class="n">TableSchema</span><span class="w"> </span><span class="n">sinkTableSchema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dwsClient</span><span class="p">.</span><span class="na">getTableSchema</span><span class="p">(</span><span class="n">TableName</span><span class="p">.</span><span class="na">valueOf</span><span class="p">(</span><span class="n">SINK_TABLE_NAME</span><span class="p">));</span>
<span class="w"> </span><span class="c1">// Columns to be written</span>
<span class="w"> </span><span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="w"> </span><span class="n">sinkColumns</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sinkTableSchema</span><span class="p">.</span><span class="na">getColumnNames</span><span class="p">();</span>
<span class="w"> </span><span class="c1">// Thread pool</span>
<span class="w"> </span><span class="n">DwsConnectionPool</span><span class="w"> </span><span class="n">dwsConnectionPool</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">DwsConnectionPool</span><span class="p">(</span><span class="n">dwsConfig</span><span class="p">);</span>
<span class="w"> </span><span class="c1">//Queue for storing data</span>
<span class="w"> </span><span class="n">BlockingQueue</span><span class="o">&lt;</span><span class="n">BinlogRecord</span><span class="o">&gt;</span><span class="w"> </span><span class="n">queue</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">LinkedBlockingQueue</span><span class="o">&lt;&gt;</span><span class="p">();</span>
<span class="w"> </span><span class="c1">//Columns to be synchronized</span>
<span class="w"> </span><span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="w"> </span><span class="n">sourceColumnNames</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sourceTableSchema</span><span class="p">.</span><span class="na">getColumnNames</span><span class="p">();</span>
<span class="w"> </span><span class="n">BinlogReader</span><span class="w"> </span><span class="n">binlogReader</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">BinlogReader</span><span class="p">(</span><span class="n">dwsConfig</span><span class="p">,</span><span class="w"> </span><span class="n">queue</span><span class="p">,</span><span class="w"> </span><span class="n">sourceColumnNames</span><span class="p">,</span><span class="w"> </span><span class="n">dwsConnectionPool</span><span class="p">);</span>
<span class="w"> </span><span class="c1">//Start the read task.</span>
<span class="w"> </span><span class="n">binlogReader</span><span class="p">.</span><span class="na">start</span><span class="p">();</span>
<span class="w"> </span><span class="n">binlogReader</span><span class="p">.</span><span class="na">getRecords</span><span class="p">();</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">binlogReader</span><span class="p">.</span><span class="na">isStart</span><span class="p">())</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">queue</span><span class="p">.</span><span class="na">isEmpty</span><span class="p">()</span><span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span><span class="o">!</span><span class="n">binlogReader</span><span class="p">.</span><span class="na">hasException</span><span class="p">())</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Read data.</span>
<span class="w"> </span><span class="n">BinlogRecord</span><span class="w"> </span><span class="n">record</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">queue</span><span class="p">.</span><span class="na">poll</span><span class="p">();</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">Objects</span><span class="p">.</span><span class="na">isNull</span><span class="p">(</span><span class="n">record</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">continue</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">BinlogRecordType</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BinlogRecordType</span><span class="p">.</span><span class="na">toBinlogRecordType</span><span class="p">(</span><span class="n">record</span><span class="p">.</span><span class="na">getType</span><span class="p">());</span>
<span class="w"> </span><span class="n">List</span><span class="o">&lt;</span><span class="n">Object</span><span class="o">&gt;</span><span class="w"> </span><span class="n">columnValues</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">record</span><span class="p">.</span><span class="na">getColumnValues</span><span class="p">();</span>
<span class="w"> </span><span class="c1">// Write data.</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">BinlogRecordType</span><span class="p">.</span><span class="na">INSERT</span><span class="p">.</span><span class="na">equals</span><span class="p">(</span><span class="n">type</span><span class="p">)</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="n">BinlogRecordType</span><span class="p">.</span><span class="na">UPDATE_AFTER</span><span class="p">.</span><span class="na">equals</span><span class="p">(</span><span class="n">type</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Operate</span><span class="w"> </span><span class="n">upsert</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dwsClient</span><span class="p">.</span><span class="na">write</span><span class="p">(</span><span class="n">sinkTableSchema</span><span class="p">);</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">sinkColumns</span><span class="p">.</span><span class="na">size</span><span class="p">();</span><span class="w"> </span><span class="n">i</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">upsert</span><span class="p">.</span><span class="na">setObject</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">columnValues</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="n">i</span><span class="p">),</span><span class="w"> </span><span class="kc">false</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">upsert</span><span class="p">.</span><span class="na">commit</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">BinlogRecordType</span><span class="p">.</span><span class="na">DELETE</span><span class="p">.</span><span class="na">equals</span><span class="p">(</span><span class="n">type</span><span class="p">)</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="n">BinlogRecordType</span><span class="p">.</span><span class="na">UPDATE_BEFORE</span><span class="p">.</span><span class="na">equals</span><span class="p">(</span><span class="n">type</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Operate</span><span class="w"> </span><span class="n">delete</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dwsClient</span><span class="p">.</span><span class="na">delete</span><span class="p">(</span><span class="n">sinkTableSchema</span><span class="p">);</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">sinkColumns</span><span class="p">.</span><span class="na">size</span><span class="p">();</span><span class="w"> </span><span class="n">i</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">field</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sinkColumns</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="n">i</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">sinkTableSchema</span><span class="p">.</span><span class="na">isPrimaryKey</span><span class="p">(</span><span class="n">field</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">continue</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">delete</span><span class="p">.</span><span class="na">setObject</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">columnValues</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="n">i</span><span class="p">),</span><span class="w"> </span><span class="kc">false</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">delete</span><span class="p">.</span><span class="na">commit</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">binlogReader</span><span class="p">.</span><span class="na">checkException</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="n">Exception</span><span class="w"> </span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">DwsClientException</span><span class="p">(</span><span class="n">ExceptionCode</span><span class="p">.</span><span class="na">GET_BINLOG_ERROR</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;get binlog has error&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">e</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">DwsConfig</span><span class="w"> </span><span class="nf">buildDwsConfig</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">//Initialize configuration information. (Only necessary parameters are listed. For more information about the configuration, see the document.)</span>
<span class="w"> </span><span class="n">TableConfig</span><span class="w"> </span><span class="n">tableConfig</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">TableConfig</span><span class="p">().</span><span class="na">withBinlog</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">withNewSystemValue</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">withNeedRedistribution</span><span class="p">(</span><span class="kc">false</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">withBinlogSlotName</span><span class="p">(</span><span class="n">BINLOG_SLOT_NAME</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">DwsConfig</span><span class="p">.</span><span class="na">builder</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="na">withUrl</span><span class="p">(</span><span class="s">&quot;Link information&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">withUsername</span><span class="p">(</span><span class="s">&quot;Username&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">withPassword</span><span class="w"> </span><span class="p">(</span><span class="s">&quot;Password&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">withBinlogTableName</span><span class="p">(</span><span class="n">BINLOG_TABLE_NAME</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">withTableConfig</span><span class="p">(</span><span class="n">BINLOG_TABLE_NAME</span><span class="p">,</span><span class="w"> </span><span class="n">tableConfig</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="na">build</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_04_1210.html">Hybrid Data Warehouse Binlog</a></div>
</div>
</div>
View Git Blame Copy Permalink