yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
d167e4cc29458fd80ea2e4d84d6df87e142bbfd1
doc-exports/docs/dli/sqlreference/dli_08_15059.html
Su, Xiaomeng be9eabe464 dli_sqlreference_20250305 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2025-03-25 09:06:21 +00:00

730 lines
68 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<a name="dli_08_15059"></a><a name="dli_08_15059"></a>
<h1 class="topictitle1">MySQL CDC</h1>
<div id="body0000001724418634"><div class="section" id="dli_08_15059__section11231720174018"><h4 class="sectiontitle">Function</h4><p id="dli_08_15059__p12954023204019">The MySQL CDC source table, that is, the MySQL streaming source table, reads all historical data in the database first and then smoothly switches data read to the Binlog to ensure data integrity.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_15059__table3954102713514" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Supported types</caption><thead align="left"><tr id="dli_08_15059__row139551727153515"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.1.3.2.3.1.1"><p id="dli_08_15059__p169550272355">Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.1.3.2.3.1.2"><p id="dli_08_15059__p9955172713520">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_15059__row595518271358"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.1.3.2.3.1.1 "><p id="dli_08_15059__p4955182716353">Supported Table Types</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.1.3.2.3.1.2 "><p id="dli_08_15059__p1595518273356">Source table</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_15059__section153771755204816"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_15059__ul1094501134920"><li id="dli_08_15059__li1585110112719">MySQL CDC requires MySQL 5.6, 5.7, or 8.0.<em id="dli_08_15059__i29086099892527">x</em>.</li><li id="dli_08_15059__li1046993715563">Fields in the <strong id="dli_08_15059__b7494131622510">with</strong> parameter can only be enclosed in single quotes.</li><li id="dli_08_15059__li1817918345355">An enhanced datasource connection has been created for DLI to connect to the MySQL database, so that you can configure security group rules as required.
</li><li id="dli_08_15059__li5583230184910">Binlog is enabled for MySQL, and <strong id="dli_08_15059__b78282831092527">binlog_row_image</strong> is set to <strong id="dli_08_15059__b169937771492527">FULL</strong>.</li><li id="dli_08_15059__li106194349204">A MySQL user has been created and granted the <strong id="dli_08_15059__b203726586492527">SELECT</strong>, <strong id="dli_08_15059__b185314995292527">SHOW DATABASES</strong>, <strong id="dli_08_15059__b203508029092527">REPLICATION SLAVE</strong>, and <strong id="dli_08_15059__b65412997392527">REPLICATION CLIENT</strong> permissions. Note: When the <strong id="dli_08_15059__b1933216369402">scan.incremental.snapshot.enabled</strong> parameter is enabled (enabled by default), there is no need to grant the reload permission.<pre class="screen" id="dli_08_15059__screen168741694467">GRANT SELECT, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user' IDENTIFIED BY 'password';</pre>
</li></ul>
</div>
<div class="section" id="dli_08_15059__section178291144918"><h4 class="sectiontitle">Caveats</h4><ul id="dli_08_15059__ul13107843165817"><li id="dli_08_15059__li13608118132418">When you create a Flink OpenSource SQL job, set <strong id="dli_08_15059__dli_08_15029_b163001353185217">Flink Version</strong> to <strong id="dli_08_15059__dli_08_15029_b1430115539523">1.15</strong> in the <strong id="dli_08_15059__dli_08_15029_b1030175315523">Running Parameters</strong> tab. Select <strong id="dli_08_15059__dli_08_15029_b430135325212">Save Job Log</strong>, and specify the OBS bucket for saving job logs.</li><li id="dli_08_15059__li054552110229">Storing authentication credentials such as usernames and passwords in code or plaintext poses significant security risks. It is recommended using DEW to manage credentials instead. Storing encrypted credentials in configuration files or environment variables and decrypting them when needed ensures security. For details, see .</li><li id="dli_08_15059__li16650612171616">Set a different SERVER ID for each reader.<ul id="dli_08_15059__ul15843142016157"><li id="dli_08_15059__li68071558134019">Each MySQL client used for reading Binlog should have a unique server ID to ensure that the MySQL server can distinguish between different clients and maintain their respective Binlog reading positions.</li><li id="dli_08_15059__li72071414115">Sharing the same server ID among different jobs may lead to reading data from incorrect Binlog positions, resulting in data inconsistency.</li><li id="dli_08_15059__li13807858184013">You can assign a unique server ID to each source reader through SQL hints, such as using <strong id="dli_08_15059__b144621018238">SELECT * FROM source_table /*+ OPTIONS('server-id'='5401-5404') */ ;</strong> to allocate unique server IDs for four source readers.</li></ul>
</li></ul>
<ul id="dli_08_15059__ul14851049162012"><li id="dli_08_15059__li4315181094914">Set up MySQL session timeouts.<div class="p" id="dli_08_15059__p1525554012492"><a name="dli_08_15059__li4315181094914"></a><a name="li4315181094914"></a>When an initial consistent snapshot is made for large databases, your established connection could time out while the tables are being read. You can prevent this behavior by configuring interactive_timeout and wait_timeout in your MySQL configuration file.<ul id="dli_08_15059__ul19201333144920"><li id="dli_08_15059__li320173317494"><strong id="dli_08_15059__b19341841885">interactive_timeout</strong>: The number of seconds the server waits for activity on an interactive connection before closing it. See <a href="https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_interactive_timeout" target="_blank" rel="noopener noreferrer">MySQL documentations</a>.</li><li id="dli_08_15059__li12011330499"><strong id="dli_08_15059__b180819531681">wait_timeout</strong>: The number of seconds the server waits for activity on a noninteractive connection before closing it. See <a href="https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout" target="_blank" rel="noopener noreferrer">MySQL documentations</a>.</li></ul>
</div>
</li><li id="dli_08_15059__li43212354436">Precautions when using tables without primary keys:<ul id="dli_08_15059__ul19201129443"><li id="dli_08_15059__li19293156162319">To use a table without primary keys, you must configure the <strong id="dli_08_15059__b148681679178">scan.incremental.snapshot.chunk.key-column</strong> parameter and specify one non-null field.</li><li id="dli_08_15059__li015364515173">If there is an index in the table, use a column which is contained in the index in <strong id="dli_08_15059__b10126178172014">scan.incremental.snapshot.chunk.key-column</strong>. This will increase the speed of select statement.<div class="p" id="dli_08_15059__p51671931164412">The processing semantics of a MySQL CDC table without primary keys is determined based on the behavior of the column that is specified by the <strong id="dli_08_15059__b15143758121916">scan.incremental.snapshot.chunk.key-column</strong>.<ul id="dli_08_15059__ul1177617579239"><li id="dli_08_15059__li16776125717234">If no update operation is performed on the specified column, the exactly-once semantics is ensured.</li><li id="dli_08_15059__li677645718230">If the update operation is performed on the specified column, only the at-least-once semantics is ensured. However, you can specify primary keys at downstream and perform the idempotence operation to ensure data correctness.</li></ul>
</div>
</li></ul>
</li><li id="dli_08_15059__li4851649102016">Watermarks cannot be defined for MySQL CDC source tables. For details about window aggregation, see <a href="#dli_08_15059__section09412772314">FAQ</a>.</li><li id="dli_08_15059__li1554215505510">If you connect to a sink source that supports upsert, such as GaussDB(DWS) and MySQL, you need to define the primary key in the statement for creating the sink table. For details, see the printSink table creation statement in <a href="#dli_08_15059__section2787228135313">Example</a>.</li></ul>
</div>
<div class="section" id="dli_08_15059__section13284312111414"><h4 class="sectiontitle">Features</h4><ul id="dli_08_15059__ul129361023121416"><li id="dli_08_15059__li14936123181420">Incremental snapshot reading<p id="dli_08_15059__p147661652141718"><a name="dli_08_15059__li14936123181420"></a><a name="li14936123181420"></a>Incremental snapshot reading is a new mechanism to read snapshot of a table. Compared to the old snapshot mechanism, the incremental snapshot has many advantages, including:</p>
<ul id="dli_08_15059__ul057719791813"><li id="dli_08_15059__li15577876188">MySQL CDC Source can be parallel during snapshot reading</li><li id="dli_08_15059__li165771172189">MySQL CDC Source can perform checkpoints in the chunk granularity during snapshot reading</li><li id="dli_08_15059__li10577476187">MySQL CDC Source does not need to acquire global read lock (FLUSH TABLES WITH READ LOCK) before snapshot reading</li></ul>
<p id="dli_08_15059__p1577378180">If you would like the source run in parallel, each parallel reader should have a unique server ID, so the <strong id="dli_08_15059__b81452157117">server-id</strong> must be a range like <strong id="dli_08_15059__b17183202601110">5400-6400</strong>, and the range must be larger than the parallelism. During the incremental snapshot reading, the MySQL CDC Source firstly splits snapshot chunks (splits) by primary key of table, and then MySQL CDC Source assigns the chunks to multiple readers to read the data of snapshot chunk.</p>
</li><li id="dli_08_15059__li98881948101719">Lock-free<p id="dli_08_15059__p860852721818"><a name="dli_08_15059__li98881948101719"></a><a name="li98881948101719"></a>The MySQL CDC source uses incremental snapshot algorithm, which avoids acquiring global read lock (FLUSH TABLES WITH READ LOCK) and thus does not need <strong id="dli_08_15059__b1941919171315">RELOAD</strong> permission.</p>
</li><li id="dli_08_15059__li844322441814">Controlling parallelism<p id="dli_08_15059__p4034361915"><a name="dli_08_15059__li844322441814"></a><a name="li844322441814"></a>Incremental snapshot reading provides the ability to read snapshot data parallelly.</p>
</li><li id="dli_08_15059__li63383369190">Checkpoint<p id="dli_08_15059__p1226031117202"><a name="dli_08_15059__li63383369190"></a><a name="li63383369190"></a>Incremental snapshot reading provides the ability to perform checkpoint in chunk level. It resolves the checkpoint timeout problem in previous version with old snapshot reading mechanism.</p>
</li></ul>
</div>
<div class="section" id="dli_08_15059__section1332752476"><h4 class="sectiontitle">Syntax</h4><pre class="screen" id="dli_08_15059__screen1996932684815">create table mySqlCdcSource (
attr_name attr_type
(',' attr_name attr_type)*
(','PRIMARY KEY (attr_name, ...) NOT ENFORCED)
)
with (
'connector' = 'mysql-cdc',
'hostname' = '<em id="dli_08_15059__i9899205202018">mysqlHostname</em>',
'username' = '<em id="dli_08_15059__i16692205614208">mysql</em><em id="dli_08_15059__i4215102142114">Username</em>',
'password' = '<em id="dli_08_15059__i16620711122116">mysql</em><em id="dli_08_15059__i7743151782119">Password</em>',
'database-name' = '<em id="dli_08_15059__i1518061412217">mysql</em><em id="dli_08_15059__i81921720172118">DatabaseName</em>',
'table-name' = '<em id="dli_08_15059__i16457111517215">mysql</em><em id="dli_08_15059__i62705226211">TableName</em>'
);</pre>
</div>
<div class="section" id="dli_08_15059__section53607322481"><h4 class="sectiontitle">Parameter Description</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_15059__table849825425517" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Source table parameters</caption><thead align="left"><tr id="dli_08_15059__row5498185405511"><th align="left" class="cellrowborder" valign="top" width="17.19%" id="mcps1.3.6.2.2.6.1.1"><p id="dli_08_15059__p349812548553">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="12.8%" id="mcps1.3.6.2.2.6.1.2"><p id="dli_08_15059__p144981154105515">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="11.020000000000001%" id="mcps1.3.6.2.2.6.1.3"><p id="dli_08_15059__p1749810542553">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="14.41%" id="mcps1.3.6.2.2.6.1.4"><p id="dli_08_15059__p14981154165516">Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="44.58%" id="mcps1.3.6.2.2.6.1.5"><p id="dli_08_15059__p34982544552">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_15059__row19498105445515"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p049865435510">connector</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p19498125414555">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p449875418553">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p14983548556">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p1526613419233">Specify what connector to use, here should be <strong id="dli_08_15059__b657418652011">mysql-cdc</strong>.</p>
</td>
</tr>
<tr id="dli_08_15059__row5498145455517"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p84983549553">hostname</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p849815425518">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p7441133135910">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p1637124413594">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p1049817540551">IP address or hostname of the MySQL database server.</p>
</td>
</tr>
<tr id="dli_08_15059__row19498205445512"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p84986542551">username</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p10498125413552">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p149711833205911">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p779394425912">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p20512182019566">Name of the MySQL database to use when connecting to the MySQL database server.</p>
</td>
</tr>
<tr id="dli_08_15059__row1249820545550"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p4498254185515">password</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p5498135420559">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p1431154391910">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p13503164512597">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p1963312725616">Password to use when connecting to the MySQL database server.</p>
</td>
</tr>
<tr id="dli_08_15059__row1749835425514"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p134981454165518">database-name</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p74981554135512">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p85511935165915">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p9250124620591">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p64742418577">Database name of the MySQL server to monitor.</p>
<p id="dli_08_15059__p898518675719">The <strong id="dli_08_15059__b12630713152116">database-name</strong> also supports regular expressions to monitor multiple tables match the regular expression.</p>
<ul id="dli_08_15059__ul151111941165619"><li id="dli_08_15059__li20111204105611">Prefix matching: <strong id="dli_08_15059__b1642118354414">^(test).*</strong> matches database names with the prefix <strong id="dli_08_15059__b53041351134318">test</strong>, for example, <strong id="dli_08_15059__b136295418439">test1</strong> and <strong id="dli_08_15059__b1780810556433">test2</strong>.</li><li id="dli_08_15059__li93713314571">Suffix matching: <strong id="dli_08_15059__b14835142184515">.*[p$]</strong> matches database names with the suffix <strong id="dli_08_15059__b17115131916451">p</strong>, for example, <strong id="dli_08_15059__b9665102224520">cdcp</strong> and <strong id="dli_08_15059__b1741313247457">edcp</strong>.</li><li id="dli_08_15059__li553182215571">Specific matching: <strong id="dli_08_15059__b77843914464">txc</strong> matches a specific database name.</li></ul>
</td>
</tr>
<tr id="dli_08_15059__row649895420554"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p154981054105512">table-name</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p249835435515">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p1321233675918">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p1521647155913">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p1124702411575">Table name of the MySQL database to monitor. The table-name also supports regular expressions to monitor multiple tables that satisfy the regular expressions.</p>
<div class="note" id="dli_08_15059__note1844515561452"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_15059__p124987542557">When the MySQL CDC connector regularly matches the table name, it will concat the database-name and table-name filled in by the user through the string `\\.` to form a full-path regular expression, and then use the regular expression to match the fully qualified name of the table in the MySQL database.</p>
<ul id="dli_08_15059__ul72457539579"><li id="dli_08_15059__li112450539571">Prefix matching: <strong id="dli_08_15059__b563141715460">^(test).*</strong> matches table names with the prefix <strong id="dli_08_15059__b263211724610">test</strong>, for example, <strong id="dli_08_15059__b263313171463">test1</strong> and <strong id="dli_08_15059__b36331417184613">test2</strong>.</li><li id="dli_08_15059__li14245165335720">Suffix matching: <strong id="dli_08_15059__b19559233114611">.*[p$]</strong> matches table names with the suffix <strong id="dli_08_15059__b9560123310462">p</strong>, for example, <strong id="dli_08_15059__b2561173354611">cdcp</strong> and <strong id="dli_08_15059__b1956183384612">edcp</strong>.</li><li id="dli_08_15059__li1245125316579">Specific matching: <strong id="dli_08_15059__b1442074119463">txc</strong> matches a specific table name.</li></ul>
</div></div>
</td>
</tr>
<tr id="dli_08_15059__row0498454205517"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p54987546551">port</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p44983541557">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p549875485510">3306</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p4498135475510">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p12498195405519">Integer port number of the MySQL database server.</p>
</td>
</tr>
<tr id="dli_08_15059__row1149825405513"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p649945412556">server-id</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p049975412553">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p11499105413557">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p1069411548597">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p18535710175915">A numeric ID or a numeric ID range of this database client. The numeric ID syntax is like <strong id="dli_08_15059__b399731915234">5400</strong>, the numeric ID range syntax is like <strong id="dli_08_15059__b1541662982318">5400-5408</strong>.</p>
<p id="dli_08_15059__p16672158144615">The numeric ID range syntax is recommended when <strong id="dli_08_15059__b126841230182018">scan.incremental.snapshot.enabled</strong> enabled.</p>
<p id="dli_08_15059__p494316151593">Every ID must be unique across all currently-running database processes in the MySQL cluster. This connector joins the MySQL cluster as another server (with this unique ID) so it can read the binlog. By default, a random number is generated between 5400 and 6400, though we recommend setting an explicit value.</p>
</td>
</tr>
<tr id="dli_08_15059__row1749918543558"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p1549975419557">scan.incremental.snapshot.enabled</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p19499145435512">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p11499454105513">true</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p118561016107">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p171649191609">Incremental snapshot is a new mechanism to read snapshot of a table. Compared to the old snapshot mechanism, the incremental snapshot has many advantages, including:</p>
<ul id="dli_08_15059__ul1346102310010"><li id="dli_08_15059__li16150527106">MySQL CDC Source can be parallel during snapshot reading</li><li id="dli_08_15059__li55323311200">MySQL CDC Source can perform checkpoints in the chunk granularity during snapshot reading</li><li id="dli_08_15059__li15640162814313">MySQL CDC Source does not need to acquire global read lock (FLUSH TABLES WITH READ LOCK) before snapshot reading<p id="dli_08_15059__p56115302034"><a name="dli_08_15059__li15640162814313"></a><a name="li15640162814313"></a>If you would like the source run in parallel, each parallel reader should have a unique server ID, so the <strong id="dli_08_15059__b7494131182613">server-id</strong> must be a range like <strong id="dli_08_15059__b20172103719265">5400-6400</strong>, and the range must be larger than the parallelism.</p>
</li></ul>
</td>
</tr>
<tr id="dli_08_15059__row6499154185515"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p1549917548554">scan.incremental.snapshot.chunk.size</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p449975465513">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p38971824111">8096</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p1877118178011">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p198669311215">The chunk size (number of rows) of table snapshot, captured tables are split into multiple chunks when reading the snapshot of table.</p>
</td>
</tr>
<tr id="dli_08_15059__row7526161517215"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p45268154213">scan.startup.mode</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p105261515628">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p652610151622">initial</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p45263151129">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p1352651512218">Optional startup mode for MySQL CDC consumer, valid enumerations are <strong id="dli_08_15059__b122617743418">initial</strong>, <strong id="dli_08_15059__b1544641019346">earliest-offset</strong>, <strong id="dli_08_15059__b103411914203412">latest-offset</strong>, <strong id="dli_08_15059__b13648141983417">specific-offset</strong>, and <strong id="dli_08_15059__b3575142623413">timestamp</strong>.</p>
<ul id="dli_08_15059__ul71673520417"><li id="dli_08_15059__li101676516419"><strong id="dli_08_15059__b7453156184810">initial</strong> (default): Perform an initial snapshot on the monitored database tables upon first startup, and continue to read the latest binlog.</li><li id="dli_08_15059__li71674513411"><strong id="dli_08_15059__b1829463834820">earliest-offset</strong>: Skip snapshot phase and start reading binlog events from the earliest accessible binlog offset.</li><li id="dli_08_15059__li6167751549"><strong id="dli_08_15059__b16487100134912">latest-offset</strong>: Never to perform snapshot on the monitored database tables upon first startup, just read from the end of the binlog which means only have the changes since the connector was started.</li><li id="dli_08_15059__li11671654419"><strong id="dli_08_15059__b145232217494">specific-offset</strong>: Skip snapshot phase and start reading binlog events from a specific offset. The offset could be specified with binlog filename and position, or a GTID set if GTID is enabled on server.</li><li id="dli_08_15059__li13167125844"><strong id="dli_08_15059__b1185165674913">timestamp</strong>: Skip snapshot phase and start reading binlog events from a specific timestamp.</li></ul>
</td>
</tr>
<tr id="dli_08_15059__row174972015416"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p20749122017416">scan.startup.specific-offset.file</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p1674917204417">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p197491620147">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p10749320946">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p1374932017413">Optional binlog file name used in case of <strong id="dli_08_15059__b1717717598359">specific-offset</strong> startup mode</p>
</td>
</tr>
<tr id="dli_08_15059__row824545116414"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p102455514416">scan.startup.specific-offset.pos</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p2245135117412">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p9245105120419">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p1524555115416">Long</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p5245651141">Optional binlog file position used in case of <strong id="dli_08_15059__b1213632113612">specific-offset</strong> startup mode</p>
</td>
</tr>
<tr id="dli_08_15059__row157531153043"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p117531953248">scan.startup.specific-offset.gtid-set</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p13753353647">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p1753165319414">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p7753153946">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p2075320531241">Optional GTID set used in case of <strong id="dli_08_15059__b207991113133718">specific-offset</strong> startup mode</p>
</td>
</tr>
<tr id="dli_08_15059__row10987165515420"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p209871755342">scan.startup.specific-offset.skip-events</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p998755511413">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p13987105512410">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p098711550418">Long</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p1798719553411">Optional number of events to skip after the specific starting offset</p>
</td>
</tr>
<tr id="dli_08_15059__row11522142266"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p052217210619">scan.startup.specific-offset.skip-rows</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p4522162167">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p175221217620">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p19522621965">Long</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p2052282862">Optional number of rows to skip after the specific starting offset</p>
</td>
</tr>
<tr id="dli_08_15059__row33136719618"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p7313137862">server-time-zone</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p731315720613">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p3313771267">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p6314677611">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p1273465314273">Session time zone on the database server</p>
<p id="dli_08_15059__p1278315444715">It controls how the TIMESTAMP type in MYSQL converted to STRING. If not set, then <strong id="dli_08_15059__b64471258143516">ZoneId.systemDefault()</strong> is used to determine the server time zone.</p>
</td>
</tr>
<tr id="dli_08_15059__row112323520616"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p19232175869">debezium.min.row. count.to.stream.result</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p1232115366">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p12321751361">1000</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p1223217511618">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p7646755879">During a snapshot operation, the connector will query each included table to produce a read event for all rows in that table.</p>
<p id="dli_08_15059__p1063946816">This parameter determines whether the MySQL connection will pull all results for a table into memory (which is fast but requires large amounts of memory), or whether the results will instead be streamed (can be slower, but will work for very large tables). The value specifies the minimum number of rows a table must contain before the connector will stream results, and defaults to <strong id="dli_08_15059__b1779785364514">1000</strong>.</p>
<p id="dli_08_15059__p10232185962">Set this parameter to <strong id="dli_08_15059__b22547694616">0</strong> to skip all table size checks and always stream all results during a snapshot.</p>
</td>
</tr>
<tr id="dli_08_15059__row19584818811"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p25868383">connect.timeout</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p1958981584">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p65819815819">30s</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p135815817812">Duration</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p2598816814">The maximum time that the connector should wait after trying to connect to the MySQL database server before timing out.</p>
</td>
</tr>
<tr id="dli_08_15059__row714415534812"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p314495320811">connect.max-retries</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p11441553789">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p1414416531820">3</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p13144353783">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p81443538814">The max retry times that the connector should retry to build MySQL database server connection.</p>
</td>
</tr>
<tr id="dli_08_15059__row425815551181"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p325815512813">connection.pool.size</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p17258155289">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p162586551483">20</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p1125865518812">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p52580551384">The connection pool size.</p>
</td>
</tr>
<tr id="dli_08_15059__row194467582810"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p17446155816814">jdbc.properties.*</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p1644612589814">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p644614589812">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p84463581811">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p195921719111015">Option to pass custom JDBC URL properties.</p>
<p id="dli_08_15059__p204460585816">User can pass custom properties like <strong id="dli_08_15059__b53931845194717">'jdbc.properties.useSSL' = 'false'</strong>.</p>
</td>
</tr>
<tr id="dli_08_15059__row21557120910"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p91551014919">heartbeat.interval</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p8155715915">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p715513120915">30s</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p6155131299">Duration</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p915515116919">The interval of sending heartbeat event for tracing the latest available binlog offsets.</p>
</td>
</tr>
<tr id="dli_08_15059__row5452133790"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p04522318913">debezium.*</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p7452631891">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p1545218316915">None</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p17452132913">String</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p205888614114">Pass-through Debezium's properties to Debezium Embedded Engine which is used to capture data changes from MySQL server.</p>
<p id="dli_08_15059__p945219316912">For example: <strong id="dli_08_15059__b9452143491">'debezium.snapshot.mode' = 'never'</strong>. See more about the <a href="https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-connector-properties" target="_blank" rel="noopener noreferrer">Debezium's MySQL Connector properties</a>.</p>
</td>
</tr>
<tr id="dli_08_15059__row151341012141110"><td class="cellrowborder" valign="top" width="17.19%" headers="mcps1.3.6.2.2.6.1.1 "><p id="dli_08_15059__p2013421221111">scan.incremental.close-idle-reader.enabled</p>
</td>
<td class="cellrowborder" valign="top" width="12.8%" headers="mcps1.3.6.2.2.6.1.2 "><p id="dli_08_15059__p1134121261115">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.020000000000001%" headers="mcps1.3.6.2.2.6.1.3 "><p id="dli_08_15059__p713461219113">false</p>
</td>
<td class="cellrowborder" valign="top" width="14.41%" headers="mcps1.3.6.2.2.6.1.4 "><p id="dli_08_15059__p2134112201115">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="44.58%" headers="mcps1.3.6.2.2.6.1.5 "><p id="dli_08_15059__p11134912121115">Whether to close idle readers at the end of the snapshot phase. This feature requires that <strong id="dli_08_15059__b132421343165010">execution.checkpointing.checkpoints-after-tasks-finish.enabled</strong> be set to <strong id="dli_08_15059__b1617794585012">true</strong>.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_15059__section176571852181811"><h4 class="sectiontitle">Metadata</h4><p id="dli_08_15059__p648195514182">The following format metadata can be exposed as read-only (VIRTUAL) columns in DDL.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_15059__table617531141919" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Metadata</caption><thead align="left"><tr id="dli_08_15059__row91717318191"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.7.3.2.4.1.1"><p id="dli_08_15059__p1317131121914">Key</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.7.3.2.4.1.2"><p id="dli_08_15059__p317163171919">Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.7.3.2.4.1.3"><p id="dli_08_15059__p317931121914">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_15059__row12171531171911"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.7.3.2.4.1.1 "><p id="dli_08_15059__p12171631131910">table_name</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.7.3.2.4.1.2 "><p id="dli_08_15059__p817163151915">STRING NOT NULL</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.7.3.2.4.1.3 "><p id="dli_08_15059__p81719319198">Name of the table that contains the row.</p>
</td>
</tr>
<tr id="dli_08_15059__row101743118193"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.7.3.2.4.1.1 "><p id="dli_08_15059__p4171531181914">database_name</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.7.3.2.4.1.2 "><p id="dli_08_15059__p31763118193">STRING NOT NULL</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.7.3.2.4.1.3 "><p id="dli_08_15059__p201713101914">Name of the database that contains the row.</p>
</td>
</tr>
<tr id="dli_08_15059__row1717103131915"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.7.3.2.4.1.1 "><p id="dli_08_15059__p161783118191">op_ts</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.7.3.2.4.1.2 "><p id="dli_08_15059__p217173161913">TIMESTAMP_LTZ(3) NOT NULL</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.7.3.2.4.1.3 "><p id="dli_08_15059__p879411419213">It indicates the time that the change was made in the database.</p>
<p id="dli_08_15059__p01712315195">If the record is read from snapshot of the table instead of the binlog, the value is always <strong id="dli_08_15059__b112546016520">0</strong>.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_15059__section144091727219"><h4 class="sectiontitle">Data Type Mapping</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_15059__table12802936828" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Data type mapping</caption><thead align="left"><tr id="dli_08_15059__row168026361224"><th align="left" class="cellrowborder" valign="top" width="38.18381838183818%" id="mcps1.3.8.2.2.4.1.1"><p id="dli_08_15059__p2802153611217">MySQL Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="28.48284828482848%" id="mcps1.3.8.2.2.4.1.2"><p id="dli_08_15059__p1080217361429">Flink SQL Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.8.2.2.4.1.3"><p id="dli_08_15059__p118027363216">Remarks</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_15059__row16802163616218"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p28028362212">TINYINT</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p1780203615211">TINYINT</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p19802536829">-</p>
</td>
</tr>
<tr id="dli_08_15059__row1980243612214"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p3938855437">SMALLINT</p>
<p id="dli_08_15059__p16575185915314">TINYINT UNSIGNED</p>
<p id="dli_08_15059__p8802183613219">TINYINT UNSIGNED ZEROFILL</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p58021836821">SMALLINT</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p98021436227">-</p>
</td>
</tr>
<tr id="dli_08_15059__row980215361528"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p1142133447">INT</p>
<p id="dli_08_15059__p5523188348">MEDIUMINT</p>
<p id="dli_08_15059__p55811121344">SMALLINT UNSIGNED</p>
<p id="dli_08_15059__p48025361427">SMALLINT UNSIGNED ZEROFILL</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p780215361128">INT</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p16802236422">-</p>
</td>
</tr>
<tr id="dli_08_15059__row68029361724"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p11554131711410">BIGINT</p>
<p id="dli_08_15059__p1844514211414">INT UNSIGNED</p>
<p id="dli_08_15059__p627210295415">INT UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p6281173517415">MEDIUMINT UNSIGNED</p>
<p id="dli_08_15059__p38029366220">MEDIUMINT UNSIGNED ZEROFILL</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p6802143612216">BIGINT</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p1080214363220">-</p>
</td>
</tr>
<tr id="dli_08_15059__row3802236122"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p1739015395411">BIGINT UNSIGNED</p>
<p id="dli_08_15059__p1522445244">BIGINT UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p1680213620212">SERIAL</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p1480211364213">DECIMAL(20, 0)</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p118023361325">-</p>
</td>
</tr>
<tr id="dli_08_15059__row880212369211"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p125237548414">FLOAT</p>
<p id="dli_08_15059__p947011567414">FLOAT UNSIGNED</p>
<p id="dli_08_15059__p580217369210">FLOAT UNSIGNED ZEROFILL</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p13802236127">FLOAT</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p18802153615213">-</p>
</td>
</tr>
<tr id="dli_08_15059__row1580283618218"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p492010138511">REAL</p>
<p id="dli_08_15059__p7590181512515">REAL UNSIGNED</p>
<p id="dli_08_15059__p116694311353">REAL UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p1847914353515">DOUBLE</p>
<p id="dli_08_15059__p3712437155">DOUBLE UNSIGNED</p>
<p id="dli_08_15059__p13759740354">DOUBLE UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p83814448513">DOUBLE PRECISION</p>
<p id="dli_08_15059__p1121494618519">DOUBLE PRECISION UNSIGNED</p>
<p id="dli_08_15059__p780214361521">DOUBLE PRECISION UNSIGNED ZEROFILL</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p17802536621">DOUBLE</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p58023362217">-</p>
</td>
</tr>
<tr id="dli_08_15059__row188027361829"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p346875910517">NUMERIC(p, s)</p>
<p id="dli_08_15059__p085818216620">NUMERIC(p, s) UNSIGNED</p>
<p id="dli_08_15059__p1850210763">NUMERIC(p, s) UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p205607141360">DECIMAL(p, s)</p>
<p id="dli_08_15059__p130182014612">DECIMAL(p, s) UNSIGNE</p>
<p id="dli_08_15059__p166801241463">DDECIMAL(p, s) UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p1057116269615">FIXED(p, s)</p>
<p id="dli_08_15059__p56971628762">FIXED(p, s) UNSIGNED</p>
<p id="dli_08_15059__p13548183312611">FIXED(p, s) UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p1680215361320">where p &lt;= 38</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p19802936923">DECIMAL(p, s)</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p148023361423">-</p>
</td>
</tr>
<tr id="dli_08_15059__row1680218363212"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p115155018610">NUMERIC(p, s)</p>
<p id="dli_08_15059__p63918521762">NUMERIC(p, s) UNSIGNED</p>
<p id="dli_08_15059__p1998719561860">NUMERIC(p, s) UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p1376019594614">DECIMAL(p, s)</p>
<p id="dli_08_15059__p113698212714">DECIMAL(p, s) UNSIGNED</p>
<p id="dli_08_15059__p31491616717">DECIMAL(p, s) UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p10569691079">FIXED(p, s)</p>
<p id="dli_08_15059__p12540191211713">FIXED(p, s) UNSIGNED</p>
<p id="dli_08_15059__p1812713167715">FIXED(p, s) UNSIGNED ZEROFILL</p>
<p id="dli_08_15059__p08021336924">where 38 &lt; p &lt;= 65</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p6802193616215">STRING</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p185676541439">The precision for DECIMAL data type is up to 65 in MySQL, but the precision for DECIMAL is limited to 38 in Flink.</p>
<p id="dli_08_15059__p621714381411">So if you define a decimal column whose precision is greater than 38, you should map it to STRING to avoid precision loss.</p>
</td>
</tr>
<tr id="dli_08_15059__row2802736528"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p123263345717">BOOLEAN</p>
<p id="dli_08_15059__p138394351772">TINYINT(1)</p>
<p id="dli_08_15059__p1080313610216">BIT(1)</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p58032368214">BOOLEAN</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p38036369215">-</p>
</td>
</tr>
<tr id="dli_08_15059__row1080310361218"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p48038361211">DATE</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p138031836821">DATE</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p17536445144711">-</p>
</td>
</tr>
<tr id="dli_08_15059__row15803736526"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p5803236421">TIME [(p)]</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p18803133619210">TIME [(p)]</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p1354544564713">-</p>
</td>
</tr>
<tr id="dli_08_15059__row8803836429"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p16605316189">TIMESTAMP [(p)]</p>
<p id="dli_08_15059__p480313619216">DATETIME [(p)]</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p680393611213">TIMESTAMP [(p)]</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p855364594718">-</p>
</td>
</tr>
<tr id="dli_08_15059__row680312362211"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p680319361424">CHAR(n)</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p15803123611217">CHAR(n)</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p2056004504718">-</p>
</td>
</tr>
<tr id="dli_08_15059__row19803536926"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p1480319361527">VARCHAR(n)</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p1580314361629">VARCHAR(n)</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p456816452476">-</p>
</td>
</tr>
<tr id="dli_08_15059__row1180315364212"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p2080313361210">BIT(n)</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p1080313361627">BINARY(⌈n/8⌉)</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p1857534512474">-</p>
</td>
</tr>
<tr id="dli_08_15059__row178031361212"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p19803336427">BINARY(n)</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p138031336321">BINARY(n)</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p8584194584718">-</p>
</td>
</tr>
<tr id="dli_08_15059__row178038361123"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p17803436923">VARBINARY(N)</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p380320365220">VARBINARY(N)</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p17591194513477">-</p>
</td>
</tr>
<tr id="dli_08_15059__row780393612212"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p96243181297">TINYTEXT</p>
<p id="dli_08_15059__p177342211297">TEXT</p>
<p id="dli_08_15059__p687312241991">MEDIUMTEXT</p>
<p id="dli_08_15059__p178031836822">LONGTEXT</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p9803123611215">STRING</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p159944574716">-</p>
</td>
</tr>
<tr id="dli_08_15059__row1080313361124"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p132732371798">TINYBLOB</p>
<p id="dli_08_15059__p134034391691">BLOB</p>
<p id="dli_08_15059__p0593143791">MEDIUMBLOB</p>
<p id="dli_08_15059__p128032361211">LONGBLOB</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p68034361727">BYTES</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p1180316365216">Currently, for BLOB data type in MySQL, only the blob whose length is not greater than 2,147,483,647(2 ** 31 – 1) is supported.</p>
</td>
</tr>
<tr id="dli_08_15059__row18034361327"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p180317367219">YEAR</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p880317361224">INT</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p19944184744717">-</p>
</td>
</tr>
<tr id="dli_08_15059__row9803193610212"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p19803136627">ENUM</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p780320363217">STRING</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p19952947124715">-</p>
</td>
</tr>
<tr id="dli_08_15059__row1480316362029"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p6803536227">JSON</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p38031636921">STRING</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p080311367210">The JSON data type will be converted into STRING with JSON format in Flink.</p>
</td>
</tr>
<tr id="dli_08_15059__row2803193610213"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p7803163617212">SET</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p88031836324">ARRAY&lt;STRING&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p48031236622">As the SET data type in MySQL is a string object that can have zero or more values, it should always be mapped to an array of string.</p>
</td>
</tr>
<tr id="dli_08_15059__row14803143619218"><td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.8.2.2.4.1.1 "><p id="dli_08_15059__p5781114311019">GEOMETRY</p>
<p id="dli_08_15059__p1901747201011">POINT</p>
<p id="dli_08_15059__p037015010106">LINESTRING</p>
<p id="dli_08_15059__p496715241016">POLYGON</p>
<p id="dli_08_15059__p20275185616109">MULTIPOINT</p>
<p id="dli_08_15059__p840015091120">MULTILINESTRING</p>
<p id="dli_08_15059__p1173013714114">MULTIPOLYGON</p>
<p id="dli_08_15059__p7803153619211">GEOMETRYCOLLECTION</p>
</td>
<td class="cellrowborder" valign="top" width="28.48284828482848%" headers="mcps1.3.8.2.2.4.1.2 "><p id="dli_08_15059__p58031436724">STRING</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.2.2.4.1.3 "><p id="dli_08_15059__p080333614216">The spatial data types in MySQL will be converted into STRING with a fixed Json format.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_15059__section2787228135313"><a name="dli_08_15059__section2787228135313"></a><a name="section2787228135313"></a><h4 class="sectiontitle">Example</h4><p id="dli_08_15059__p62922413482">This example demonstrates the use of MySQL-CDC to read data and metadata from an RDS for MySQL database in real-time and write it to a Print result table.</p>
<p id="dli_08_15059__p19594121134818">In this example, the RDS for MySQL database engine version is MySQL 5.7.33.</p>
<ol id="dli_08_15059__ol5185446103217"><li id="dli_08_15059__li6127741153217">Create an enhanced datasource connection in the VPC and subnet where MySQL locates, and bind the connection to the required Flink elastic resource pool.</li><li id="dli_08_15059__li1233002211514">Set MySQL security groups and add inbound rules to allow access from the Flink queue. Test the connectivity using the MySQL address. If the connection passes the test, it is bound to the queue.</li><li id="dli_08_15059__li4437132516232">Create the <strong id="dli_08_15059__b719683175311">test</strong> user in MySQL and grant them permissions. The SQL statements are as follows:<pre class="screen" id="dli_08_15059__screen470795372316">CREATE USER 'test'@'%' IDENTIFIED BY 'xxx';
GRANT SELECT, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'test';
FLUSH PRIVILEGES;</pre>
</li><li id="dli_08_15059__li4185164615322">Create a table named <strong id="dli_08_15059__b2194102312544">cdc_order</strong> in the Flink database of MySQL. The SQL statement is as follows (this statement requires the user to have the <strong id="dli_08_15059__b45095435411">CREATE</strong> permission):<pre class="screen" id="dli_08_15059__screen9883103014217">CREATE TABLE `flink`.`cdc_order` (
`order_id` VARCHAR(32) NOT NULL,
`order_channel` VARCHAR(32) NULL,
`order_time` VARCHAR(32) NULL,
`pay_amount` DOUBLE NULL,
`real_pay` DOUBLE NULL,
`pay_time` VARCHAR(32) NULL,
`user_id` VARCHAR(32) NULL,
`user_name` VARCHAR(32) NULL,
`area_id` VARCHAR(32) NULL,
PRIMARY KEY (`order_id`)
) ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8mb4
COLLATE = utf8mb4_general_ci;</pre>
</li><li id="dli_08_15059__li133558543514">Create a Flink OpenSource SQL job. Enter the following job script and submit the job.<div class="p" id="dli_08_15059__p145941442173519"><a name="dli_08_15059__li133558543514"></a><a name="li133558543514"></a>When you create a job, set <strong id="dli_08_15059__b182783435422734">Flink Version</strong> to <strong id="dli_08_15059__b92516962022734">1.15</strong> in the <strong id="dli_08_15059__b189202099122734">Running Parameters</strong> tab. Select <strong id="dli_08_15059__b109732349822734">Save Job Log</strong>, and specify the OBS bucket for saving job logs. <strong id="dli_08_15059__b11942753191129">Change the values of the parameters in bold as needed in the following script.</strong><pre class="screen" id="dli_08_15059__screen2639193553516">create table mysqlCdcSource(
database_name STRING METADATA VIRTUAL,
table_name STRING METADATA VIRTUAL,
operation_ts TIMESTAMP_LTZ(3) METADATA FROM 'op_ts' VIRTUAL,
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id STRING,
primary key(order_id) not enforced
) with (
'connector' = 'mysql-cdc',
'hostname' = '<em id="dli_08_15059__i52374559382"><strong id="dli_08_15059__b152371855133818">mysqlHostname</strong></em>',
'username' = '<em id="dli_08_15059__i1460181113917"><strong id="dli_08_15059__b1446018111391">mysql</strong></em><em id="dli_08_15059__i184601214391"><strong id="dli_08_15059__b18460619392">Username</strong></em>',
'password' = '<em id="dli_08_15059__i31771684391"><strong id="dli_08_15059__b6177108103916">mysql</strong></em><em id="dli_08_15059__i517711818394"><strong id="dli_08_15059__b19178148193919">Password</strong></em>',
'database-name' = '<em id="dli_08_15059__i1841961319399"><strong id="dli_08_15059__b1419141373913">mysql</strong></em><em id="dli_08_15059__i10419713103911"><strong id="dli_08_15059__b20419141363920">DatabaseName</strong></em>',
'table-name' = '<em id="dli_08_15059__i1540011916393"><strong id="dli_08_15059__b11400121910392">mysql</strong></em><em id="dli_08_15059__i144003194395"><strong id="dli_08_15059__b24008194396">TableName</strong></em>'
);
create table printSink(
database_name string,
table_name string,
operation_ts TIMESTAMP_LTZ(3),
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id STRING,
primary key(order_id) not enforced
) with (
'connector' = 'print'
);
insert into printSink select * from mysqlCdcSource;</pre>
</div>
</li><li id="dli_08_15059__li18185194619329">Run the following commands in MySQL to insert test data (this statement requires the user to have the corresponding permission):<pre class="screen" id="dli_08_15059__screen11849141112415">insert into flink.cdc_order values
('202103241000000001','webShop','2021-03-24 10:00:00','100.00','100.00','2021-03-24 10:02:03','0001','Alice','330106'),
('202103241606060001','appShop','2021-03-24 16:06:06','200.00','180.00','2021-03-24 16:10:06','0001','Alice','330106');
delete from flink.cdc_order where order_channel = 'webShop';
insert into flink.cdc_order values('202103251202020001','miniAppShop','2021-03-25 12:02:02','60.00','60.00','2021-03-25 12:03:00','0002','Bob','330110');</pre>
</li><li id="dli_08_15059__li10185946183216">Perform the following operations to view the data result in the <strong id="dli_08_15059__b212427117292527">taskmanager.out</strong> file:<ol type="a" id="dli_08_15059__ol864115198285"><li id="dli_08_15059__li10901621122819">Log in to the DLI management console. In the navigation pane on the left, choose <strong id="dli_08_15059__b3514540326">Job Management</strong> &gt; <strong id="dli_08_15059__b1551434016219">Flink Jobs</strong>.</li><li id="dli_08_15059__li1912163912282">Click the name of the corresponding Flink job, choose <strong id="dli_08_15059__b148609897192527">Run Log</strong>, click <strong id="dli_08_15059__b62158056592527">OBS Bucket</strong>, and locate the folder of the log you want to view according to the date.</li><li id="dli_08_15059__li0641191914285">Go to the folder of the date, find the folder whose name contains <strong id="dli_08_15059__b37112848891933">taskmanager</strong>, download the <strong id="dli_08_15059__b116917576291933">taskmanager.out</strong> file, and view result logs.</li></ol>
<p id="dli_08_15059__p1282118785912">The data result is as follows:</p>
<pre class="screen" id="dli_08_15059__screen17252518182418"> +I[flink, cdc_order, 2023-11-10T07:41:12Z, 202103241000000001, webShop, 2021-03-24 10:00:00, 100.0, 100.0, 2021-03-24 10:02:03, 0001, Alice, 330106]
+I[flink, cdc_order, 2023-11-10T07:41:12Z, 202103241606060001, appShop, 2021-03-24 16:06:06, 200.0, 180.0, 2021-03-24 16:10:06, 0001, Alice, 330106]
-D[flink, cdc_order, 2023-11-10T07:41:59Z, 202103241000000001, webShop, 2021-03-24 10:00:00, 100.0, 100.0, 2021-03-24 10:02:03, 0001, Alice, 330106]
+I[flink, cdc_order, 2023-11-10T07:42:00Z, 202103251202020001, miniAppShop, 2021-03-25 12:02:02, 60.0, 60.0, 2021-03-25 12:03:00, 0002, Bob, 330110]</pre>
</li></ol>
</div>
<div class="section" id="dli_08_15059__section09412772314"><a name="dli_08_15059__section09412772314"></a><a name="section09412772314"></a><h4 class="sectiontitle">FAQ</h4><p id="dli_08_15059__p83831313395">Q: How do I perform window aggregation if the MySQL CDC source table does not support definition of watermarks?</p>
<p id="dli_08_15059__p33213195213">A: You can use the non-window aggregation method. That is, convert the time field into a window value, and then use <strong id="dli_08_15059__b27667344992527">GROUP BY</strong> to perform aggregation based on the window value.</p>
<p id="dli_08_15059__p4218115919240">For example, you can use the following script to collect statistics on the number of orders per minute (<strong id="dli_08_15059__b214133868192527">order_time</strong> indicates the order time, in the string format):</p>
<pre class="screen" id="dli_08_15059__screen16550125612260">insert into printSink select DATE_FORMAT(order_time, 'yyyy-MM-dd HH:mm'), count(*) from mysqlCdcSource group by DATE_FORMAT(order_time, 'yyyy-MM-dd HH:mm');</pre>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15027.html">Connectors</a></div>
</div>
</div>
View Git Blame Copy Permalink