vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0388.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

251 lines
28 KiB
HTML
Raw Blame History

<a name="dli_08_0388"></a><a name="dli_08_0388"></a>
<h1 class="topictitle1">Postgres CDC Source Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section16834144252514"><h4 class="sectiontitle">Function</h4><p id="dli_08_0388__en-us_topic_0000001310215785_p4587104618253">The Postgres CDC source table, that is, Postgres streaming source table, is used to read the full snapshot data and changed data of the PostgreSQL database in sequence. The exactly-once processing semantics is used to ensure data accuracy even if a failure occurs.</p>
</div>
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section4375753152514"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0388__en-us_topic_0000001310215785_ul1170451184015"><li id="dli_08_0388__en-us_topic_0000001310215785_li107041511114015">The PostgreSQL version be 9.6, 10, 11, or 12.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li18597171744011">An enhanced datasource connection with the database has been established, so that you can configure security group rules as required.
</li></ul>
</div>
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section1852611002610"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0388__en-us_topic_0000001310215785_ul1552562641320"><li id="dli_08_0388__en-us_topic_0000001310215785_li5709020162317">When you create a Flink OpenSource SQL job, set <strong id="dli_08_0388__en-us_topic_0000001310215785_b1124516290305">Flink Version</strong> to <strong id="dli_08_0388__en-us_topic_0000001310215785_b11246102911303">1.12</strong> on the <strong id="dli_08_0388__en-us_topic_0000001310215785_b1124652911308">Running Parameters</strong> tab of the job editing page, select <strong id="dli_08_0388__en-us_topic_0000001310215785_b1824617294303">Save Job Log</strong>, and set the OBS bucket for saving job logs.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li4525202631311">The PostgreSQL version cannot be earlier than PostgreSQL 11.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li118251538161814">If operations such as update will be performed on the Postgres table, you need to run the following statement in PostgreSQL. Note: Replace <strong id="dli_08_0388__en-us_topic_0000001310215785_b9253442103112">test.cdc_order</strong> with the actual database and table.<pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen8340431181914">ALTER TABLE <em id="dli_08_0388__en-us_topic_0000001310215785_i4560579314">test.cdc_order</em> REPLICA IDENTITY FULL</pre>
</li><li id="dli_08_0388__en-us_topic_0000001310215785_li12763152295310">Before creating the PostgreSQL CDC source table, check whether the current PostgreSQL contains the default plug-in. You can run the following statement in PostgreSQL to query the current plug-ins:<pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen18362631325">SELECT name FROM pg_available_extensions;</pre>
<p id="dli_08_0388__en-us_topic_0000001310215785_p1952477422">If the default plug-in <strong id="dli_08_0388__en-us_topic_0000001310215785_b27966519338">decoderbufs</strong> is not available, you need to set the <strong id="dli_08_0388__en-us_topic_0000001310215785_b47548187339">decoding.plugin.name</strong> parameter to specify an existing plug-in in PostgreSQL when creating the PostgreSQL CDC source table.</p>
</li></ul>
</div>
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section107021158182914"><h4 class="sectiontitle">Syntax</h4><pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen123120613307">create table postgresCdcSource (
attr_name attr_type
(',' attr_name attr_type)*
(','PRIMARY KEY (attr_name, ...) NOT ENFORCED)
)
with (
'connector' = 'postgres-cdc',
'hostname' = '<em id="dli_08_0388__en-us_topic_0000001310215785_i15915212952">PostgresHostname</em>',
'username' = '<em id="dli_08_0388__en-us_topic_0000001310215785_i153881625513">Postgres</em><em id="dli_08_0388__en-us_topic_0000001310215785_i15797315954">Username</em>',
'password' = '<em id="dli_08_0388__en-us_topic_0000001310215785_i5620648516">Postgres</em><em id="dli_08_0388__en-us_topic_0000001310215785_i1726512191058">Password</em>',
'database-name' = '<em id="dli_08_0388__en-us_topic_0000001310215785_i853666655">Postgres</em><em id="dli_08_0388__en-us_topic_0000001310215785_i1758118211558">DatabaseName</em>',
'schema-name' = '<em id="dli_08_0388__en-us_topic_0000001310215785_i15361981550">Postgres</em><em id="dli_08_0388__en-us_topic_0000001310215785_i1668192315510">SchemaName</em>',
'table-name' = '<em id="dli_08_0388__en-us_topic_0000001310215785_i187802096512">Postgres</em><em id="dli_08_0388__en-us_topic_0000001310215785_i151684267515">TableName</em>'
);</pre>
</div>
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section183609813304"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0388__en-us_topic_0000001310215785_table28274351306" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0388__en-us_topic_0000001310215785_row19827735103012"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0388__en-us_topic_0000001310215785_p108275356303">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="6.909999999999999%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0388__en-us_topic_0000001310215785_p98271635103012">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10.67%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0388__en-us_topic_0000001310215785_p1182773510308">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.36%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0388__en-us_topic_0000001310215785_p68271352303">Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="53.059999999999995%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0388__en-us_topic_0000001310215785_p28271035113017">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0388__en-us_topic_0000001310215785_row1782717352302"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1827173512307">connector</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p782773518306">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p18827535163010">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1982773518308">String</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1282793511301">Connector to be used. Set this parameter to <strong id="dli_08_0388__en-us_topic_0000001310215785_b882271703516">postgres-cdc</strong>.</p>
</td>
</tr>
<tr id="dli_08_0388__en-us_topic_0000001310215785_row2082763519302"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1827183503018">hostname</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p16827123513014">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p21469397575">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p148271935153016">String</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p28271935153017">IP address or hostname of the Postgres database.</p>
</td>
</tr>
<tr id="dli_08_0388__en-us_topic_0000001310215785_row98271235183015"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p2827335163018">username</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p168273358301">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p724254512611">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p10827113503014">String</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1582743510305">Username of the Postgres database.</p>
</td>
</tr>
<tr id="dli_08_0388__en-us_topic_0000001310215785_row19827135183015"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p13827113513018">password</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1182793533011">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p12340540205717">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1682733511307">String</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p118271535133020">Password of the Postgres database.</p>
</td>
</tr>
<tr id="dli_08_0388__en-us_topic_0000001310215785_row108271235173012"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1082773510303">database-name</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p8827235123012">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p15891154017576">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p68273350306">String</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p7827735183010">Database name.</p>
</td>
</tr>
<tr id="dli_08_0388__en-us_topic_0000001310215785_row38271135103017"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p128271435113011">schema-name</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p5827113512308">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p191792481061">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1082753516307">String</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p16827113533015">Postgres schema name.</p>
<p id="dli_08_0388__en-us_topic_0000001310215785_p7819344019">The schema name supports regular expressions to read data from multiple schemas. For example, <strong id="dli_08_0388__en-us_topic_0000001310215785_b1967902715385">test(.)*</strong> indicates all schema names starting with <strong id="dli_08_0388__en-us_topic_0000001310215785_b171121235193816">test</strong>.</p>
</td>
</tr>
<tr id="dli_08_0388__en-us_topic_0000001310215785_row2827103523017"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1827235173011">table-name</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p482763520308">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p13141104220579">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p14827235133017">String</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p8827835143015">Postgres table name.</p>
<p id="dli_08_0388__en-us_topic_0000001310215785_p17857164118012">The table name supports regular expressions to read data from multiple tables. For example, <strong id="dli_08_0388__en-us_topic_0000001310215785_b7622551143819">cdc_order(.)*</strong> indicates all table names starting with <strong id="dli_08_0388__en-us_topic_0000001310215785_b1622195113383">cdc_order</strong>.</p>
</td>
</tr>
<tr id="dli_08_0388__en-us_topic_0000001310215785_row78271358305"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p982743514308">port</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p2827173515302">No</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1982763543014">5432</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p13827163519306">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p16827133513304">Port number of the Postgres database.</p>
</td>
</tr>
<tr id="dli_08_0388__en-us_topic_0000001310215785_row14827183514303"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p28271635143017">decoding.plugin.name</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1828163543011">No</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1682843543014">decoderbufs</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p1382811353302">String</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p63241787110">Determined based on the plug-in that is installed in the PostgreSQL database. The value can be:</p>
<ul id="dli_08_0388__en-us_topic_0000001310215785_ul53241682015"><li id="dli_08_0388__en-us_topic_0000001310215785_li1324148816">decoderbufs (default)</li><li id="dli_08_0388__en-us_topic_0000001310215785_li11324108119">wal2json</li><li id="dli_08_0388__en-us_topic_0000001310215785_li5324148419">wal2json_rds</li><li id="dli_08_0388__en-us_topic_0000001310215785_li332410817119">wal2json_streaming</li><li id="dli_08_0388__en-us_topic_0000001310215785_li13324787117">wal2json_rds_streaming</li><li id="dli_08_0388__en-us_topic_0000001310215785_li10324128313">pgoutput</li></ul>
</td>
</tr>
<tr id="dli_08_0388__en-us_topic_0000001310215785_row5828163523019"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0388__en-us_topic_0000001310215785_p19828183510306">debezium.*</p>
</td>
<td class="cellrowborder" valign="top" width="6.909999999999999%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0388__en-us_topic_0000001310215785_p6828735123014">No</p>
</td>
<td class="cellrowborder" valign="top" width="10.67%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0388__en-us_topic_0000001310215785_p164445413818">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.36%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0388__en-us_topic_0000001310215785_p382853513307">String</p>
</td>
<td class="cellrowborder" valign="top" width="53.059999999999995%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0388__en-us_topic_0000001310215785_p582813519305">Fine-grained control over the behavior of Debezium clients, for example, <strong id="dli_08_0388__en-us_topic_0000001310215785_b69813915918">'debezium.snapshot.mode' = 'never'</strong>.</p>
<p id="dli_08_0388__en-us_topic_0000001310215785_p1713111112100">You are advised to set the <strong id="dli_08_0388__en-us_topic_0000001310215785_b5421144174416">debezium.slot.name</strong> parameter for each table to avoid the following error: "PSQLException: ERROR: replication slot "debezium" is active for PID 974"</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section2074110411437"><h4 class="sectiontitle">Example</h4><p id="dli_08_0388__en-us_topic_0000001310215785_p15682201721018">In this example, Postgres-CDC is used to read data from RDS for PostgreSQL in real time and write the data to the Print result table. The procedure is as follows (PostgreSQL 11.11 is used in this example):</p>
<ol id="dli_08_0388__en-us_topic_0000001310215785_ol9422103714102"><li id="dli_08_0388__en-us_topic_0000001310215785_li6127741153217">Create an enhanced datasource connection in the VPC and subnet where PostgreSQL locates, and bind the connection to the required Flink elastic resource pool.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li1233002211514">Set PostgreSQL security groups and add inbound rules to allow access from the Flink queue. Test the connectivity using the PostgreSQL address. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li1342219376104">In PostgreSQL, create database <strong id="dli_08_0388__en-us_topic_0000001310215785_b7255623124920">flink</strong> and schema <strong id="dli_08_0388__en-us_topic_0000001310215785_b5660133317496">test</strong>.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li1242211376106">Create table <strong id="dli_08_0388__en-us_topic_0000001310215785_b7372152144915">cdc_order</strong> in the schema <strong id="dli_08_0388__en-us_topic_0000001310215785_b1328711225019">test</strong> of database <strong id="dli_08_0388__en-us_topic_0000001310215785_b56272132508">flink</strong> in PostgreSQL.<pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen9883103014217">create table test.cdc_order(
order_id VARCHAR,
order_channel VARCHAR,
order_time VARCHAR,
pay_amount FLOAT8,
real_pay FLOAT8,
pay_time VARCHAR,
user_id VARCHAR,
user_name VARCHAR,
area_id VARCHAR,
primary key(order_id)
);</pre>
</li><li id="dli_08_0388__en-us_topic_0000001310215785_li1542210373102">Run the following SQL statement in PostgreSQL. If you do not run this statement, an error will be reported when the Flink job is executed. For details, see the error message in <a href="#dli_08_0388__en-us_topic_0000001310215785_li197238199359">FAQ</a>.<pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen4397125413105">ALTER TABLE test.cdc_order REPLICA IDENTITY FULL</pre>
</li><li id="dli_08_0388__en-us_topic_0000001310215785_li2511193592015">Create a Flink OpenSource SQL job. Enter the following job script and submit the job.<div class="p" id="dli_08_0388__en-us_topic_0000001310215785_p1093521182115"><a name="dli_08_0388__en-us_topic_0000001310215785_li2511193592015"></a><a name="en-us_topic_0000001310215785_li2511193592015"></a>When you create a job, set <strong id="dli_08_0388__en-us_topic_0000001310215785_b84351914175220">Flink Version</strong> to <strong id="dli_08_0388__en-us_topic_0000001310215785_b843514149528">1.12</strong> on the <strong id="dli_08_0388__en-us_topic_0000001310215785_b743581495212">Running Parameters</strong> tab. Select <strong id="dli_08_0388__en-us_topic_0000001310215785_b11436014195211">Save Job Log</strong>, and specify the OBS bucket for saving job logs. <strong id="dli_08_0388__en-us_topic_0000001310215785_b84341712527">Change the values of the parameters in bold as needed in the following script.</strong><pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen9151357162019">create table postgresCdcSource(
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id STRING,
primary key (order_id) not enforced
) with (
'connector' = 'postgres-cdc',
'hostname' = '<em id="dli_08_0388__en-us_topic_0000001310215785_i1575592010216"><strong id="dli_08_0388__en-us_topic_0000001310215785_b53724481217">Postgres</strong></em><em id="dli_08_0388__en-us_topic_0000001310215785_i17143163452110"><strong id="dli_08_0388__en-us_topic_0000001310215785_b937214815215">Hostname</strong></em>',
'username' = '<em id="dli_08_0388__en-us_topic_0000001310215785_i1515663016218"><strong id="dli_08_0388__en-us_topic_0000001310215785_b7201145092113">Postgres</strong></em><em id="dli_08_0388__en-us_topic_0000001310215785_i170317368216"><strong id="dli_08_0388__en-us_topic_0000001310215785_b1620225014210">Username</strong></em>',
'password' = '<em id="dli_08_0388__en-us_topic_0000001310215785_i952083112211"><strong id="dli_08_0388__en-us_topic_0000001310215785_b20957195111214">Postgres</strong></em><em id="dli_08_0388__en-us_topic_0000001310215785_i1861733815211"><strong id="dli_08_0388__en-us_topic_0000001310215785_b209588512218">Password</strong></em>',
'database-name' = 'flink',
'schema-name' = 'test',
'table-name' = 'cdc_order'
);
create table printSink(
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id STRING,
primary key(order_id) not enforced
) with (
'connector' = 'print'
);
insert into printSink select * from postgresCdcSource;</pre>
</div>
</li><li id="dli_08_0388__en-us_topic_0000001310215785_li15422173781019">Run the following command in PostgreSQL:<pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen11849141112415">insert into test.cdc_order
(order_id,
order_channel,
order_time,
pay_amount,
real_pay,
pay_time,
user_id,
user_name,
area_id) values
('202103241000000001', 'webShop', '2021-03-24 10:00:00', '100.00', '100.00', '2021-03-24 10:02:03', '0001', 'Alice', '330106'),
('202103251202020001', 'miniAppShop', '2021-03-25 12:02:02', '60.00', '60.00', '2021-03-25 12:03:00', '0002', 'Bob', '330110');
update test.cdc_order set order_channel = 'webShop' where order_id = '202103251202020001';
delete from test.cdc_order where order_id = '202103241000000001';</pre>
</li><li id="dli_08_0388__en-us_topic_0000001310215785_li12422123717106">Perform the following operations to view the data result in the <strong id="dli_08_0388__en-us_topic_0000001310215785_b7249174345213">taskmanager.out</strong> file:<ol type="a" id="dli_08_0388__en-us_topic_0000001310215785_ol864115198285"><li id="dli_08_0388__en-us_topic_0000001310215785_li10901621122819">Log in to the DLI console. In the navigation pane, choose <strong id="dli_08_0388__en-us_topic_0000001310215785_b11676184410521">Job Management</strong> &gt; <strong id="dli_08_0388__en-us_topic_0000001310215785_b1367784415220">Flink Jobs</strong>.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li1912163912282">Click the name of the corresponding Flink job, choose <strong id="dli_08_0388__en-us_topic_0000001310215785_b316884618522">Run Log</strong>, click <strong id="dli_08_0388__en-us_topic_0000001310215785_b181691346155218">OBS Bucket</strong>, and locate the folder of the log you want to view according to the date.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li0641191914285">Go to the folder of the date, find the folder whose name contains <strong id="dli_08_0388__en-us_topic_0000001310215785_b5211204816527">taskmanager</strong>, download the <strong id="dli_08_0388__en-us_topic_0000001310215785_b121164865216">taskmanager.out</strong> file, and view result logs.</li></ol>
<p id="dli_08_0388__en-us_topic_0000001310215785_p1229191018247">The data result is as follows:</p>
<pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen17252518182418">+I(202103241000000001,webShop,2021-03-24 10:00:00,100.0,100.0,2021-03-24 10:02:03,0001,Alice,330106)
+I(202103251202020001,miniAppShop,2021-03-25 12:02:02,60.0,60.0,2021-03-25 12:03:00,0002,Bob,330110)
-U(202103251202020001,miniAppShop,2021-03-25 12:02:02,60.0,60.0,2021-03-25 12:03:00,0002,Bob,330110)
+U(202103251202020001,webShop,2021-03-25 12:02:02,60.0,60.0,2021-03-25 12:03:00,0002,Bob,330110)
-D(202103241000000001,webShop,2021-03-24 10:00:00,100.0,100.0,2021-03-24 10:02:03,0001,Alice,330106)</pre>
</li></ol>
</div>
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section16770656182416"><h4 class="sectiontitle">FAQ</h4><ul id="dli_08_0388__en-us_topic_0000001310215785_ul2562105152517"><li id="dli_08_0388__en-us_topic_0000001310215785_li19683190141314">Q: What should I do if the Flink job execution fails and the log contains the following error information?<pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen8747852192516">org.postgresql.util.PSQLException: ERROR: logical decoding requires wal_level &gt;= logical</pre>
</li><li id="dli_08_0388__en-us_topic_0000001310215785_li656265142511">A: Change the value of <strong id="dli_08_0388__en-us_topic_0000001310215785_b1850111235314">wal_level</strong> to <strong id="dli_08_0388__en-us_topic_0000001310215785_b122484156532">logical</strong> and restart the PostgreSQL database. <p id="dli_08_0388__en-us_topic_0000001310215785_p1812650115714">After modifying the PostgreSQL parameter, restart the RDS PostgreSQL instance for the modification to take effect.</p>
</li></ul>
</div>
<ul id="dli_08_0388__en-us_topic_0000001310215785_ul1511811401778"><li id="dli_08_0388__en-us_topic_0000001310215785_li197238199359"><a name="dli_08_0388__en-us_topic_0000001310215785_li197238199359"></a><a name="en-us_topic_0000001310215785_li197238199359"></a>Q: What should I do if the Flink job execution fails and the log contains the following error information?<pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen03271515486">java.lang.IllegalStateException: The "before" field of UPDATE/DELETE message is null, please check the Postgres table has been set REPLICA IDENTITY to FULL level. You can update the setting by running the command in Postgres '<strong id="dli_08_0388__en-us_topic_0000001310215785_b061185302714">ALTER TABLE test.cdc_order REPLICA IDENTITY FULL</strong>'. </pre>
<p id="dli_08_0388__en-us_topic_0000001310215785_p1172692103512">A: If a similar error is reported in the run log, run the <strong id="dli_08_0388__en-us_topic_0000001310215785_b9186544145513">ALTER TABLE test.cdc_order REPLICA IDENTITY FULL</strong> statement in PostgreSQL.</p>
</li></ul>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0381.html">Creating Source Tables</a></div>
</div>
</div>
View Git Blame Copy Permalink