lboka/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_24753.html
Yang, Tong 48706b7552 MRS COMP-LTS 320-lts.1 version 
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2024-04-12 12:51:10 +00:00

239 lines
22 KiB
HTML
Raw Blame History

<a name="mrs_01_24753"></a><a name="mrs_01_24753"></a>
<h1 class="topictitle1">Synchronizing Data from Hudi to DWS</h1>
<div id="body0000001583150757"><div class="section" id="mrs_01_24753__section7722164042318"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24753__p184673811319">This section describes how to import data from Hudi to DWS by using the CDLService web UI of a cluster with Kerberos authentication enabled.</p>
</div>
<div class="section" id="mrs_01_24753__section1438820494231"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24753__ul2041871362311"><li id="mrs_01_24753__li12418213122313">The CDL and Hudi services have been installed in a cluster and are running properly.</li><li id="mrs_01_24753__li362910581468">The prerequisites for the DWS database have been met. For details, see <a href="mrs_01_24124.html#mrs_01_24124__li6127144552014">Prerequisites for the DWS database</a>.</li><li id="mrs_01_24753__li1441819135232">You have created a human-machine user, for example, <strong id="mrs_01_24753__b84981214133013">cdluser</strong>, added the user to user groups <strong id="mrs_01_24753__b95043145307">cdladmin</strong> (primary group), <strong id="mrs_01_24753__b250451417302">hadoop</strong>, <strong id="mrs_01_24753__b95041814183014">kafka</strong>, and <strong id="mrs_01_24753__b750514141307">supergroup</strong>, and associated the user with the <strong id="mrs_01_24753__b850518145303">System_administrator</strong> role on FusionInsight Manager.</li></ul>
</div>
<div class="section" id="mrs_01_24753__section19362472245"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24753__ol18871133225317"><li id="mrs_01_24753__li1365211683614"><span>Log in to FusionInsight Manager as user <strong id="mrs_01_24753__b1629103518169">cdluser</strong> (change the password upon your first login), choose <strong id="mrs_01_24753__b2062919357163">Cluster</strong> &gt; <strong id="mrs_01_24753__b186301135191614">Services</strong> &gt; <strong id="mrs_01_24753__b36301535151611">CDL</strong>, and click the link next to <strong id="mrs_01_24753__b1630173512169">CDLService UI</strong> to go to the CDLService web UI.</span></li><li id="mrs_01_24753__li3618121145616"><span>Choose <strong id="mrs_01_24753__b18228133921614">Link Management</strong> and click <strong id="mrs_01_24753__b1322812390163">Add Link</strong>. In the displayed dialog box, set parameters for adding the <strong id="mrs_01_24753__b192291539181618">dws</strong> and <strong id="mrs_01_24753__b222973910166">hudi</strong> links by referring to the following tables.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24753__table134120364291" frame="border" border="1" rules="all"><caption><b>Table 1 </b>DWS data link parameters</caption><thead align="left"><tr id="mrs_01_24753__row2041836162918"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.1.2.3.1.1"><p id="mrs_01_24753__p4411136172915">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.1.2.3.1.2"><p id="mrs_01_24753__p19411736142919">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24753__row10411236152920"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24753__p154119368298">Link Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24753__p154153632911">dws</p>
</td>
</tr>
<tr id="mrs_01_24753__row16426365294"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24753__p44220365290">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24753__p104253620293">dwstest</p>
</td>
</tr>
<tr id="mrs_01_24753__row442636152915"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24753__p9427368295">Host</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24753__p17303172811553">10.10.10.10</p>
</td>
</tr>
<tr id="mrs_01_24753__row11423363293"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24753__p10421364293">Port</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24753__p17303122811553">8000</p>
</td>
</tr>
<tr id="mrs_01_24753__row11507173716396"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24753__p10507153793919">DB Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24753__p14507637153910">dwsdb</p>
</td>
</tr>
<tr id="mrs_01_24753__row164263622915"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24753__p1442636162910">User</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24753__p4916845152114">dbuser</p>
</td>
</tr>
<tr id="mrs_01_24753__row104317366296"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24753__p194314367297">Password</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24753__p391712458215"><em id="mrs_01_24753__i6988052161617">Password of the <strong id="mrs_01_24753__b21821257191615">dbuser</strong> user</em></p>
</td>
</tr>
<tr id="mrs_01_24753__row124313614299"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24753__p74353613299">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24753__p184383615296">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24753__table1437691715220" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Hudi data link parameters</caption><thead align="left"><tr id="mrs_01_24753__row1437618171029"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.2.2.3.1.1"><p id="mrs_01_24753__p137651713218">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.2.2.3.1.2"><p id="mrs_01_24753__p1237720171128">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24753__row33815178212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24753__p153814179219">Link Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24753__p938112176210">hudi</p>
</td>
</tr>
<tr id="mrs_01_24753__row193811117822"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24753__p138161715211">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24753__p1738113173219">hudilink</p>
</td>
</tr>
<tr id="mrs_01_24753__row83811417425"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24753__p125527208312">Storage Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24753__p35331203318">hdfs</p>
</td>
</tr>
<tr id="mrs_01_24753__row736874119319"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24753__p1814611463913">Auth KeytabFile</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24753__p137021903412">/opt/Bigdata/third_lib/CDL/user_libs/cdluser.keytab</p>
</td>
</tr>
<tr id="mrs_01_24753__row936915411130"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24753__p14146546699">Principal</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24753__p1414619465918">cdluser</p>
</td>
</tr>
<tr id="mrs_01_24753__row938313177210"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24753__p1938310171727">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24753__p5383161716218">xxx</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="mrs_01_24753__li1829512373488"><span>After the parameters are configured, click <strong id="mrs_01_24753__b77842641717">Test</strong> to check whether the data link is normal.</span><p><p id="mrs_01_24753__p13530140134817">After the test is successful, click <strong id="mrs_01_24753__b15425610161716">OK</strong>.</p>
</p></li><li id="mrs_01_24753__li976974910717"><span>(Optional) Choose <strong id="mrs_01_24753__en-us_topic_0000001446993949_b45961936164618">ENV Management</strong> and click <strong id="mrs_01_24753__en-us_topic_0000001446993949_b03551454164613">Add Env</strong>. In the displayed dialog box, configure the parameters based on the following table.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24753__en-us_topic_0000001446993949_table54201432181218" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Parameters for adding an ENV</caption><thead align="left"><tr id="mrs_01_24753__en-us_topic_0000001446993949_row10418153291216"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.1.2.3.1.1"><p id="mrs_01_24753__en-us_topic_0000001446993949_p18418032161213">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.1.2.3.1.2"><p id="mrs_01_24753__en-us_topic_0000001446993949_p441810320123">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24753__en-us_topic_0000001446993949_row7419193251216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p1141815324123">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p14419183241213">test-env</p>
</td>
</tr>
<tr id="mrs_01_24753__en-us_topic_0000001446993949_row04191332161212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p18419133214122">Driver Memory</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p341923215124">1 GB</p>
</td>
</tr>
<tr id="mrs_01_24753__en-us_topic_0000001446993949_row1468718181894"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p156889181495">Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p106881618096">spark</p>
</td>
</tr>
<tr id="mrs_01_24753__en-us_topic_0000001446993949_row2419632151212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p941918323125">Executor Memory</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p1419832121219">1 GB</p>
</td>
</tr>
<tr id="mrs_01_24753__en-us_topic_0000001446993949_row20419143211122"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p041933213124">Executor Cores</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p1341983217125">1</p>
</td>
</tr>
<tr id="mrs_01_24753__en-us_topic_0000001446993949_row341915321123"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p194191032181217">Number Executors</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p841933251210">1</p>
</td>
</tr>
<tr id="mrs_01_24753__en-us_topic_0000001446993949_row77610527328"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p1525831203317">Queue</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p17675211325">-</p>
</td>
</tr>
<tr id="mrs_01_24753__en-us_topic_0000001446993949_row34191032171216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p94191032181210">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24753__en-us_topic_0000001446993949_p5419193231211">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24753__en-us_topic_0000001446993949_p145511354104314">Click <strong id="mrs_01_24753__en-us_topic_0000001446993949_b1436468274744">OK</strong>.</p>
</p></li><li id="mrs_01_24753__li8419191320242"><span>Choose <strong id="mrs_01_24753__b1296983012170">Job Management</strong> &gt; <strong id="mrs_01_24753__b1997513041710">Data synchronization task</strong> and click <strong id="mrs_01_24753__b10975330151714">Add Job</strong>. In the displayed dialog box, set parameters and click <strong id="mrs_01_24753__b3975143010173">Next</strong>.</span><p><div class="p" id="mrs_01_24753__p1442915568513">Parameters are as follows.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24753__table1379713112014" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_24753__row13797011905"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.1.1.3.1.1"><p id="mrs_01_24753__p17797411309">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.1.1.3.1.2"><p id="mrs_01_24753__p137971011309">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24753__row1679841118018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.1 "><p id="mrs_01_24753__p391613171005">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.2 "><p id="mrs_01_24753__p149141171404">job_huditodws</p>
</td>
</tr>
<tr id="mrs_01_24753__row779916111602"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.1 "><p id="mrs_01_24753__p83069155152">Desc</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.2 "><p id="mrs_01_24753__p14306191511512">-</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</p></li><li id="mrs_01_24753__li2737114320203"><span>Configure Hudi job parameters.</span><p><ol type="a" id="mrs_01_24753__ol14971759152111"><li id="mrs_01_24753__li637518545212">On the <strong id="mrs_01_24753__b11593164571715">Job Management</strong> page, drag the <strong id="mrs_01_24753__b159974517171">hudi</strong> icon in the Source area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Configure parameters based on the following table.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24753__table1415410186546" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Source Hudi job parameters</caption><thead align="left"><tr id="mrs_01_24753__row2153121875414"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.3.2.3.1.1"><p id="mrs_01_24753__p191530188543">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.3.2.3.1.2"><p id="mrs_01_24753__p2153818135411">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24753__row81531218125420"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24753__p21538187547">Link</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24753__p1615361825417">hudilink</p>
</td>
</tr>
<tr id="mrs_01_24753__row01531318175417"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24753__p3153121835411">Interval</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24753__p71531018205410">10</p>
</td>
</tr>
<tr id="mrs_01_24753__row115410183541"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24753__p7154121815414">Table Info</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24753__p315451805416">{"table1":[{"source.database":"dwsdb","source.tablename":"tabletest","target.tablename":"tabletest"}]}</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24753__p1015411817544"><span><img id="mrs_01_24753__image3154101855419" src="en-us_image_0000001532632204.png"></span></p>
</li><li id="mrs_01_24753__li5602185288">Click <strong id="mrs_01_24753__b7657318111817">OK</strong>. The Hudi job parameters are configured.</li></ol>
</p></li><li id="mrs_01_24753__li1736682513364"><span>Configure DWS job parameters.</span><p><ol type="a" id="mrs_01_24753__ol5126919103712"><li id="mrs_01_24753__li1838281915919">On the <strong id="mrs_01_24753__b938183313182">Job Management</strong> page, drag the <strong id="mrs_01_24753__b63993381818">dws</strong> icon on the left to the editing area on the right and double-click the icon to go to the DWS job configuration page. Configure parameters based on the following table.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24753__table4414181314558" frame="border" border="1" rules="all"><caption><b>Table 5 </b>DWS job parameters</caption><thead align="left"><tr id="mrs_01_24753__row44137137550"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.7.2.1.1.3.2.3.1.1"><p id="mrs_01_24753__p13413131365512">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.7.2.1.1.3.2.3.1.2"><p id="mrs_01_24753__p641313135553">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24753__row1341301312559"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24753__p16413191325514">Link</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24753__p841351311559">dwstest</p>
</td>
</tr>
<tr id="mrs_01_24753__row241451355520"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24753__p74136139552">Query Timeout</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24753__p1841351305513">180000</p>
</td>
</tr>
<tr id="mrs_01_24753__row14414313165511"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24753__p164141913115518">Batch Size</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24753__p1341421355510">10</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24753__p1060623111242"></p>
<div class="note" id="mrs_01_24753__note1298221362812"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24753__ul723010234287"><li id="mrs_01_24753__li172303233286">Data can be synchronized from Hudi to GaussDB(DWS) only when both Hudi and GaussDB(DWS) contain the <strong id="mrs_01_24753__b3301562465">precombine</strong> field.</li><li id="mrs_01_24753__li11134193618295">GaussDB(DWS) tables must contain the <strong id="mrs_01_24753__b2042223354612">precombine</strong> field and the primary key.</li><li id="mrs_01_24753__li13554141016309">By default, the Hudi built-in field <strong id="mrs_01_24753__b86618345305">_hoodie_event_time</strong> is used. If this field is not used, <strong id="mrs_01_24753__b1341163419519">enable.sink.precombine</strong> must be specified. An example is as follows:<p id="mrs_01_24753__p96307591301"><span><img id="mrs_01_24753__image159761611145914" src="en-us_image_0000001532472732.png"></span></p>
</li></ul>
</div></div>
</li><li id="mrs_01_24753__li16689025123917">Click <strong id="mrs_01_24753__b13287287194">OK</strong>.</li></ol>
</p></li><li id="mrs_01_24753__li032595183911"><span>Drag the two icons to associate the job parameters and click <strong id="mrs_01_24753__b10658181015190">Save</strong>. The job configuration is complete.</span><p><p id="mrs_01_24753__p51802192443"><span><img id="mrs_01_24753__image23081221540" src="en-us_image_0000001532791952.png"></span></p>
</p></li><li id="mrs_01_24753__li7968152124413"><span>In the job list on the <strong id="mrs_01_24753__b81387259194">Job Management</strong> page, locate the created job, click <strong id="mrs_01_24753__b814414255193">Start</strong> in the <strong id="mrs_01_24753__b16145162531916">Operation</strong> column, and wait until the job is started.</span><p><p id="mrs_01_24753__p29726217442">Check whether the data transmission takes effect, for example, insert data into the Hudi table and view the content of the file imported to DWS.</p>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24240.html">Common CDL Jobs</a></div>
</div>
</div>
View Git Blame Copy Permalink