yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
a2e28c76231c99d8bbac87147becbb2b7e6be32e
doc-exports/docs/dws/dev/dws_04_1072.html
luhuayi 177cd61a57 DWS DEVG 910.211 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: luhuayi <luhuayi@huawei.com>
Co-committed-by: luhuayi <luhuayi@huawei.com>
2025-05-05 07:44:03 +00:00

158 lines
14 KiB
HTML
Raw Blame History

<a name="EN-US_TOPIC_0000001811490465"></a><a name="EN-US_TOPIC_0000001811490465"></a>
<h1 class="topictitle1">Hudi User Interfaces</h1>
<div id="body0000001546395652"><div class="section" id="EN-US_TOPIC_0000001811490465__section7895194316152"><h4 class="sectiontitle">Querying Real-Time Views and Incremental Views</h4><p id="EN-US_TOPIC_0000001811490465__p147362491715">GaussDB(DWS) provides table-level parameters similar to spark-sql to support real-time and incremental views.</p>
<p id="EN-US_TOPIC_0000001811490465__p1130494619151">The parameters are described as follows. Replace <strong id="EN-US_TOPIC_0000001811490465__b58069173416">SCHEMA.FOREIGN_TABLE</strong> with the actual schema name and foreign table name.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="EN-US_TOPIC_0000001811490465__table34765416181" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters for querying real-time views and incremental views</caption><thead align="left"><tr id="EN-US_TOPIC_0000001811490465__row84795471815"><th align="left" class="cellrowborder" valign="top" width="43.064306430643065%" id="mcps1.3.1.4.2.4.1.1"><p id="EN-US_TOPIC_0000001811490465__p447954131818">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="21.012101210121013%" id="mcps1.3.1.4.2.4.1.2"><p id="EN-US_TOPIC_0000001811490465__p18471254181810">Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="35.923592359235926%" id="mcps1.3.1.4.2.4.1.3"><p id="EN-US_TOPIC_0000001811490465__p247454121810">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="EN-US_TOPIC_0000001811490465__row34735491816"><td class="cellrowborder" rowspan="2" valign="top" width="43.064306430643065%" headers="mcps1.3.1.4.2.4.1.1 "><p id="EN-US_TOPIC_0000001811490465__p842313717235">hoodie.SCHEMA.FOREIGN_TABLE.consume.mode</p>
</td>
<td class="cellrowborder" valign="top" width="21.012101210121013%" headers="mcps1.3.1.4.2.4.1.2 "><p id="EN-US_TOPIC_0000001811490465__p1759216432216">SNAPSHOT</p>
</td>
<td class="cellrowborder" valign="top" width="35.923592359235926%" headers="mcps1.3.1.4.2.4.1.3 "><p id="EN-US_TOPIC_0000001811490465__p125921647222">Queries the real-time view.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row64855401817"><td class="cellrowborder" valign="top" headers="mcps1.3.1.4.2.4.1.1 "><p id="EN-US_TOPIC_0000001811490465__p6592647224">INCREMENTAL</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.1.4.2.4.1.2 "><p id="EN-US_TOPIC_0000001811490465__p14592648221">Queries the incremental view.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row104885418182"><td class="cellrowborder" valign="top" width="43.064306430643065%" headers="mcps1.3.1.4.2.4.1.1 "><p id="EN-US_TOPIC_0000001811490465__p149888496203">hoodie.SCHEMA.FOREIGN_TABLE.consume. start.timestamp</p>
</td>
<td class="cellrowborder" valign="top" width="21.012101210121013%" headers="mcps1.3.1.4.2.4.1.2 "><p id="EN-US_TOPIC_0000001811490465__p183673223214">hudi timestamp</p>
</td>
<td class="cellrowborder" valign="top" width="35.923592359235926%" headers="mcps1.3.1.4.2.4.1.3 "><p id="EN-US_TOPIC_0000001811490465__p11367102272117">Specifies the start commit of incremental synchronization.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row15481354181818"><td class="cellrowborder" valign="top" width="43.064306430643065%" headers="mcps1.3.1.4.2.4.1.1 "><p id="EN-US_TOPIC_0000001811490465__p99881549172016">hoodie.SCHEMA.FOREIGN_TABLE.consume. ending.timestamp</p>
</td>
<td class="cellrowborder" valign="top" width="21.012101210121013%" headers="mcps1.3.1.4.2.4.1.2 "><p id="EN-US_TOPIC_0000001811490465__p17367152211210">hudi timestamp</p>
</td>
<td class="cellrowborder" valign="top" width="35.923592359235926%" headers="mcps1.3.1.4.2.4.1.3 "><p id="EN-US_TOPIC_0000001811490465__p1236782218217">Specifies the end commit of incremental synchronization. If this parameter is not specified, the latest commit is used.</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="EN-US_TOPIC_0000001811490465__note850972392412"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="EN-US_TOPIC_0000001811490465__ul185852217250"><li id="EN-US_TOPIC_0000001811490465__li14585132162514">The preceding parameters can be set by running the set command and are valid only in the current session. You can run the reset command to restore the default values.</li><li id="EN-US_TOPIC_0000001811490465__li195851721256">You can use the system function <strong id="EN-US_TOPIC_0000001811490465__b153431341164113">pg_catalog.pg_show_custom_settings()</strong> to query the parameter setting details.</li><li id="EN-US_TOPIC_0000001811490465__li131624911228">When querying the incremental view of the MOR table, you need to use the <strong id="EN-US_TOPIC_0000001811490465__b33317513426">WHERE</strong> condition to filter the<strong id="EN-US_TOPIC_0000001811490465__b1470111711422"> _hoodie_commit_time</strong> column to prevent the log file data that is not compacted from being read. This operation is not required for the COW table.</li></ul>
</div></div>
</div>
<div class="section" id="EN-US_TOPIC_0000001811490465__section14409747152517"><h4 class="sectiontitle">Querying Hudi Foreign Table and Automatically Synchronizing Tasks</h4><p id="EN-US_TOPIC_0000001811490465__p15861112602619">GaussDB(DWS) provides a series of system functions to obtain Hudi foreign table information and create Hudi automatic synchronization tasks. The automatic Hudi synchronization task periodically synchronizes data from Hudi foreign tables to GaussDB(DWS) internal tables.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="EN-US_TOPIC_0000001811490465__table15666428183110" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Hudi system functions</caption><thead align="left"><tr id="EN-US_TOPIC_0000001811490465__row366642833115"><th align="left" class="cellrowborder" valign="top" width="10.94%" id="mcps1.3.2.3.2.5.1.1"><p id="EN-US_TOPIC_0000001811490465__p13667928173116">No.</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="39.06%" id="mcps1.3.2.3.2.5.1.2"><p id="EN-US_TOPIC_0000001811490465__p1667628193118">Function</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="13.389999999999999%" id="mcps1.3.2.3.2.5.1.3"><p id="EN-US_TOPIC_0000001811490465__p166717286312">Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="36.61%" id="mcps1.3.2.3.2.5.1.4"><p id="EN-US_TOPIC_0000001811490465__p966992810318">Functionality</p>
</th>
</tr>
</thead>
<tbody><tr id="EN-US_TOPIC_0000001811490465__row06691328153119"><td class="cellrowborder" valign="top" width="10.94%" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p66691228143113">1</p>
</td>
<td class="cellrowborder" valign="top" width="39.06%" headers="mcps1.3.2.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001811490465__p2093013953214">pg_show_custom_settings()</p>
</td>
<td class="cellrowborder" valign="top" width="13.389999999999999%" headers="mcps1.3.2.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001811490465__p89051214133318">Built-in functions</p>
</td>
<td class="cellrowborder" valign="top" width="36.61%" headers="mcps1.3.2.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001811490465__p89051114153313">Queries details about the parameter settings of an HUDI foreign table.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row666912883116"><td class="cellrowborder" valign="top" width="10.94%" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p136693285317">2</p>
</td>
<td class="cellrowborder" valign="top" width="39.06%" headers="mcps1.3.2.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001811490465__p493003963211">hudi_get_options(regclass)</p>
</td>
<td class="cellrowborder" valign="top" width="13.389999999999999%" headers="mcps1.3.2.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001811490465__p1190511143332">Built-in functions</p>
</td>
<td class="cellrowborder" valign="top" width="36.61%" headers="mcps1.3.2.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001811490465__p19905314143317">Queries the attributes of an HUDI foreign table (hoodie.properties).</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row46698282315"><td class="cellrowborder" valign="top" width="10.94%" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p16670328153114">3</p>
</td>
<td class="cellrowborder" valign="top" width="39.06%" headers="mcps1.3.2.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001811490465__p59301039123215">hudi_get_max_commit(regclass)</p>
</td>
<td class="cellrowborder" valign="top" width="13.389999999999999%" headers="mcps1.3.2.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001811490465__p390515144333">Built-in functions</p>
</td>
<td class="cellrowborder" valign="top" width="36.61%" headers="mcps1.3.2.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001811490465__p6905111433313">Obtains the latest commit timestamp of the current HUDI foreign table.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row16701428163117"><td class="cellrowborder" rowspan="2" valign="top" width="10.94%" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p1367022873114">4</p>
</td>
<td class="cellrowborder" valign="top" width="39.06%" headers="mcps1.3.2.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001811490465__p1093018394323">hudi_sync_task_submit(regclass, regclass)</p>
</td>
<td class="cellrowborder" rowspan="2" valign="top" width="13.389999999999999%" headers="mcps1.3.2.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001811490465__p129051814173319">Built-in functions</p>
<p id="EN-US_TOPIC_0000001811490465__p719817334332"></p>
</td>
<td class="cellrowborder" rowspan="2" valign="top" width="36.61%" headers="mcps1.3.2.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001811490465__p159052144334">Submits the HUDI automatic synchronization task.</p>
<p id="EN-US_TOPIC_0000001811490465__p1195233103319"></p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row26701928133119"><td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p093019395329">hudi_sync_task_submit(regclass, regclass, text, text)</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row166706285319"><td class="cellrowborder" valign="top" width="10.94%" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p4671192811310">5</p>
</td>
<td class="cellrowborder" valign="top" width="39.06%" headers="mcps1.3.2.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001811490465__p09301739133213">hudi_show_sync_state()</p>
</td>
<td class="cellrowborder" valign="top" width="13.389999999999999%" headers="mcps1.3.2.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001811490465__p1390531415334">Built-in functions</p>
</td>
<td class="cellrowborder" valign="top" width="36.61%" headers="mcps1.3.2.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001811490465__p290561443319">Obtains the synchronization status of the HUDI automatic synchronization task.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row1367111288312"><td class="cellrowborder" valign="top" width="10.94%" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p06711628113115">6</p>
</td>
<td class="cellrowborder" valign="top" width="39.06%" headers="mcps1.3.2.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001811490465__p15930139143210">hudi_sync(regclass, regclass)</p>
</td>
<td class="cellrowborder" valign="top" width="13.389999999999999%" headers="mcps1.3.2.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001811490465__p149058148337">Stored procedure</p>
</td>
<td class="cellrowborder" valign="top" width="36.61%" headers="mcps1.3.2.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001811490465__p13906114123311">Specifies the entry for invoking the HUDI automatic synchronization task.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row16671152814317"><td class="cellrowborder" valign="top" width="10.94%" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p167102810316">7</p>
</td>
<td class="cellrowborder" valign="top" width="39.06%" headers="mcps1.3.2.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001811490465__p6930239103213">hudi_sync_custom(regclass, regclass, text)</p>
</td>
<td class="cellrowborder" valign="top" width="13.389999999999999%" headers="mcps1.3.2.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001811490465__p13906171483319">Stored procedure</p>
</td>
<td class="cellrowborder" valign="top" width="36.61%" headers="mcps1.3.2.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001811490465__p11906101423313">Specifies the entry for invoking the HUDI automatic synchronization task. Users can define the mapping between fields in the target table and data source table.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row367242817317"><td class="cellrowborder" rowspan="2" valign="top" width="10.94%" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p36293016218">8</p>
<p id="EN-US_TOPIC_0000001811490465__p1851019383145"></p>
<p id="EN-US_TOPIC_0000001811490465__p48033015218"></p>
</td>
<td class="cellrowborder" valign="top" width="39.06%" headers="mcps1.3.2.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001811490465__p14467634142611">hudi_set_sync_commit(regclass, regclass, text)</p>
</td>
<td class="cellrowborder" rowspan="2" valign="top" width="13.389999999999999%" headers="mcps1.3.2.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001811490465__p2865175111614"></p>
<p id="EN-US_TOPIC_0000001811490465__p6684161312272">Built-in functions</p>
<p id="EN-US_TOPIC_0000001811490465__p923112568269"></p>
<p id="EN-US_TOPIC_0000001811490465__p6884175121615"></p>
<p id="EN-US_TOPIC_0000001811490465__p380113072115"></p>
</td>
<td class="cellrowborder" valign="top" width="36.61%" headers="mcps1.3.2.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001811490465__p12866155111617">Sets the start timestamp of the first synchronization of the HUDI automatic synchronization task to prevent resynchronization.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001811490465__row33995163219"><td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001811490465__p143885309282">hudi_set_sync_commit(text, text)</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001811490465__p18231356162616">Sets the start timestamp of the next synchronization of a HUDI automatic synchronization task. You can use it to sync historical data again or to skip some data.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_04_1069.html">SQL on Hudi</a></div>
</div>
</div>
View Git Blame Copy Permalink