Compare commits

..

1 Commits

Author SHA1 Message Date
8808c652ee Update .gitea/workflows/run-hcdc.yaml 2025-09-26 07:47:50 +00:00
48 changed files with 571 additions and 602 deletions

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -8,14 +8,9 @@
</th>
</tr>
</thead>
<tbody><tr id="dli_08_00005__row3388181562412"><td class="cellrowborder" valign="top" width="26.91%" headers="mcps1.3.1.1.3.1.1 "><p id="dli_08_00005__p1020431922412">2025-04-30</p>
<tbody><tr id="dli_08_00005__row117793121078"><td class="cellrowborder" valign="top" width="26.91%" headers="mcps1.3.1.1.3.1.1 "><p id="dli_08_00005__p8777202392813">2025-03-04</p>
</td>
<td class="cellrowborder" valign="top" width="73.09%" headers="mcps1.3.1.1.3.1.2 "><p id="dli_08_00005__p520414195244">Added descriptions that data multi-versioning will be deprecated and is no longer recommended to <a href="dli_08_0349.html">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</a>.</p>
</td>
</tr>
<tr id="dli_08_00005__row117793121078"><td class="cellrowborder" valign="top" width="26.91%" headers="mcps1.3.1.1.3.1.1 "><p id="dli_08_00005__p8777202392813">2025-03-04</p>
</td>
<td class="cellrowborder" valign="top" width="73.09%" headers="mcps1.3.1.1.3.1.2 "><p id="dli_08_00005__p177719238285">Added <a href="dli_08_15000.html">Flink OpenSource SQL 1.15 Syntax Reference</a>.</p>
<td class="cellrowborder" valign="top" width="73.09%" headers="mcps1.3.1.1.3.1.2 "><p id="dli_08_00005__p177719238285">Added <a href="en-us_topic_0000002172127506.html">Flink OpenSource SQL 1.15 Syntax Reference</a>.</p>
</td>
</tr>
<tr id="dli_08_00005__row02881225102114"><td class="cellrowborder" valign="top" width="26.91%" headers="mcps1.3.1.1.3.1.1 "><p id="dli_08_00005__p18857192716215">2025-02-08</p>

View File

@ -1,8 +1,7 @@
<a name="dli_08_0057"></a><a name="dli_08_0057"></a>
<h1 class="topictitle1">Overview</h1>
<div id="body8662426"><p id="dli_08_0057__p1625813109247">Data type is a basic attribute of data and used to distinguish different types of data. Different data types occupy different storage space and support different operations.</p>
<p id="dli_08_0057__p1910192232411">Data is stored in data tables in the database. Each column of a table defines the data type. During storage, data must be stored according to data types.</p>
<div id="body8662426"><p id="dli_08_0057__en-us_topic_0093947008_a60c0b18611f4456188c9308bc450734b">Data type is a basic attribute of data. It is used to distinguish different types of data. Different data types occupy different storage space and support different operations. Data is stored in data tables in the database. A data type is specified for each column of a data table. Therefore, data to be stored in a data table must comply with the attribute of the specific data type. Otherwise, errors may occur.</p>
<p id="dli_08_0057__en-us_topic_0093947008_a63de052ecbb24a88a5db1c9f680643b0">DLI only supports primitive data types.</p>
</div>
<div>

View File

@ -3,7 +3,7 @@
<h1 class="topictitle1">Creating an OBS Table Using the DataSource Syntax</h1>
<div id="body8662426"><div class="section" id="dli_08_0076__en-us_topic_0114776170_en-us_topic_0093946797_sb2a2dad92c524677baa470c6a05dc948"><h4 class="sectiontitle">Function</h4><p id="dli_08_0076__p28131188218">Create an OBS table using the DataSource syntax.</p>
<p id="dli_08_0076__en-us_topic_0114776170_en-us_topic_0093946797_p374295012356">The main differences between the DataSource and the Hive syntax lie in the supported data formats and the number of supported partitions. For details, see syntax and precautions.</p>
<div class="note" id="dli_08_0076__note2901719181211"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0076__p1990121912128">You are advised to use an OBS parallel file system for storage. A parallel file system is a high-performance file system that provides latency in milliseconds, TB/s-level bandwidth, and millions of IOPS. It applies to interactive big data analysis scenarios.</p>
<div class="note" id="dli_08_0076__note2901719181211"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0076__p1990121912128">You are advised to use the OBS parallel file system for storage. A parallel file system is a high-performance file system that provides latency in milliseconds, TB/s-level bandwidth, and millions of IOPS. It applies to interactive big data analysis scenarios.</p>
</div></div>
</div>
<div class="section" id="dli_08_0076__en-us_topic_0114776170_en-us_topic_0093946797_s2af9615060a543218bb4f231acfcd929"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0076__en-us_topic_0114776170_en-us_topic_0093946797_u0132702d68ac4186be555e682b694899"><li id="dli_08_0076__dli_08_0076_li15318170215">The size of a table is not calculated when the table is created.</li><li id="dli_08_0076__dli_08_0076_li1453121719215">When data is added, the table size will be changed to 0.</li><li id="dli_08_0076__dli_08_0076_li25313179210">You can check the table size on OBS.</li><li id="dli_08_0076__li27231725165616">Table properties cannot be specified using CTAS table creation statements.</li><li id="dli_08_0076__li119261906338"><strong id="dli_08_0076__b8773182162519">An OBS directory containing subdirectories:</strong><p id="dli_08_0076__p19145415173313">If you specify an OBS directory that contains subdirectories when creating a table, all file types and content within those subdirectories will also be included as table content.</p>
@ -448,10 +448,6 @@
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0076__section1216992824118"><h4 class="sectiontitle">FAQ</h4><ul id="dli_08_0076__ul14911104711418"><li id="dli_08_0076__li1791164720414"><strong id="dli_08_0076__b16496113704610">What should I do if the error message "xxx dli datasource v2 tables is only supported in spark3.3 or later version." appears when I create a DataSource table using the default queue?</strong><p id="dli_08_0076__p19155311418">Ensure that you use Spark 3.3.1 or a later version when creating such a table. If the error message appears, use the Hive syntax to create the table. For details, see <a href="dli_08_0077.html">Creating an OBS Table Using the Hive Syntax</a>.</p>
</li><li id="dli_08_0076__li01781618194315"><strong id="dli_08_0076__b102412910474">What should I do if the error message "xxx don`t support dli v1 table." appears when I use Spark 3.3.1 to run a Jar job?</strong><p id="dli_08_0076__p111863544616">This error message indicates that table operations cannot be performed when Spark 3.3.1 is used to execute the Jar job. Use the Hive syntax to recreate the tables' data structure. For example, you can use <strong id="dli_08_0076__b41612316581">[STORED AS file_format] CTAS</strong> to recreate the table and then run the job. For details, see <a href="dli_08_0077.html">Creating an OBS Table Using the Hive Syntax</a>.</p>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">

View File

@ -43,7 +43,7 @@
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0077__en-us_topic_0114776171_en-us_topic_0093946792_s3805b86231dd4c41b26864162d6c4ed5"><h4 class="sectiontitle">Keywords</h4><ul id="dli_08_0077__en-us_topic_0114776171_ul74261946687"><li id="dli_08_0077__en-us_topic_0114776171_en-us_topic_0093946792_l4409cf7808da45c6abd51a1e32c0b909">EXTERNAL: Creates an OBS table.</li><li id="dli_08_0077__en-us_topic_0114776171_li12814539191914">IF NOT EXISTS: Prevents system errors when the created table exists.</li><li id="dli_08_0077__en-us_topic_0114776171_li11143171444414">COMMENT: Field or table description.</li><li id="dli_08_0077__en-us_topic_0114776171_li572161314209">PARTITIONED BY: Partition field.</li><li id="dli_08_0077__li1761110581424">ROW FORMAT: Row data format.</li><li id="dli_08_0077__en-us_topic_0114776171_en-us_topic_0093946792_l2b4d833ae66e470dae553c300f0783b8">STORED AS: Specifies the format of the file to be stored. Currently, only the TEXTFILE, AVRO, ORC, SEQUENCEFILE, RCFILE, and PARQUET format are supported.</li><li id="dli_08_0077__en-us_topic_0114776171_en-us_topic_0093946792_le38d766b4f094c178071c05b9717391f">LOCATION: Specifies the path of OBS. This keyword is mandatory when you create OBS tables.</li><li id="dli_08_0077__li6331130191815">TBLPROPERTIES: Allows you to add the <strong id="dli_08_0077__b1550212295461">key/value</strong> properties to a table.<ul id="dli_08_0077__ul1611116494185"><li id="dli_08_0077__li7473164411185">(The multi-versioning function has been deprecated and is not recommended.) Enables data multi-versioning for table data backup and restoration. After the multiversion function is enabled, the system automatically backs up table data when you delete or modify the data using <strong id="dli_08_0077__b17731842484">insert overwrite</strong> or <strong id="dli_08_0077__b18773204194816">truncate</strong>, and retains the data for a certain period. You can quickly restore data within the retention period. For details about the SQL syntax for the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Data Multi-Versioning (Deprecated, Not Recommended)</a> and <a href="dli_08_0349.html">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</a>.<p id="dli_08_0077__p1882095415365">When creating an OBS table, you can use <strong id="dli_08_0077__b12439113516404">TBLPROPERTIES ("dli.multi.version.enable"="true")</strong> to enable multiversion. For details, see the following example.</p>
<div class="section" id="dli_08_0077__en-us_topic_0114776171_en-us_topic_0093946792_s3805b86231dd4c41b26864162d6c4ed5"><h4 class="sectiontitle">Keywords</h4><ul id="dli_08_0077__en-us_topic_0114776171_ul74261946687"><li id="dli_08_0077__en-us_topic_0114776171_en-us_topic_0093946792_l4409cf7808da45c6abd51a1e32c0b909">EXTERNAL: Creates an OBS table.</li><li id="dli_08_0077__en-us_topic_0114776171_li12814539191914">IF NOT EXISTS: Prevents system errors when the created table exists.</li><li id="dli_08_0077__en-us_topic_0114776171_li11143171444414">COMMENT: Field or table description.</li><li id="dli_08_0077__en-us_topic_0114776171_li572161314209">PARTITIONED BY: Partition field.</li><li id="dli_08_0077__li1761110581424">ROW FORMAT: Row data format.</li><li id="dli_08_0077__en-us_topic_0114776171_en-us_topic_0093946792_l2b4d833ae66e470dae553c300f0783b8">STORED AS: Specifies the format of the file to be stored. Currently, only the TEXTFILE, AVRO, ORC, SEQUENCEFILE, RCFILE, and PARQUET format are supported.</li><li id="dli_08_0077__en-us_topic_0114776171_en-us_topic_0093946792_le38d766b4f094c178071c05b9717391f">LOCATION: Specifies the path of OBS. This keyword is mandatory when you create OBS tables.</li><li id="dli_08_0077__li6331130191815">TBLPROPERTIES: Allows you to add the <strong id="dli_08_0077__b1550212295461">key/value</strong> properties to a table.<ul id="dli_08_0077__ul1611116494185"><li id="dli_08_0077__li7473164411185">You can use this statement to enable the multiversion function to back up and restore table data. After the multiversion function is enabled, the system automatically backs up table data when you delete or modify the data using <strong id="dli_08_0077__b17731842484">insert overwrite</strong> or <strong id="dli_08_0077__b18773204194816">truncate</strong>, and retains the data for a certain period. You can quickly restore data within the retention period. For details about the SQL syntax for the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Multiversion Backup</a> and <a href="dli_08_0349.html">Backing Up and Restoring Data of Multiple Versions</a>.<p id="dli_08_0077__p1882095415365">When creating an OBS table, you can use <strong id="dli_08_0077__b12439113516404">TBLPROPERTIES ("dli.multi.version.enable"="true")</strong> to enable multiversion. For details, see the following example.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0077__table2861727205115" frame="border" border="1" rules="all"><caption><b>Table 1 </b>TBLPROPERTIES parameters</caption><thead align="left"><tr id="dli_08_0077__row2862827195111"><th align="left" class="cellrowborder" valign="top" width="32.35%" id="mcps1.3.4.2.8.2.1.6.2.3.1.1"><p id="dli_08_0077__p1686272745115">Key</p>
</th>

View File

@ -3,11 +3,7 @@
<h1 class="topictitle1">Inserting Data</h1>
<div id="body8662426"><div class="section" id="dli_08_0095__en-us_topic_0114776189_en-us_topic_0093946901_s9964bb020c52461992d2c7711fd7feed"><h4 class="sectiontitle">Function</h4><p id="dli_08_0095__en-us_topic_0114776189_en-us_topic_0093946901_a453a1e6d2a6e4b05a42b498a72d224bc">This statement is used to insert the SELECT query result or a certain data record into a table.</p>
</div>
<div class="section" id="dli_08_0095__section1480717121549"><h4 class="sectiontitle">Notes and Constraints</h4><ul id="dli_08_0095__ul1282125143915"><li id="dli_08_0095__li194403272292">The <strong id="dli_08_0095__b11593143874813">insert overwrite</strong> syntax does not apply to self-read and self-write scenarios within the same table (including both partitioned and non-partitioned tables). Directly executing <strong id="dli_08_0095__b16963121124914">insert overwrite</strong> on the original table may lead to risks of data loss or inconsistency.<p id="dli_08_0095__p153511828172917">To implement data operations in self-read and self-write scenarios, you are advised to use a temporary table to handle the data. See <a href="#dli_08_0095__fig8181339957">Figure 1</a>.</p>
<p id="dli_08_0095__p127271127173912">Self-read and self-write means that the destination table and the data source table are the same table. For example, suppose you want to extract information of students with <strong id="dli_08_0095__b17972543541">class_no = 1</strong> from the <strong id="dli_08_0095__b1166212579549">student</strong> table and overwrite the original table, the following statements represent typical operations in self-read and self-write scenarios:</p>
<pre class="screen" id="dli_08_0095__screen8847151312282">INSERT OVERWRITE TABLE student
SELECT name FROM student WHERE class_no = 1;</pre>
<div class="fignone" id="dli_08_0095__fig8181339957"><a name="dli_08_0095__fig8181339957"></a><a name="fig8181339957"></a><span class="figcap"><b>Figure 1 </b>Alternative solution for self-read and self-write scenarios by running insert overwrite</span><br><span><img id="dli_08_0095__image16599286715" src="en-us_image_0000002272860560.png"></span></div>
<div class="section" id="dli_08_0095__section1480717121549"><h4 class="sectiontitle">Notes and Constraints</h4><ul id="dli_08_0095__ul1282125143915"><li id="dli_08_0095__li8821525103916">The <strong id="dli_08_0095__b19412132395515">INSERT OVERWRITE</strong> syntax is not suitable for "read-write" scenarios, where data is continuously processed and updated. Using this syntax in such scenarios may result in data loss.<p id="dli_08_0095__p127271127173912">"Read-write" refers to the ability to read data while generating new data or modifying existing data during data processing.</p>
</li><li id="dli_08_0095__li181767227540">When using Hive and Datasource tables (excluding Hudi), executing data modification commands (such as <strong id="dli_08_0095__b236110285217">insert into</strong> and <strong id="dli_08_0095__b1467713331723">load data</strong>) may result in data duplication or inconsistency if the data source does not support transactions and there is a system failure or queue restart.<p id="dli_08_0095__p15597549191616">To avoid this situation, you are advised to prioritize data sources that support transactions, such as Hudi data sources. This type of data source has Atomicity, Consistency, Isolation, Durability (ACID) capabilities, which helps ensure data consistency and accuracy.</p>
<p id="dli_08_0095__p1542252311541">To learn more, refer to <a href="#dli_08_0095__section1516329541">How Do I Handle Duplicate Records After Executing the INSERT INTO Statement?</a></p>
</li></ul>
@ -84,14 +80,14 @@ SELECT name FROM student WHERE class_no = 1;</pre>
</div>
<div class="section" id="dli_08_0095__en-us_topic_0114776189_en-us_topic_0093946901_sc650d4ee825647c2a760ac5fcc4ab2c8"><h4 class="sectiontitle">Example</h4><div class="note" id="dli_08_0095__note1149934611312"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0095__p161225010319">Before importing data, you must create a table. For details, see <a href="dli_08_0223.html">Creating an OBS Table</a> or <a href="dli_08_0224.html">Creating a DLI Table</a>.</p>
</div></div>
<ul id="dli_08_0095__en-us_topic_0114776189_en-us_topic_0093946901_ul18471028173914"><li id="dli_08_0095__li193703191708">Example 1: Insert the SELECT query result into a table.<ul id="dli_08_0095__ul4308115011213"><li id="dli_08_0095__li1494424416214">Use the DataSource syntax to create a parquet partitioned table.<pre class="screen" id="dli_08_0095__screen20881115535">CREATE TABLE data_source_tab1 (col1 INT, p1 INT, p2 INT)
<ul id="dli_08_0095__en-us_topic_0114776189_en-us_topic_0093946901_ul18471028173914"><li id="dli_08_0095__li193703191708">Insert the SELECT query result into a table.<ul id="dli_08_0095__ul4308115011213"><li id="dli_08_0095__li1494424416214">Use the DataSource syntax to create a parquet partitioned table.<pre class="screen" id="dli_08_0095__screen20881115535">CREATE TABLE data_source_tab1 (col1 INT, p1 INT, p2 INT)
USING PARQUET PARTITIONED BY (p1, p2);</pre>
</li><li id="dli_08_0095__li136969131212">Insert the query result to the partition (p1 = 3, p2 = 4).<pre class="screen" id="dli_08_0095__screen09951301751">INSERT INTO data_source_tab1 PARTITION (p1 = 3, p2 = 4)
SELECT id FROM RANGE(1, 3);</pre>
</li><li id="dli_08_0095__li28072014151212">Insert the new query result to the partition (p1 = 3, p2 = 4).<pre class="screen" id="dli_08_0095__screen623634219616">INSERT OVERWRITE TABLE data_source_tab1 PARTITION (p1 = 3, p2 = 4)
SELECT id FROM RANGE(3, 5);</pre>
</li></ul>
</li><li id="dli_08_0095__li1928215175112">Example 2: Insert a piece of data into a table.<ul id="dli_08_0095__ul9733124541111"><li id="dli_08_0095__li10508854201111">Create a Parquet partitioned table with Hive format<pre class="screen" id="dli_08_0095__screen11281528191410">CREATE TABLE hive_serde_tab1 (col1 INT, p1 INT, p2 INT)
</li><li id="dli_08_0095__li1928215175112">Insert a data record into a table.<ul id="dli_08_0095__ul9733124541111"><li id="dli_08_0095__li10508854201111">Create a Parquet partitioned table with Hive format<pre class="screen" id="dli_08_0095__screen11281528191410">CREATE TABLE hive_serde_tab1 (col1 INT, p1 INT, p2 INT)
USING HIVE OPTIONS(fileFormat 'PARQUET') PARTITIONED BY (p1, p2);</pre>
</li><li id="dli_08_0095__li19793643111416">Insert two data records into the partition (p1 = 3, p2 = 4).<pre class="screen" id="dli_08_0095__screen13173181071519">INSERT INTO hive_serde_tab1 PARTITION (p1 = 3, p2 = 4)
VALUES (1), (2);</pre>

View File

@ -254,10 +254,6 @@
</div>
</li></ul>
</div>
<div class="section" id="dli_08_0098__section1216992824118"><h4 class="sectiontitle">FAQ</h4><ul id="dli_08_0098__ul14911104711418"><li id="dli_08_0098__li1791164720414"><strong id="dli_08_0098__b97691718107">What should I do if the error message "xxx dli datasource v2 tables is only supported in spark3.3 or later version." appears when I create a DataSource table using the default queue?</strong><p id="dli_08_0098__p19155311418">Ensure that you use Spark 3.3.1 or a later version when creating such a table. If the error message appears, use the Hive syntax to create the table. For details, see <a href="dli_08_0204.html">Creating a DLI Table Using the Hive Syntax</a>.</p>
</li><li id="dli_08_0098__li01781618194315"><strong id="dli_08_0098__b13969132412012">What should I do if the error message "xxx don`t support dli v1 table." appears when I use Spark 3.3.1 to run a Jar job?</strong><p id="dli_08_0098__p73279166509">This error message indicates that table operations cannot be performed when Spark 3.3.1 is used to execute the Jar job. Use the Hive syntax to recreate the tables' data structure. For example, you can use <strong id="dli_08_0098__b16148291106">[STORED AS file_format] CTAS</strong> to recreate the table and then run the job. For how to create a table, see <a href="dli_08_0204.html">Creating a DLI Table Using the Hive Syntax</a>.</p>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">

View File

@ -34,7 +34,7 @@
</td>
<td class="cellrowborder" valign="top" width="84.2%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0197__en-us_topic_0142907233_p216518281377">Create a datasource connection before you can obtain this RDS connection URL.</p>
<p id="dli_08_0197__en-us_topic_0142907233_p17658829185311">After an enhanced datasource connection is created, use the internal network domain name or internal network address and database port number provided by RDS to connect to DLI. If MySQL is used, the format is <strong id="dli_08_0197__b197693862414"><em id="dli_08_0197__i1875143812410">protocol header</em>://<em id="dli_08_0197__i4757382246">internal IP address</em>:<em id="dli_08_0197__i2768386240">internal network port number</em></strong>. If PostgreSQL is used, the format is <strong id="dli_08_0197__b178838192411"><em id="dli_08_0197__i776193813242">protocol header</em>://<em id="dli_08_0197__i67733882412">internal IP address</em>:<em id="dli_08_0197__i577133872411">internal network port number</em>/<em id="dli_08_0197__i1478123852412">database name</em></strong>.</p>
<p id="dli_08_0197__en-us_topic_0142907233_p14832153644513">For example: <strong id="dli_08_0197__b932218404249">jdbc:mysql://192.168.0.193:3306</strong> or <strong id="dli_08_0197__b8322184082413">jdbc:postgresql://192.168.0.193:5432/postgres</strong>.</p>
<p id="dli_08_0197__en-us_topic_0142907233_p14832153644513">For example: <strong id="dli_08_0197__b932218404249">jdbc:mysql://192.168.0.193:3306</strong> or <strong id="dli_08_0197__b8322184082413">jdbc:postgresql://192.168.0.193:3306/postgres</strong>.</p>
</td>
</tr>
<tr id="dli_08_0197__en-us_topic_0142907233_en-us_topic_0114776213_en-us_topic_0103157088_row166137253465"><td class="cellrowborder" valign="top" width="15.8%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0197__en-us_topic_0142907233_en-us_topic_0114776213_en-us_topic_0103157088_p584237211576">driver</p>
@ -128,8 +128,7 @@
</table>
</div>
</div>
<div class="section" id="dli_08_0197__en-us_topic_0142907233_en-us_topic_0114776213_en-us_topic_0103157088_section1856617202225"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0197__p19948131624612">When creating up an RDS association table for the first time, there is no need to define the table's schema. DLI will automatically retrieve the schema from the RDS parameter <strong id="dli_08_0197__b11207131645417">dbtable</strong> to create the association table.</p>
<p id="dli_08_0197__p159481816144618">If you make changes to the fields in the RDS table, the associated table will not update automatically. In such cases, you need to recreate the association table to ensure its schema matches the modified RDS table.</p>
<div class="section" id="dli_08_0197__en-us_topic_0142907233_en-us_topic_0114776213_en-us_topic_0103157088_section1856617202225"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0197__en-us_topic_0142907233_p912611523415">When creating a table associated with RDS, you do not need to specify the <strong id="dli_08_0197__b29521246102519">Schema</strong> of the associated table. DLI automatically obtains the schema of the table in the <strong id="dli_08_0197__b5362548122511">dbtable</strong> parameter of RDS.</p>
</div>
<div class="section" id="dli_08_0197__en-us_topic_0142907233_en-us_topic_0114776213_en-us_topic_0103157088_section9892337112213"><h4 class="sectiontitle">Example</h4><p id="dli_08_0197__en-us_topic_0142907233_p14325175413169">Accessing MySQL</p>
<div class="codecoloring" codetype="Sql" id="dli_08_0197__en-us_topic_0142907233_screen156701345141612"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
@ -157,7 +156,7 @@
<span class="normal">6</span>
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="n">dli_to_rds</span>
<span class="w"> </span><span class="k">USING</span><span class="w"> </span><span class="n">JDBC</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'url'</span><span class="o">=</span><span class="s1">'jdbc:postgresql://to-rds-1174405119-oLRHAGE7.datasource.com:5432/postgreDB'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'url'</span><span class="o">=</span><span class="s1">'jdbc:postgresql://to-rds-1174405119-oLRHAGE7.datasource.com:3306/postgreDB'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'driver'</span><span class="o">=</span><span class="s1">'org.postgresql.Driver'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'dbtable'</span><span class="o">=</span><span class="s1">'pg_schema.test1'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'passwdauth'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xxx'</span><span class="p">,</span>

View File

@ -1,6 +1,6 @@
<a name="dli_08_0219"></a><a name="dli_08_0219"></a>
<h1 class="topictitle1">SQL Syntax Overview for Batch Jobs</h1>
<h1 class="topictitle1">SQL Syntax Overview of Batch Jobs</h1>
<div id="body1578361990117"><p id="dli_08_0219__p7811840729">This section describes the Spark SQL syntax list provided by DLI. For details about the parameters and examples, see the syntax description.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0219__en-us_topic_0178353918_table3712193011019" frame="border" border="1" rules="all"><caption><b>Table 1 </b>SQL syntax of batch jobs</caption><thead align="left"><tr id="dli_08_0219__en-us_topic_0178353918_row57131230101015"><th align="left" class="cellrowborder" valign="top" width="39.989999999999995%" id="mcps1.3.2.2.3.1.1"><p id="dli_08_0219__p12912191921618">Classification</p>
@ -128,15 +128,15 @@
<tr id="dli_08_0219__row894531915318"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p6930191975313"><a href="dli_08_0124.html">Querying an OpenTSDB Table</a></p>
</td>
</tr>
<tr id="dli_08_0219__row16889357205419"><td class="cellrowborder" rowspan="3" valign="top" width="39.989999999999995%" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p6860175714542">Syntax for Datasource Connection to a GaussDB(DWS) Table</p>
<tr id="dli_08_0219__row16889357205419"><td class="cellrowborder" rowspan="3" valign="top" width="39.989999999999995%" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p6860175714542">Syntax for Datasource Connection to a DWS Table</p>
</td>
<td class="cellrowborder" valign="top" width="60.01%" headers="mcps1.3.2.2.3.1.2 "><p id="dli_08_0219__p88601057185417"><a href="dli_08_0193.html">Creating a Table and Associating It with GaussDB(DWS)</a></p>
<td class="cellrowborder" valign="top" width="60.01%" headers="mcps1.3.2.2.3.1.2 "><p id="dli_08_0219__p88601057185417"><a href="dli_08_0193.html">Creating a Table and Associating It with DWS</a></p>
</td>
</tr>
<tr id="dli_08_0219__row13889105735419"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p78611757155416"><a href="dli_08_0194.html">Inserting Data to a GaussDB(DWS) Table</a></p>
<tr id="dli_08_0219__row13889105735419"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p78611757155416"><a href="dli_08_0194.html">Inserting Data to a DWS Table</a></p>
</td>
</tr>
<tr id="dli_08_0219__row14888125775419"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p5861155755417"><a href="dli_08_0195.html">Querying a GaussDB(DWS) Table</a></p>
<tr id="dli_08_0219__row14888125775419"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p5861155755417"><a href="dli_08_0195.html">Querying a DWS Table</a></p>
</td>
</tr>
<tr id="dli_08_0219__row13909144265619"><td class="cellrowborder" rowspan="3" valign="top" width="39.989999999999995%" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p188621042195619">Syntax for Datasource Connection to an RDS Table</p>
@ -246,7 +246,7 @@
<tr id="dli_08_0219__row19781623131017"><td class="cellrowborder" valign="top" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p1254623191016"><a href="dli_08_0285.html">Displaying All Functions</a></p>
</td>
</tr>
<tr id="dli_08_0219__row134000231125"><td class="cellrowborder" valign="top" width="39.989999999999995%" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p1040016235124">Data Multi-Versioning-related Syntax (Deprecated, Not Recommended)</p>
<tr id="dli_08_0219__row134000231125"><td class="cellrowborder" valign="top" width="39.989999999999995%" headers="mcps1.3.2.2.3.1.1 "><p id="dli_08_0219__p1040016235124">Multiversion-related Syntax</p>
</td>
<td class="cellrowborder" valign="top" width="60.01%" headers="mcps1.3.2.2.3.1.2 "><p id="dli_08_0219__p1340012311123"><a href="dli_08_0077.html">Enabling Multiversion Backup When Creating an OBS Table</a></p>
<p id="dli_08_0219__p268881001318"><a href="dli_08_0354.html">Enabling or Disabling Multiversion Backup When Modifying a Table</a></p>

View File

@ -7,7 +7,7 @@
<ul class="ullinks">
<li class="ulchildlink"><strong><a href="dli_08_0266.html">Common Configuration Items of Batch SQL Jobs</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_08_0219.html">SQL Syntax Overview for Batch Jobs</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_08_0219.html">SQL Syntax Overview of Batch Jobs</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_08_0477.html">Spark Open Source Commands</a></strong><br>
</li>
@ -35,7 +35,7 @@
</li>
<li class="ulchildlink"><strong><a href="dli_08_0205.html">Exporting Search Results</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_08_0349.html">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_08_0349.html">Backing Up and Restoring Data of Multiple Versions</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_08_0465.html">Table Lifecycle Management</a></strong><br>
</li>

View File

@ -1,7 +1,7 @@
<a name="dli_08_0261"></a><a name="dli_08_0261"></a>
<h1 class="topictitle1">Creating an RDS Table</h1>
<div id="body1574393121912"><p id="dli_08_0261__p7798335163614">Create an RDS or GaussDB(DWS) table to connect to the source stream.</p>
<div id="body1574393121912"><p id="dli_08_0261__p7798335163614">Create an RDS/DWS table to connect to the source stream.</p>
<p id="dli_08_0261__p174595520164">For details about the JOIN syntax, see <a href="dli_08_0106.html">JOIN</a>.</p>
<div class="section" id="dli_08_0261__section149656154515"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0261__en-us_topic_0111499975_ul810551215398"><li id="dli_08_0261__en-us_topic_0111499975_li6425436163814">Ensure that you have created a PostgreSQL or MySQL RDS instance in RDS.<p id="dli_08_0261__en-us_topic_0111499975_p104771013203812"><a name="dli_08_0261__en-us_topic_0111499975_li6425436163814"></a><a name="en-us_topic_0111499975_li6425436163814"></a>For details about how to create an RDS instance, see <span class="filepath" id="dli_08_0261__filepath102767631292142"><b>Creating an Instance</b></span> in the <em id="dli_08_0261__i25507952392142">Relational Database Service User Guide</em>.</p>
</li><li id="dli_08_0261__en-us_topic_0111499975_li14111191212396">In this scenario, jobs must run on the dedicated queue of DLI. Therefore, DLI must interconnect with the enhanced datasource connection that has been connected with RDS instance. You can also set the security group rules as required.<p id="dli_08_0261__p94091618103916"><a name="dli_08_0261__en-us_topic_0111499975_li14111191212396"></a><a name="en-us_topic_0111499975_li14111191212396"></a>For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_08_0261__dli_08_0249_en-us_topic_0132966325_filepath13726182511815"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_08_0261__dli_08_0249_cite534516e564224b13bad1de4471da3139181527">Data Lake Insight User Guide</cite></i>.</p>
@ -73,7 +73,7 @@
</td>
<td class="cellrowborder" valign="top" width="71.35353535353536%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0261__p14859300387">Database connection address, for example, <strong id="dli_08_0261__b26002101492142">{database_type}://ip:port/database</strong>.</p>
<p id="dli_08_0261__p193531452515">Currently, two types of database connections are supported: MySQL and PostgreSQL.</p>
<ul id="dli_08_0261__ul18228175913311"><li id="dli_08_0261__li122812591335">MySQL: 'mysql://ip:port/database'</li><li id="dli_08_0261__li112281159133316">PostgreSQL: 'postgresql://ip:port/database'<div class="note" id="dli_08_0261__note19207195162811"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_0261__p62071951142817">To create a GaussDB(DWS) dimension table, set the database connection address to a GaussDB(DWS) database address. If the GaussDB(DWS) database version is later than 8.1.0, the open-source PostgreSQL driver cannot be used for connection. You need to use the GaussDB driver for connection.</p>
<ul id="dli_08_0261__ul18228175913311"><li id="dli_08_0261__li122812591335">MySQL: 'mysql://ip:port/database'</li><li id="dli_08_0261__li112281159133316">PostgreSQL: 'postgresql://ip:port/database'<div class="note" id="dli_08_0261__note19207195162811"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_0261__p62071951142817">To create a DWS dimension table, set the database connection address to a DWS database address. If the DWS database version is later than 8.1.0, the open-source PostgreSQL driver cannot be used for connection. You need to use the GaussDB driver for connection.</p>
</div></div>
</li></ul>
</td>
@ -204,7 +204,7 @@
</pre></div></td></tr></table></div>
</div>
</div>
<div class="note" id="dli_08_0261__note13797314164316"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0261__p379716147437">To create a GaussDB(DWS) dimension table, set the database connection address to a GaussDB(DWS) database address. If the GaussDB(DWS) database version is later than 8.1.0, the open-source PostgreSQL driver cannot be used for connection. You need to use the GaussDB driver for connection.</p>
<div class="note" id="dli_08_0261__note13797314164316"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0261__p379716147437">To create a DWS dimension table, set the database connection address to a DWS database address. If the DWS database version is later than 8.1.0, the open-source PostgreSQL driver cannot be used for connection. You need to use the GaussDB driver for connection.</p>
</div></div>
</div>
</div>

View File

@ -9,7 +9,7 @@
</li>
<li class="ulchildlink"><strong><a href="dli_08_0470.html">Modifying Column Comments</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_08_0354.html">Enabling or Disabling Data Multi-Versioning (Deprecated, Not Recommended)</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_08_0354.html">Enabling or Disabling Multiversion Backup</a></strong><br>
</li>
</ul>

View File

@ -78,18 +78,14 @@
<td class="cellrowborder" valign="top" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0266__p16441413464">false</p>
</td>
</tr>
<tr id="dli_08_0266__row101021133154210"><td class="cellrowborder" valign="top" width="28.349999999999998%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0266__p16103123320426">spark.sql.dli.job.shareLevel</p>
<tr id="dli_08_0266__row11135125418304"><td class="cellrowborder" valign="top" width="28.349999999999998%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0266__p2013525463013">spark.sql.optimizer.dynamicPartitionPruning.enabled</p>
</td>
<td class="cellrowborder" valign="top" width="9.56%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0266__p2103123314427">Queue</p>
<td class="cellrowborder" valign="top" width="9.56%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0266__p1135854183018">true</p>
</td>
<td class="cellrowborder" valign="top" width="62.09%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0266__p1449141415431">This configuration item is used to set the isolation level of SQL statements. Different isolation levels (job, user, project, queue) determine whether SQL jobs are executed by independent Spark Drivers and Executors or share existing ones.</p>
<ul id="dli_08_0266__ul178673438527"><li id="dli_08_0266__li164481049155212"><strong id="dli_08_0266__b1522419919555">job</strong>:<ul id="dli_08_0266__ul166155519524"><li id="dli_08_0266__li198671343175215">Each SQL job will independently start a Spark Driver and a set of Executors for execution.</li><li id="dli_08_0266__li1486724335210">This is suitable for jobs that require complete isolation, ensuring that each job's execution environment is entirely independent.</li></ul>
</li><li id="dli_08_0266__li18717333105311"><strong id="dli_08_0266__b19700611155510">user:</strong><ul id="dli_08_0266__ul8635137549"><li id="dli_08_0266__li172701757125318">If a Spark Driver started by this user already exists and can continue submitting tasks, the new SQL job will be submitted to this existing Driver for execution.</li><li id="dli_08_0266__li11270125713538">If there is no existing Driver or the current Driver cannot continue submitting tasks, a new Spark Driver will be started for this user.</li><li id="dli_08_0266__li2270125755315">This is suitable for scenarios where multiple jobs from the same user need to share resources.</li></ul>
</li><li id="dli_08_0266__li16832196546"><strong id="dli_08_0266__b147091217185516">project:</strong><ul id="dli_08_0266__ul17995173016543"><li id="dli_08_0266__li1212311230546">If a Spark Driver started by this project already exists and can continue submitting tasks, the new SQL job will be submitted to this existing Driver for execution.</li><li id="dli_08_0266__li1212319231549">If there is no existing Driver or the current Driver cannot continue submitting tasks, a new Spark Driver will be started for this project.</li><li id="dli_08_0266__li912332311545">This is suitable for scenarios where multiple jobs within the same project need to share resources.</li></ul>
</li><li id="dli_08_0266__li7712164012544"><strong id="dli_08_0266__b11265172216558">queue:</strong><ul id="dli_08_0266__ul213054919541"><li id="dli_08_0266__li202986440544">If a Spark Driver started by this queue already exists and can continue submitting tasks, the new SQL job will be submitted to this existing Driver for execution.</li><li id="dli_08_0266__li729824415540">If there is no existing Driver or the current Driver cannot continue submitting tasks, a new Spark Driver will be started for this queue.</li><li id="dli_08_0266__li192981044185413">This is suitable for scenarios where resources are managed by queues, allowing for more granular control over resource allocation.</li></ul>
<td class="cellrowborder" valign="top" width="62.09%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0266__p0120112114312">This parameter is used to control whether to enable dynamic partition pruning. Dynamic partition pruning can help reduce the amount of data that needs to be scanned and improve query performance when executing SQL queries.</p>
<ul id="dli_08_0266__ul10431123219399"><li id="dli_08_0266__li34312325392">When set to <strong id="dli_08_0266__b1293118303259">true</strong>, dynamic partition pruning is enabled. SQL automatically detects and deletes partitions that do not meet the WHERE clause conditions during query. This is useful for tables that have a large number of partitions.</li><li id="dli_08_0266__li18431163263919">If SQL queries contain a large number of nested left join operations and the table has a large number of dynamic partitions, a large number of memory resources may be consumed during data parsing. As a result, the memory of the driver node is insufficient and there are frequent Full GCs.<p id="dli_08_0266__p9628429134018"><a name="dli_08_0266__li18431163263919"></a><a name="li18431163263919"></a>To avoid such issues, you can disable dynamic partition pruning by setting this parameter to <strong id="dli_08_0266__b155611420352">false</strong>.</p>
<p id="dli_08_0266__p317112359398">However, disabling this optimization may reduce query performance. Once disabled, Spark does not automatically prun the partitions that do not meet the requirements.</p>
</li></ul>
<div class="note" id="dli_08_0266__note070313588548"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_0266__p4703175812547">The maximum number of Spark Drivers that can be started (maximum Spark Driver instances) and the maximum number of concurrent SQL queries that can be executed by each Spark Driver (maximum concurrency per Spark Driver instance) can be configured in the queue properties.</p>
</div></div>
</td>
</tr>
</tbody>

View File

@ -11,7 +11,7 @@ FROM (
FROM table_name)
WHERE rownum = 1</pre>
</div>
<div class="section" id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_section1627916611011"><h4 class="sectiontitle">Description</h4><ul id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_ul12220145108"><li id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_li82201045703">ROW_NUMBER(): Assigns a unique, sequential number to each row, starting with one.</li><li id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_li152201045504">PARTITION BY col1[, col2...]: Specifies the partition columns, for example, the deduplicate key.</li><li id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_li1422020451403">ORDER BY time_attr [asc|desc]: Specifies the ordering column, which must be a time attribute. Currently Flink supports proctime only. Ascending (ASC) sorting keeps only the first row, while descending (DESC) sorting keeps only the last row.</li><li id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_li1922012451800">WHERE rownum = 1: The rownum = 1 is required for Flink to recognize this query is deduplication.</li></ul>
<div class="section" id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_section1627916611011"><h4 class="sectiontitle">Description</h4><ul id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_ul12220145108"><li id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_li82201045703">ROW_NUMBER(): Assigns a unique, sequential number to each row, starting with one.</li><li id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_li152201045504">PARTITION BY col1[, col2...]: Specifies the partition columns, for example, the deduplicate key.</li><li id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_li1422020451403">ORDER BY time_attr [asc|desc]: Specifies the ordering column, it must be a time attribute. Currently Flink supports proctime only. Ordering by ASC means to keep the first row, ordering by DESC means to keep the last row.</li><li id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_li1922012451800">WHERE rownum = 1: The rownum = 1 is required for Flink to recognize this query is deduplication.</li></ul>
</div>
<div class="section" id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_section17171113913120"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0328__en-us_topic_0000001119232092_en-us_topic_0000001132426601_p10805114812117">None</p>
</div>

File diff suppressed because it is too large Load Diff

View File

@ -1,19 +1,22 @@
<a name="dli_08_0349"></a><a name="dli_08_0349"></a>
<h1 class="topictitle1">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</h1>
<div id="body0000001164736148"><p id="dli_08_0349__p8060118"></p>
<h1 class="topictitle1">Backing Up and Restoring Data of Multiple Versions</h1>
<div id="body0000001164736148"><p id="dli_08_0349__p8060118"></p>
</div>
<div>
<ul class="ullinks">
<li class="ulchildlink"><strong><a href="dli_08_0350.html">Setting the Retention Period of Multi-Versioning Backup Data (Deprecated, Not Recommended)</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_08_0350.html">Setting the Retention Period for Multiversion Backup Data</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_08_0351.html">Viewing Multi-Versioning Backup Data (Deprecated, Not Recommended)</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_08_0351.html">Checking Multiversion Backup Data</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_08_0352.html">Restoring Multi-Versioning Backup Data (Deprecated, Not Recommended)</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_08_0352.html">Restoring Multiversion Backup Data</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_08_0353.html">Configuring the Recycle Bin for Expired Multi-Versioning Data (Deprecated, Not Recommended)</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_08_0353.html">Configuring the Trash Bin for Expired Multiversion Data</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_08_0355.html">Clearing Multi-Versioning Data (Deprecated, Not Recommended)</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_08_0355.html">Deleting Multiversion Backup Data</a></strong><br>
</li>
</ul>

View File

@ -1,9 +1,8 @@
<a name="dli_08_0350"></a><a name="dli_08_0350"></a>
<h1 class="topictitle1">Setting the Retention Period of Multi-Versioning Backup Data (Deprecated, Not Recommended)</h1>
<div id="body0000001164577596"><div class="section" id="dli_08_0350__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0350__p16127159363">The multi-versioning feature is to be deprecated and is not recommended. </p>
<p id="dli_08_0350__p13340145761319">After multiversion is enabled, backup data is retained for seven days by default. You can change the retention period by setting system parameter<strong id="dli_08_0350__b4986125961619"> dli.multi.version.retention.days</strong>. Multiversion data out of the retention period will be automatically deleted when the <strong id="dli_08_0350__b762172213182">insert overwrite</strong> or <strong id="dli_08_0350__b43001226161810">truncate</strong> statement is executed. You can also set table attribute <strong id="dli_08_0350__b6149135817184">dli.multi.version.retention.days</strong> to adjust the retention period when adding a column or modifying a partitioned table.</p>
<p id="dli_08_0350__p1678843136">For details about the syntax for enabling or disabling the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Data Multi-Versioning (Deprecated, Not Recommended)</a>.</p>
<h1 class="topictitle1">Setting the Retention Period for Multiversion Backup Data</h1>
<div id="body0000001164577596"><div class="section" id="dli_08_0350__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0350__p13340145761319">After multiversion is enabled, backup data is retained for seven days by default. You can change the retention period by setting system parameter<strong id="dli_08_0350__b4986125961619"> dli.multi.version.retention.days</strong>. Multiversion data out of the retention period will be automatically deleted when the <strong id="dli_08_0350__b762172213182">insert overwrite</strong> or <strong id="dli_08_0350__b43001226161810">truncate</strong> statement is executed. You can also set table attribute <strong id="dli_08_0350__b6149135817184">dli.multi.version.retention.days</strong> to adjust the retention period when adding a column or modifying a partitioned table.</p>
<p id="dli_08_0350__p1678843136">For details about the syntax for enabling or disabling the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Multiversion Backup</a>.</p>
<p id="dli_08_0350__p699538151513">Currently, the multiversion function supports only OBS tables created using the Hive syntax. For details about the syntax for creating a table, see <a href="dli_08_0077.html">Creating an OBS Table Using the Hive Syntax</a>.</p>
</div>
<div class="section" id="dli_08_0350__en-us_topic_0114776165_en-us_topic_0093946907_s24c0b560c57b49d1bba9b64b7baf3f21"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0350__en-us_topic_0114776165_screen141961254556"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
@ -52,7 +51,7 @@
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Data of Multiple Versions</a></div>
</div>
</div>

View File

@ -1,8 +1,7 @@
<a name="dli_08_0351"></a><a name="dli_08_0351"></a>
<h1 class="topictitle1">Viewing Multi-Versioning Backup Data (Deprecated, Not Recommended)</h1>
<div id="body0000001210296071"><div class="section" id="dli_08_0351__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0351__p16127159363">The multi-versioning feature is to be deprecated and is not recommended. </p>
<p id="dli_08_0351__p1678843136">After the multiversion function is enabled, you can run the <strong id="dli_08_0351__b16491031153317">SHOW HISTORY</strong> command to view the backup data of a table. For details about the syntax for enabling or disabling the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Data Multi-Versioning (Deprecated, Not Recommended)</a>.</p>
<h1 class="topictitle1">Checking Multiversion Backup Data</h1>
<div id="body0000001210296071"><div class="section" id="dli_08_0351__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0351__p1678843136">After the multiversion function is enabled, you can run the <strong id="dli_08_0351__b16491031153317">SHOW HISTORY</strong> command to view the backup data of a table. For details about the syntax for enabling or disabling the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Multiversion Backup</a>.</p>
<p id="dli_08_0351__p79746134159">Currently, the multiversion function supports only OBS tables created using the Hive syntax. For details about the syntax for creating a table, see <a href="dli_08_0077.html">Creating an OBS Table Using the Hive Syntax</a>.</p>
</div>
<div class="section" id="dli_08_0351__en-us_topic_0114776165_en-us_topic_0093946907_s24c0b560c57b49d1bba9b64b7baf3f21"><h4 class="sectiontitle">Syntax</h4><ul id="dli_08_0351__ul864214239471"><li id="dli_08_0351__li136428239473">View the backup data of a non-partitioned table.<pre class="screen" id="dli_08_0351__screen592211334812"><strong id="dli_08_0351__b8186133963318">SHOW HISTORY FOR TABLE</strong> [db_name.]table_name;</pre>
@ -56,7 +55,7 @@
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Data of Multiple Versions</a></div>
</div>
</div>

View File

@ -1,15 +1,14 @@
<a name="dli_08_0352"></a><a name="dli_08_0352"></a>
<h1 class="topictitle1">Restoring Multi-Versioning Backup Data (Deprecated, Not Recommended)</h1>
<div id="body0000001209977507"><div class="section" id="dli_08_0352__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0352__p16127159363">The multi-versioning feature is to be deprecated and is not recommended. </p>
<p id="dli_08_0352__p1678843136">After the multiversion function is enabled, you can run the <strong id="dli_08_0352__b1759201055810">RESTORE TABLE</strong> statement to restore a table or partition of a specified version. For details about the syntax for enabling or disabling the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Data Multi-Versioning (Deprecated, Not Recommended)</a>.</p>
<h1 class="topictitle1">Restoring Multiversion Backup Data</h1>
<div id="body0000001209977507"><div class="section" id="dli_08_0352__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0352__p1678843136">After the multiversion function is enabled, you can run the <strong id="dli_08_0352__b1759201055810">RESTORE TABLE</strong> statement to restore a table or partition of a specified version. For details about the syntax for enabling or disabling the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Multiversion Backup</a>.</p>
<p id="dli_08_0352__p157671522171515">Currently, the multiversion function supports only OBS tables created using the Hive syntax. For details about the syntax for creating a table, see <a href="dli_08_0077.html">Creating an OBS Table Using the Hive Syntax</a>.</p>
</div>
<div class="section" id="dli_08_0352__en-us_topic_0114776165_en-us_topic_0093946907_s24c0b560c57b49d1bba9b64b7baf3f21"><h4 class="sectiontitle">Syntax</h4><ul id="dli_08_0352__ul864214239471"><li id="dli_08_0352__li136428239473">Restore the non-partitioned table data to the backup data of a specified version.<pre class="screen" id="dli_08_0352__screen592211334812"><strong id="dli_08_0352__b1246553118250">RESTORE TABLE</strong> [db_name.]table_name<strong id="dli_08_0352__b2816153411252"> TO VERSION</strong> 'version_id';</pre>
</li><li id="dli_08_0352__li136517574819">Restore the data of a single partition in a partitioned table to the backup data of a specified version.<pre class="screen" id="dli_08_0352__screen9615636184811"><strong id="dli_08_0352__b16814837172510">RESTORE TABLE</strong> [db_name.]table_name <strong id="dli_08_0352__b77258412257">PARTITION </strong>(column = value, ...) <strong id="dli_08_0352__b140184572514">TO VERSION</strong> 'version_id';</pre>
</li></ul>
</div>
<div class="section" id="dli_08_0352__en-us_topic_0114776165_en-us_topic_0093946907_s15f56ad46e8f476986860b44d58d0545"><h4 class="sectiontitle">Keywords</h4><ul id="dli_08_0352__en-us_topic_0114776165_en-us_topic_0093946907_ul19329628111649"><li id="dli_08_0352__en-us_topic_0114776165_en-us_topic_0093946907_li34892402111649">RESTORE TABLE: Used to restore backup data</li><li id="dli_08_0352__li113091203414">PARTITION: Used to specify the partition column</li><li id="dli_08_0352__li196161910443">TO VERSION: Used to specify the version number You can run the <strong id="dli_08_0352__b6124177992">SHOW HISTORY</strong> command to obtain the version number. For details, see <a href="dli_08_0351.html">Viewing Multi-Versioning Backup Data (Deprecated, Not Recommended)</a>.</li></ul>
<div class="section" id="dli_08_0352__en-us_topic_0114776165_en-us_topic_0093946907_s15f56ad46e8f476986860b44d58d0545"><h4 class="sectiontitle">Keywords</h4><ul id="dli_08_0352__en-us_topic_0114776165_en-us_topic_0093946907_ul19329628111649"><li id="dli_08_0352__en-us_topic_0114776165_en-us_topic_0093946907_li34892402111649">RESTORE TABLE: Used to restore backup data</li><li id="dli_08_0352__li113091203414">PARTITION: Used to specify the partition column</li><li id="dli_08_0352__li196161910443">TO VERSION: Used to specify the version number You can run the <strong id="dli_08_0352__b6124177992">SHOW HISTORY</strong> command to obtain the version number. For details, see <a href="dli_08_0351.html">Checking Multiversion Backup Data</a>.</li></ul>
</div>
<div class="section" id="dli_08_0352__en-us_topic_0114776170_section1130610201331"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0352__en-us_topic_0114776170_table137579239220" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="dli_08_0352__en-us_topic_0114776170_row17564232219"><th align="left" class="cellrowborder" valign="top" width="21.3%" id="mcps1.3.4.2.2.3.1.1"><p id="dli_08_0352__en-us_topic_0114776170_p3756152318220">Parameter</p>
@ -40,7 +39,7 @@
</tr>
<tr id="dli_08_0352__row1754218361844"><td class="cellrowborder" valign="top" width="21.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0352__p145773531747">version_id</p>
</td>
<td class="cellrowborder" valign="top" width="78.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0352__p754217364418">Target version of the backup data to be restored You can run the <strong id="dli_08_0352__b8454223191012">SHOW HISTORY</strong> command to obtain the version number. For details, see <a href="dli_08_0351.html">Viewing Multi-Versioning Backup Data (Deprecated, Not Recommended)</a>.</p>
<td class="cellrowborder" valign="top" width="78.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0352__p754217364418">Target version of the backup data to be restored You can run the <strong id="dli_08_0352__b8454223191012">SHOW HISTORY</strong> command to obtain the version number. For details, see <a href="dli_08_0351.html">Checking Multiversion Backup Data</a>.</p>
</td>
</tr>
</tbody>
@ -61,7 +60,7 @@
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Data of Multiple Versions</a></div>
</div>
</div>

View File

@ -1,8 +1,7 @@
<a name="dli_08_0353"></a><a name="dli_08_0353"></a>
<h1 class="topictitle1">Configuring the Recycle Bin for Expired Multi-Versioning Data (Deprecated, Not Recommended)</h1>
<div id="body0000001164417608"><div class="section" id="dli_08_0353__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0353__p16127159363">The multi-versioning feature is to be deprecated and is not recommended. </p>
<p id="dli_08_0353__p1678843136">After the multiversion function is enabled, expired backup data will be directly deleted by the system when the<strong id="dli_08_0353__b17132111719172"> insert overwrite</strong> or <strong id="dli_08_0353__b1911152014177">truncate</strong> statement is executed. You can configure the trash bin of the OBS parallel file system to accelerate the deletion of expired backup data. To enable the trash bin, add <strong id="dli_08_0353__b11428652131814">dli.multi.version.trash.dir</strong> to the table properties. For details about the syntax for enabling or disabling the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Data Multi-Versioning (Deprecated, Not Recommended)</a>.</p>
<h1 class="topictitle1">Configuring the Trash Bin for Expired Multiversion Data</h1>
<div id="body0000001164417608"><div class="section" id="dli_08_0353__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0353__p1678843136">After the multiversion function is enabled, expired backup data will be directly deleted by the system when the<strong id="dli_08_0353__b17132111719172"> insert overwrite</strong> or <strong id="dli_08_0353__b1911152014177">truncate</strong> statement is executed. You can configure the trash bin of the OBS parallel file system to accelerate the deletion of expired backup data. To enable the trash bin, add <strong id="dli_08_0353__b11428652131814">dli.multi.version.trash.dir</strong> to the table properties. For details about the syntax for enabling or disabling the multiversion function, see <a href="dli_08_0354.html">Enabling or Disabling Multiversion Backup</a>.</p>
<p id="dli_08_0353__p18220193723915">Currently, the multiversion function supports only OBS tables created using the Hive syntax. For details about the syntax for creating a table, see <a href="dli_08_0077.html">Creating an OBS Table Using the Hive Syntax</a>.</p>
</div>
<div class="section" id="dli_08_0353__en-us_topic_0114776165_en-us_topic_0093946907_s24c0b560c57b49d1bba9b64b7baf3f21"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0353__en-us_topic_0114776165_screen141961254556"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
@ -52,7 +51,7 @@
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Data of Multiple Versions</a></div>
</div>
</div>

View File

@ -1,8 +1,7 @@
<a name="dli_08_0354"></a><a name="dli_08_0354"></a>
<h1 class="topictitle1">Enabling or Disabling Data Multi-Versioning (Deprecated, Not Recommended)</h1>
<div id="body0000001209844553"><div class="section" id="dli_08_0354__en-us_topic_0156816309_en-us_topic_0093946741_s86887e0a86644c22b61a4c2ccc84025e"><h4 class="sectiontitle">Function</h4><p id="dli_08_0354__p16127159363">The multi-versioning feature is to be deprecated and is not recommended. </p>
<p id="dli_08_0354__en-us_topic_0156816309_en-us_topic_0093946741_a9755a68b9d0d47668ac554151d03e0ce">DLI controls multiple versions of backup data for restoration. After the multiversion function is enabled, the system automatically backs up table data when you delete or modify the data using <strong id="dli_08_0354__b1798871681213">insert overwrite</strong> or <strong id="dli_08_0354__b161591819191220">truncate</strong>, and retains the data for a certain period. You can quickly restore data within the retention period. For details about the syntax related to the multiversion function, see <a href="dli_08_0349.html">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</a>.</p>
<h1 class="topictitle1">Enabling or Disabling Multiversion Backup</h1>
<div id="body0000001209844553"><div class="section" id="dli_08_0354__en-us_topic_0156816309_en-us_topic_0093946741_s86887e0a86644c22b61a4c2ccc84025e"><h4 class="sectiontitle">Function</h4><p id="dli_08_0354__en-us_topic_0156816309_en-us_topic_0093946741_a9755a68b9d0d47668ac554151d03e0ce">DLI controls multiple versions of backup data for restoration. After the multiversion function is enabled, the system automatically backs up table data when you delete or modify the data using <strong id="dli_08_0354__b1798871681213">insert overwrite</strong> or <strong id="dli_08_0354__b161591819191220">truncate</strong>, and retains the data for a certain period. You can quickly restore data within the retention period. For details about the syntax related to the multiversion function, see <a href="dli_08_0349.html">Backing Up and Restoring Data of Multiple Versions</a>.</p>
<p id="dli_08_0354__p18220193723915">Currently, the multiversion function supports only OBS tables created using the Hive syntax. For details about the syntax for creating a table, see <a href="dli_08_0077.html">Creating an OBS Table Using the Hive Syntax</a>.</p>
</div>
<div class="section" id="dli_08_0354__en-us_topic_0156816309_en-us_topic_0093946741_sd6725bae42f0429e8bc6dfa2e92b9664"><h4 class="sectiontitle">Syntax</h4><ul id="dli_08_0354__ul199756384918"><li id="dli_08_0354__li597618316496">Enable the multiversion function.<pre class="screen" id="dli_08_0354__screen121651215164918"><strong id="dli_08_0354__b10266754135512">ALTER TABLE</strong> [db_name.]table_name

View File

@ -1,8 +1,7 @@
<a name="dli_08_0355"></a><a name="dli_08_0355"></a>
<h1 class="topictitle1">Clearing Multi-Versioning Data (Deprecated, Not Recommended)</h1>
<div id="body0000001210069181"><div class="section" id="dli_08_0355__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0355__p16127159363">The multi-versioning feature is to be deprecated and is not recommended. </p>
<p id="dli_08_0355__p1678843136">The retention period of multiversion backup data takes effect each time the <strong id="dli_08_0355__b12559616114411">insert overwrite</strong> or <strong id="dli_08_0355__b6361121817446">truncate</strong> statement is executed. If neither statement is executed for the table, multiversion backup data out of the retention period will not be automatically deleted. You can run the SQL commands described in this section to manually delete multiversion backup data.</p>
<h1 class="topictitle1">Deleting Multiversion Backup Data</h1>
<div id="body0000001210069181"><div class="section" id="dli_08_0355__en-us_topic_0114776165_en-us_topic_0093946907_s487369d31d4243aba0e537e5932f8b36"><h4 class="sectiontitle">Function</h4><p id="dli_08_0355__p1678843136">The retention period of multiversion backup data takes effect each time the <strong id="dli_08_0355__b12559616114411">insert overwrite</strong> or <strong id="dli_08_0355__b6361121817446">truncate</strong> statement is executed. If neither statement is executed for the table, multiversion backup data out of the retention period will not be automatically deleted. You can run the SQL commands described in this section to manually delete multiversion backup data.</p>
</div>
<div class="section" id="dli_08_0355__en-us_topic_0114776165_en-us_topic_0093946907_s24c0b560c57b49d1bba9b64b7baf3f21"><h4 class="sectiontitle">Syntax</h4><div class="p" id="dli_08_0355__p174371516112120">Delete multiversion backup data out of the retention period.<pre class="screen" id="dli_08_0355__screen592211334812"><strong id="dli_08_0355__b18957442192618">clear history for table</strong> [db_name.]table_name <strong id="dli_08_0355__b1132515461261">older_than </strong>'timestamp';</pre>
</div>
@ -44,7 +43,7 @@
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Multi-Versioning Data (Deprecated, Not Recommended)</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0349.html">Backing Up and Restoring Data of Multiple Versions</a></div>
</div>
</div>

View File

@ -25,7 +25,7 @@
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.3.2.4.1.2 "><p id="dli_08_0356__en-us_topic_0000001214500927_en-us_topic_0000001168912384_p84318620278">VARCHAR</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.3.2.4.1.3 "><p id="dli_08_0356__en-us_topic_0000001214500927_en-us_topic_0000001168912384_p14431166182710">Separator. Currently, only single-character separators are supported.</p>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.3.2.4.1.3 "><p id="dli_08_0356__en-us_topic_0000001214500927_en-us_topic_0000001168912384_p14431166182710">Delimiter. Currently, only single-character delimiters are supported.</p>
</td>
</tr>
</tbody>

View File

@ -21,7 +21,7 @@
);</pre>
</div>
<div class="section" id="dli_08_0385__en-us_topic_0000001310015801_section1918619215263"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0385__en-us_topic_0000001310015801_table3535131492817" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="dli_08_0385__en-us_topic_0000001310015801_row053518147287"><th align="left" class="cellrowborder" valign="top" width="22.29%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0385__en-us_topic_0000001310015801_p153513145285">Parameter</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0385__en-us_topic_0000001310015801_table3535131492817" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0385__en-us_topic_0000001310015801_row053518147287"><th align="left" class="cellrowborder" valign="top" width="22.29%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0385__en-us_topic_0000001310015801_p153513145285">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="8.55%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0385__en-us_topic_0000001310015801_p10535014202815">Mandatory</p>
</th>

View File

@ -3,7 +3,7 @@
<h1 class="topictitle1">Postgres CDC Source Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section16834144252514"><h4 class="sectiontitle">Function</h4><p id="dli_08_0388__en-us_topic_0000001310215785_p4587104618253">The Postgres CDC source table, that is, Postgres streaming source table, is used to read the full snapshot data and changed data of the PostgreSQL database in sequence. The exactly-once processing semantics is used to ensure data accuracy even if a failure occurs.</p>
</div>
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section4375753152514"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0388__en-us_topic_0000001310215785_ul1170451184015"><li id="dli_08_0388__en-us_topic_0000001310215785_li107041511114015">The PostgreSQL version is 9.6, 10, 11, or 12.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li18597171744011">An enhanced datasource connection with the database has been established, so that you can configure security group rules as required.
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section4375753152514"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0388__en-us_topic_0000001310215785_ul1170451184015"><li id="dli_08_0388__en-us_topic_0000001310215785_li107041511114015">The PostgreSQL version be 9.6, 10, 11, or 12.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li18597171744011">An enhanced datasource connection with the database has been established, so that you can configure security group rules as required.
</li></ul>
</div>
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section1852611002610"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0388__en-us_topic_0000001310215785_ul1552562641320"><li id="dli_08_0388__en-us_topic_0000001310215785_li5709020162317">When you create a Flink OpenSource SQL job, set <strong id="dli_08_0388__en-us_topic_0000001310215785_b1124516290305">Flink Version</strong> to <strong id="dli_08_0388__en-us_topic_0000001310215785_b11246102911303">1.12</strong> on the <strong id="dli_08_0388__en-us_topic_0000001310215785_b1124652911308">Running Parameters</strong> tab of the job editing page, select <strong id="dli_08_0388__en-us_topic_0000001310215785_b1824617294303">Save Job Log</strong>, and set the OBS bucket for saving job logs.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li4525202631311">The PostgreSQL version cannot be earlier than PostgreSQL 11.</li><li id="dli_08_0388__en-us_topic_0000001310215785_li118251538161814">If operations such as update will be performed on the Postgres table, you need to run the following statement in PostgreSQL. Note: Replace <strong id="dli_08_0388__en-us_topic_0000001310215785_b9253442103112">test.cdc_order</strong> with the actual database and table.<pre class="screen" id="dli_08_0388__en-us_topic_0000001310215785_screen8340431181914">ALTER TABLE <em id="dli_08_0388__en-us_topic_0000001310215785_i4560579314">test.cdc_order</em> REPLICA IDENTITY FULL</pre>
@ -27,7 +27,7 @@ with (
);</pre>
</div>
<div class="section" id="dli_08_0388__en-us_topic_0000001310215785_section183609813304"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0388__en-us_topic_0000001310215785_table28274351306" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="dli_08_0388__en-us_topic_0000001310215785_row19827735103012"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0388__en-us_topic_0000001310215785_p108275356303">Parameter</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0388__en-us_topic_0000001310215785_table28274351306" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0388__en-us_topic_0000001310215785_row19827735103012"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0388__en-us_topic_0000001310215785_p108275356303">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="6.909999999999999%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0388__en-us_topic_0000001310215785_p98271635103012">Mandatory</p>
</th>

File diff suppressed because it is too large Load Diff

View File

@ -11,10 +11,7 @@
<p id="dli_08_0466__p145017511538">Assume that the lifecycle of a partitioned table is one day and the last modification time of the partitioned data is 15:00 on May 20, 2023. If the table is scanned before 15:00 on May 20, 2023 (less than one day), the partitions in the table will not be reclaimed. If the last data modification time (<strong id="dli_08_0466__b111213521113">LAST_ACCESS_TIME</strong>) of a table partition exceeds the time specified by the lifecycle during reclamation scan on May 20, 2023, the partition will be reclaimed.</p>
</li><li id="dli_08_0466__li2918122115338">The lifecycle function periodically reclaims tables or partitions, which are reclaimed irregularly every day depending on the level of busyness of the service. It cannot ensure that a table or partition will be reclaimed immediately after its lifecycle expires.</li><li id="dli_08_0466__li19244155012335">After a table is deleted, all properties of the table, including the lifecycle, will be deleted. After a table with the same name is created again, the lifecycle of the table will be determined by the new property.</li></ul>
</div>
<div class="section" id="dli_08_0466__section0796155182119"><h4 class="sectiontitle">Notes and Constraints</h4><ul id="dli_08_0466__ul162161589267"><li id="dli_08_0466__li10677430152518">You are advised to configure the <strong id="dli_08_0466__b11331345366">dli_data_clean_agency</strong> agency before using the lifecycle feature.<p id="dli_08_0466__p340019565252">For details about agency permission policies, refer to "Agency Permission Policies in Common Scenarios" in <em id="dli_08_0466__i16122104183916">Data Lake Insight User Guide</em>.</p>
<p id="dli_08_0466__p205181248165617">If the <strong id="dli_08_0466__b1762183415391">dli_data_clean_agency</strong> agency is not configured, the system reads the previous-generation DLI system agency <strong id="dli_08_0466__b3267124616391">dli_admin_agency</strong> by default. However, if <strong id="dli_08_0466__b750611524016">dli_admin_agency</strong> is not configured for your account, the current table lifecycle feature cannot be used.</p>
<p id="dli_08_0466__p15517165114583">You can check whether you have the <strong id="dli_08_0466__b5976910134114">dli_admin_agency</strong> agency in IAM agencies.</p>
</li><li id="dli_08_0466__li18263433152214">The table lifecycle function currently only supports creating tables and versioning tables using Hive and Datasource syntax.</li><li id="dli_08_0466__li753534652018">The unit of the lifecycle is in days. The value should be a positive integer.</li><li id="dli_08_0466__li15403171564215">The lifecycle can be set only at the table level. The lifecycle specified for a partitioned table applies to all partitions of the table.</li><li id="dli_08_0466__li1373233532617">After the lifecycle is set, DLI and OBS tables will support data backup. The backup directory for OBS tables needs to be set manually. The backup directory must be in the parallel file system and in the same bucket as the original table directory. It cannot have the same directory or subdirectory name as the original table.</li></ul>
<div class="section" id="dli_08_0466__section0796155182119"><h4 class="sectiontitle">Notes and Constraints</h4><ul id="dli_08_0466__ul162161589267"><li id="dli_08_0466__li1854174611252">Before using the lifecycle function, log in to the DLI console, choose <strong id="dli_08_0466__b13786112215140">Global Configuration</strong> &gt; <strong id="dli_08_0466__b1527753091417">Service Authorization</strong>, select <strong id="dli_08_0466__b42312321169">dli_admin_agency</strong>, and click <strong id="dli_08_0466__b1973011491717">Update</strong>.</li><li id="dli_08_0466__li18263433152214">The table lifecycle function currently only supports creating tables and versioning tables using Hive and Datasource syntax.</li><li id="dli_08_0466__li753534652018">The unit of the lifecycle is in days. The value should be a positive integer.</li><li id="dli_08_0466__li15403171564215">The lifecycle can be set only at the table level. The lifecycle specified for a partitioned table applies to all partitions of the table.</li><li id="dli_08_0466__li1373233532617">After the lifecycle is set, DLI and OBS tables will support data backup. The backup directory for OBS tables needs to be set manually. The backup directory must be in the parallel file system and in the same bucket as the original table directory. It cannot have the same directory or subdirectory name as the original table.</li></ul>
</div>
<div class="section" id="dli_08_0466__section182651545162119"><h4 class="sectiontitle">Syntax</h4><ul id="dli_08_0466__ul1942664454817"><li id="dli_08_0466__li942624418487"><strong id="dli_08_0466__b2041013431482">Creating a DLI table using the Datasource syntax</strong><pre class="screen" id="dli_08_0466__screen3458194912483">CREATE TABLE table_name(name string, id int)
USING parquet

View File

@ -15,7 +15,7 @@
</ul>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15000.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="en-us_topic_0000002172127506.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
</div>
</div>

View File

@ -95,7 +95,7 @@
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15000.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="en-us_topic_0000002172127506.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
</div>
</div>

View File

@ -31,7 +31,7 @@
</ul>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15000.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="en-us_topic_0000002172127506.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
</div>
</div>

View File

@ -1,7 +1,7 @@
<a name="dli_08_15020"></a><a name="dli_08_15020"></a>
<h1 class="topictitle1">Debezium</h1>
<div id="body0000001310215797"><div class="section" id="dli_08_15020__section9536315265"><h4 class="sectiontitle">Function</h4><p id="dli_08_15020__p4754154145319">Debezium is a Changelog Data Capture (CDC) tool that can stream changes in real-time from other databases into Kafka. Debezium provides a unified format schema for changelog and supports to serialize messages using JSON and Apache Avro.</p>
<div id="body0000001310215797"><div class="section" id="dli_08_15020__section9536315265"><h4 class="sectiontitle">Function</h4><p id="dli_08_15020__p4754154145319">Debezium is a Changelog Data Capture (CDC) tool that can stream changes in real-time from MySQL, PostgreSQL, Oracle, Microsoft SQL Server and many other databases into Kafka. Debezium provides a unified format schema for changelog and supports to serialize messages using JSON and Apache Avro.</p>
<p id="dli_08_15020__p12357153614535">Flink supports to interpret Debezium JSON and Avro messages as INSERT/UPDATE/DELETE messages into Flink SQL system. This is useful in many cases to leverage this feature, such as</p>
<ul id="dli_08_15020__ul203571936155312"><li id="dli_08_15020__li1335753610532">Synchronizing incremental data from databases to other systems</li><li id="dli_08_15020__li13357173620536">Auditing logs</li><li id="dli_08_15020__li1035753665315">Real-time materialized views on databases</li><li id="dli_08_15020__li63571836145314">Temporal join changing history of a database table</li></ul>
<p id="dli_08_15020__p1835743614531">Flink also supports to encode the INSERT/UPDATE/DELETE messages in Flink SQL as Debezium JSON or Avro messages, and emit to external systems like Kafka. However, currently Flink cannot combine UPDATE_BEFORE and UPDATE_AFTER into a single UPDATE message. Therefore, Flink encodes UPDATE_BEFORE and UDPATE_AFTER as DELETE and INSERT Debezium messages.</p>

View File

@ -40,7 +40,7 @@
</ul>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15000.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="en-us_topic_0000002172127506.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
</div>
</div>

View File

@ -1,7 +1,7 @@
<a name="dli_08_15033"></a><a name="dli_08_15033"></a>
<h1 class="topictitle1">Overview</h1>
<div id="body0000001849546341"><p id="dli_08_15033__p2544124910199">The Flink Doris Connector can support operations (read, insert, modify, delete) data stored in Doris through Flink. For details, see <a href="https://doris.apache.org/docs/2.0/ecosystem/flink-doris-connector" target="_blank" rel="noopener noreferrer">Flink Doris Connector</a>.</p>
<div id="body0000001849546341"><p id="dli_08_15033__p2544124910199">The Flink Doris Connector can support operations (read, insert, modify, delete) data stored in Doris through Flink.</p>
<div class="note" id="dli_08_15033__note210313551736"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_15033__p17970145511242">Only tables in the Unique Key model can be modified or deleted.</p>
</div></div>

View File

@ -168,7 +168,7 @@ select
orders.user_name,
dim.city,
dim.age,
dim.gender
dim.sex
from ordersSource orders
left join dorisDemo for system_time as of orders.proctime as dim on orders.user_id = dim.user_id;</pre>
</li><li id="dli_08_15036__li55501154153">Write two data records to the Kafka data source.<pre class="screen" id="dli_08_15036__screen370220338333">{"user_id": "user1", "user_name": "name1"}

View File

@ -27,7 +27,7 @@
</ul>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15000.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="en-us_topic_0000002172127506.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
</div>
</div>

View File

@ -15,7 +15,7 @@
</ul>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15000.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="en-us_topic_0000002172127506.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
</div>
</div>

View File

@ -173,7 +173,7 @@ from
dataGenSource;</pre>
</div>
<div class="section" id="dli_08_15084__section1247126192613"><h4 class="sectiontitle">Output</h4><p id="dli_08_15084__p2315155333014">On the Flin Jobs page, locate your job, and click <strong id="dli_08_15084__b158561459179">More</strong> &gt; <strong id="dli_08_15084__b1863171685">FlinkUI</strong> in the <strong id="dli_08_15084__b1282416261810">Operation</strong> column. On the displayed page, click <strong id="dli_08_15084__b18944183819810">Task Managers</strong> &gt; <strong id="dli_08_15084__b3633144018813">Stdout</strong> to view the job output.</p>
<div class="fignone" id="dli_08_15084__fig922611433229"><span class="figcap"><b>Figure 1 </b>Flink job output result</span><br><span><img id="dli_08_15084__image17226343162211" src="en-us_image_0000002264181102.png"></span></div>
<p id="dli_08_15084__p2143151011263"><span><img id="dli_08_15084__image1350521020269" src="en-us_image_0000001742720709.gif"></span></p>
</div>
</div>
<div>

View File

@ -10,7 +10,7 @@
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_15000.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
<div class="parentlink"><strong>Parent topic:</strong> <a href="en-us_topic_0000002172127506.html">Flink OpenSource SQL 1.15 Syntax Reference</a></div>
</div>
</div>

View File

@ -42,7 +42,7 @@
</td>
<td class="cellrowborder" valign="top" width="13.44%" headers="mcps1.3.4.2.2.5.1.2 "><p id="dli_spark_dateadd__en-us_topic_0000001694195245_p12830747122519">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="20.61%" headers="mcps1.3.4.2.2.5.1.3 "><p id="dli_spark_dateadd__en-us_topic_0000001694195245_p1731894542616">STRING</p>
<td class="cellrowborder" valign="top" width="20.61%" headers="mcps1.3.4.2.2.5.1.3 "><p id="dli_spark_dateadd__en-us_topic_0000001694195245_p1731894542616">BIGINT</p>
</td>
<td class="cellrowborder" valign="top" width="46.73%" headers="mcps1.3.4.2.2.5.1.4 "><p id="dli_spark_dateadd__en-us_topic_0000001694195245_p15063314374">Unit, based on which the date is modified</p>
<p id="dli_spark_dateadd__en-us_topic_0000001694195245_p1995034510202">This parameter supports the following extended date formats: year, month or mon, day, and hour.</p>

View File

@ -43,7 +43,7 @@
<td class="cellrowborder" valign="top" width="15.590000000000002%" headers="mcps1.3.4.2.2.5.1.3 "><p id="dli_spark_instr1__en-us_topic_0000001696219573_p49645111118">BIGINT</p>
</td>
<td class="cellrowborder" valign="top" width="51.77%" headers="mcps1.3.4.2.2.5.1.4 "><p id="dli_spark_instr1__en-us_topic_0000001696219573_p861212152116">Sequence number of the character in str1 the search starts from. The default start position is position 1 (position of the first character).</p>
<p id="dli_spark_instr1__p13314173913">Negative numbers are not supported.</p>
<p id="dli_spark_instr1__en-us_topic_0000001696219573_p18241749210">If this parameter is set to a negative number, the search starts from the end to the beginning of the string, and the last character is <strong id="dli_spark_instr1__en-us_topic_0000001696219573_b11781162614296">-1</strong>.</p>
</td>
</tr>
<tr id="dli_spark_instr1__en-us_topic_0000001696219573_row2964311816"><td class="cellrowborder" valign="top" width="19.220000000000002%" headers="mcps1.3.4.2.2.5.1.1 "><p id="dli_spark_instr1__en-us_topic_0000001696219573_p596431117114">nth_appearance</p>

View File

@ -24,7 +24,7 @@
<td class="cellrowborder" valign="top" width="51.77%" headers="mcps1.3.3.2.2.5.1.4 "><p id="dli_spark_url_decode__en-us_topic_0000001655381222_p9484232112711">String to be entered</p>
</td>
</tr>
<tr id="dli_spark_url_decode__en-us_topic_0000001655381222_row651959205518"><td class="cellrowborder" valign="top" width="19.220000000000002%" headers="mcps1.3.3.2.2.5.1.1 "><p id="dli_spark_url_decode__en-us_topic_0000001655381222_p135201894550">encoding</p>
<tr id="dli_spark_url_decode__en-us_topic_0000001655381222_row651959205518"><td class="cellrowborder" valign="top" width="19.220000000000002%" headers="mcps1.3.3.2.2.5.1.1 "><p id="dli_spark_url_decode__en-us_topic_0000001655381222_p135201894550">endcoding</p>
</td>
<td class="cellrowborder" valign="top" width="13.44%" headers="mcps1.3.3.2.2.5.1.2 "><p id="dli_spark_url_decode__en-us_topic_0000001655381222_p752015985511">No</p>
</td>

View File

@ -24,7 +24,7 @@
<td class="cellrowborder" valign="top" width="51.77%" headers="mcps1.3.3.2.2.5.1.4 "><p id="dli_spark_url_encode__en-us_topic_0000001655540578_p9484232112711">String to be entered</p>
</td>
</tr>
<tr id="dli_spark_url_encode__en-us_topic_0000001655540578_row145934325116"><td class="cellrowborder" valign="top" width="19.220000000000002%" headers="mcps1.3.3.2.2.5.1.1 "><p id="dli_spark_url_encode__en-us_topic_0000001655540578_p1660184318512">encoding</p>
<tr id="dli_spark_url_encode__en-us_topic_0000001655540578_row145934325116"><td class="cellrowborder" valign="top" width="19.220000000000002%" headers="mcps1.3.3.2.2.5.1.1 "><p id="dli_spark_url_encode__en-us_topic_0000001655540578_p1660184318512">endcoding</p>
</td>
<td class="cellrowborder" valign="top" width="13.43%" headers="mcps1.3.3.2.2.5.1.2 "><p id="dli_spark_url_encode__en-us_topic_0000001655540578_p1260194395115">No</p>
</td>

Binary file not shown.

After

Width:  |  Height:  |  Size: 117 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 318 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 39 KiB

View File

@ -1,7 +1,7 @@
<a name="dli_08_15000"></a><a name="dli_08_15000"></a>
<a name="EN-US_TOPIC_0000002172127506"></a><a name="EN-US_TOPIC_0000002172127506"></a>
<h1 class="topictitle1">Flink OpenSource SQL 1.15 Syntax Reference</h1>
<div id="body0000002172127506"><p id="dli_08_15000__p8060118"></p>
<div id="body0000002172127506"><p id="EN-US_TOPIC_0000002172127506__p8060118"></p>
</div>
<div>
<ul class="ullinks">