forked from docs/doc-exports
Su, Xiaomeng
76a5b1ee83
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
149 lines
19 KiB
HTML
149 lines
19 KiB
HTML
<a name="dli_08_0193"></a><a name="dli_08_0193"></a>
|
|
|
|
<h1 class="topictitle1">Creating a DLI Table and Associating It with DWS</h1>
|
|
<div id="body1578361312449"><div class="section" id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section540354616207"><h4 class="sectiontitle">Function</h4><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p13312652114">This statement is used to create a DLI table and associate it with an existing DWS table.</p>
|
|
<div class="note" id="dli_08_0193__note443995614391"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0193__dli_08_0119_p964618104214">In Spark cross-source development scenarios, there is a risk of password leakage if datasource authentication information is directly configured. You are advised to use the datasource authentication provided by DLI.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="dli_08_0193__section1886921516407"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_08_0193__p1976716256406">Before creating a DLI table and associating it with DWS, you need to create a datasource connection. For details about operations on the management console, see </p>
|
|
</div>
|
|
<div class="section" id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section20884135882017"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0193__en-us_topic_0142907229_screen11333202133014"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="p">[</span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="p">]</span><span class="w"> </span><span class="k">TABLE_NAME</span>
|
|
<span class="w"> </span><span class="k">USING</span><span class="w"> </span><span class="n">JDBC</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="s1">'url'</span><span class="o">=</span><span class="s1">'xx'</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'dbtable'</span><span class="o">=</span><span class="s1">'db_name_in_DWS.table_name_in_DWS'</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'passwdauth'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xxx'</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'encryption'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'true'</span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section784510261216"><h4 class="sectiontitle">Keywords</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_t2fa8597e0dde4884bd6412ac207ebea8" frame="border" border="1" rules="all"><caption><b>Table 1 </b>CREATE TABLE keywords </caption><thead align="left"><tr id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_r16c75fd8fdbf41c79f9aee82b032351c"><th align="left" class="cellrowborder" valign="top" width="16.3%" id="mcps1.3.4.2.2.3.1.1"><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_a9dd69379a3f54904a9c510fb817659ad"><strong id="dli_08_0193__b12779111184">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="83.7%" id="mcps1.3.4.2.2.3.1.2"><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_a7ede0d62539542baa7e367cf902bf2b2"><strong id="dli_08_0193__en-us_topic_0093946771_en-us_topic_0053447306_en-us_topic_0039551470_b6335010717346">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row9789172211455"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p1783227111576">url</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_p13413134214396">Before obtaining the DWS IP address, you need to create a datasource connection first.. </p>
|
|
<p id="dli_08_0193__en-us_topic_0142907229_p493150101918">If you have created an enhanced datasource connection, you can use the <strong id="dli_08_0193__b75481610101911">JDBC Connection String (intranet)</strong> provided by DWS or the intranet address and port number to access DWS. The format is <strong id="dli_08_0193__b355041031913"><em id="dli_08_0193__i65481910151913">protocol header</em>: //<em id="dli_08_0193__i11548171014195">Internal IP address</em>:<em id="dli_08_0193__i1254921071918">Internal network port</em>/<em id="dli_08_0193__i1454911019196">Database name</em></strong>, for example: <strong id="dli_08_0193__b255061011920">jdbc:postgresql://192.168.0.77:8000/postgres</strong>.</p>
|
|
<div class="note" id="dli_08_0193__en-us_topic_0142907229_note18934713115119"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_0193__en-us_topic_0142907229_p278332612514">The DWS IP address is in the following format: <strong id="dli_08_0193__b18146141610196"><em id="dli_08_0193__i214411651916">protocol header</em>://<em id="dli_08_0193__i414514168191">IP address</em>:<em id="dli_08_0193__i17145116191917">port number</em>/<em id="dli_08_0193__i714551601914">database name</em></strong></p>
|
|
<p id="dli_08_0193__en-us_topic_0142907229_p20979153555113">The following is an example:</p>
|
|
<p id="dli_08_0193__en-us_topic_0142907229_p32993235218">jdbc:postgresql://to-dws-1174405119-ihlUr78j.datasource.com:8000/postgres</p>
|
|
<p id="dli_08_0193__en-us_topic_0142907229_p179121425135118">If you want to connect to a database created in DWS, change <strong id="dli_08_0193__b34420194193">postgres</strong> to the corresponding database name in this connection.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row166137253465"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p584237211576">dbtable</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p5902520811576">Specifies the name or <strong id="dli_08_0193__b8365252142419">Schema name.Table name</strong> of the table that is associated with the DWS. For example: <strong id="dli_08_0193__b19314716152512">public.table_name</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row193609262164"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p247311352164">user</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p570649132164">(Discarded) DWS username.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row469853952167"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p477206762167">password</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p401695862167">User password of the DWS cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__row16695305209"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__p367053022018">passwdauth</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0151059555_p562616372455">Datasource password authentication name. For details about how to create datasource authentication, see Datasource Authentication in the <em id="dli_08_0193__i40796864092254">Data Lake Insight User Guide</em>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__row37391835142011"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__p1073983512205">encryption</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__p13739113518204">Set this parameter to <span class="parmvalue" id="dli_08_0193__parmvalue6574145514110"><b>true</b></span> when datasource password authentication is used.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_row85723281419"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_p115736213142">partitionColumn</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_p657314251414">This parameter is used to set the numeric field used concurrently when data is read.</p>
|
|
<div class="note" id="dli_08_0193__en-us_topic_0142907229_note84102206211"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0193__ul103646542512"><li id="dli_08_0193__li1436419519251">The <span class="parmname" id="dli_08_0193__parmname129675405508"><b>partitionColumn</b></span>, <span class="parmname" id="dli_08_0193__parmname7260844115018"><b>lowerBound</b></span>, <span class="parmname" id="dli_08_0193__parmname1060605110508"><b>upperBound</b></span>, and <span class="parmname" id="dli_08_0193__parmname15958155545013"><b>numPartitions</b></span> parameters must be set at the same time.</li><li id="dli_08_0193__li1336731812252">To improve the concurrent read performance, you are advised to use auto-increment columns.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_row1377317518146"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_p17774145171412">lowerBound</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_p8774175121411">Minimum value of a column specified by <strong id="dli_08_0193__b267717714227">partitionColumn</strong>. The value is contained in the returned result.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_row12626119161414"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_p106263918145">upperBound</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_p09634592231">Maximum value of a column specified by <strong id="dli_08_0193__b16810119162217">partitionColumn</strong>. The value is not contained in the returned result.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_row3550181310143"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_p175509134149">numPartitions</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_p135501013191415">Number of concurrent read operations.</p>
|
|
<div class="note" id="dli_08_0193__en-us_topic_0142907229_note1178124011241"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_0193__en-us_topic_0142907229_p3178240102415">When data is read, the number of concurrent operations are evenly allocated to each task according to the <span class="parmname" id="dli_08_0193__parmname1085316198534"><b>lowerBound</b></span> and <span class="parmname" id="dli_08_0193__parmname88542199539"><b>upperBound</b></span> to obtain data. The following is an example:</p>
|
|
<pre class="screen" id="dli_08_0193__screen694310962920">'partitionColumn'='id',
|
|
'lowerBound'='0',
|
|
'upperBound'='100',
|
|
'numPartitions'='2'</pre>
|
|
<p id="dli_08_0193__en-us_topic_0142907229_p1361474310301">Two concurrent tasks are started in DLI. The execution ID of one task is greater than or equal to <strong id="dli_08_0193__b544691812221">0</strong> and the ID is less than <strong id="dli_08_0193__b12447191812224">50</strong>, and the execution ID of the other task is greater than or equal to <strong id="dli_08_0193__b15447161812221">50</strong> and the ID is less than <strong id="dli_08_0193__b144481518202215">100</strong>.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_row9294131610146"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_p10294816131410">fetchsize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_p20295111641419">Number of data records obtained in each batch during data reading. The default value is <strong id="dli_08_0193__b1015312322213">1000</strong>. If this parameter is set to a large value, the performance is good but more memory is occupied. If this parameter is set to a large value, memory overflow may occur.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_row414443911917"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_p714473981918">batchsize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_p14144123961915">Number of data records written in each batch. The default value is <strong id="dli_08_0193__b1341122718221">1000</strong>. If this parameter is set to a large value, the performance is good but more memory is occupied. If this parameter is set to a large value, memory overflow may occur.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_row8392842101911"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_p17393542141920">truncate</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_p339304231915">Indicates whether to clear the table without deleting the original table when <strong id="dli_08_0193__b7109173112221">overwrite</strong> is executed. The options are as follows:</p>
|
|
<ul id="dli_08_0193__en-us_topic_0142907229_ul10596112382616"><li id="dli_08_0193__en-us_topic_0142907229_li7597102311264">true</li><li id="dli_08_0193__en-us_topic_0142907229_li1597123202615">false</li></ul>
|
|
<p id="dli_08_0193__en-us_topic_0142907229_p1212110107349">The default value is <span class="parmvalue" id="dli_08_0193__parmvalue56513862218"><b>false</b></span>, indicating that the original table is deleted and then a new table is created when the <strong id="dli_08_0193__b6651238142212">overwrite</strong> operation is performed.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0193__en-us_topic_0142907229_row8336958208"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0193__en-us_topic_0142907229_p73369516203">isolationLevel</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.7%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0193__en-us_topic_0142907229_p113361357206">Transaction isolation level. The options are as follows:</p>
|
|
<ul id="dli_08_0193__en-us_topic_0142907229_ul131065575264"><li id="dli_08_0193__en-us_topic_0142907229_li11065579264">NONE</li><li id="dli_08_0193__en-us_topic_0142907229_li4106155714264">READ_UNCOMMITTED</li><li id="dli_08_0193__en-us_topic_0142907229_li51066579268">READ_COMMITTED</li><li id="dli_08_0193__en-us_topic_0142907229_li1710619571263">REPEATABLE_READ</li><li id="dli_08_0193__en-us_topic_0142907229_li14107145772618">SERIALIZABLE</li></ul>
|
|
<p id="dli_08_0193__en-us_topic_0142907229_p1895985011353">The default value is <span class="parmvalue" id="dli_08_0193__parmvalue89071343192215"><b>READ_UNCOMMITTED</b></span>.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section1856617202225"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0193__en-us_topic_0142907229_p17748165312113">When creating a table associated with DWS, you do not need to specify the <strong id="dli_08_0193__b710911484229">Schema</strong> of the associated table. DLI automatically obtains the schema of the table in the <strong id="dli_08_0193__b4954164919226">dbtable</strong> parameter of DWS.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0193__en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section9892337112213"><h4 class="sectiontitle">Example</h4><div class="codecoloring" codetype="Sql" id="dli_08_0193__en-us_topic_0142907229_screen15211451306"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="n">dli_to_dws</span>
|
|
<span class="w"> </span><span class="k">USING</span><span class="w"> </span><span class="n">JDBC</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="s1">'url'</span><span class="o">=</span><span class="s1">'jdbc:postgresql://to-dws-1174405119-ih1Ur78j.datasource.com:8000/postgres'</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'dbtable'</span><span class="o">=</span><span class="s1">'test_dws'</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'passwdauth'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xxx'</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'encryption'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'true'</span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0192.html">Creating a Datasource Connection with a DWS table</a></div>
|
|
</div>
|
|
</div>
|
|
|