vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0100.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

234 lines
39 KiB
HTML
Raw Blame History

<a name="dli_08_0100"></a><a name="dli_08_0100"></a>
<h1 class="topictitle1">Importing Data to the Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_s86887e0a86644c22b61a4c2ccc84025e"><h4 class="sectiontitle">Function</h4><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_a9755a68b9d0d47668ac554151d03e0ce">The <strong id="dli_08_0100__b1625414431115">LOAD DATA</strong> function can be used to import data in <strong id="dli_08_0100__b2265164315110">CSV</strong>, <strong id="dli_08_0100__b102661643611">Parquet</strong>, <strong id="dli_08_0100__b1826718431416">ORC</strong>, <strong id="dli_08_0100__b13267124312110">JSON</strong>, and <strong id="dli_08_0100__b157545511519">Avro</strong> formats. The data is converted into the <strong id="dli_08_0100__b826915431817">Parquet</strong> data format for storage.</p>
</div>
<div class="section" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_sd6725bae42f0429e8bc6dfa2e92b9664"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0100__en-us_topic_0114776194_screen1294316133510"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">LOAD</span><span class="w"> </span><span class="k">DATA</span><span class="w"> </span><span class="n">INPATH</span><span class="w"> </span><span class="s1">'folder_path'</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="p">[</span><span class="n">db_name</span><span class="p">.]</span><span class="k">table_name</span>
<span class="w"> </span><span class="k">OPTIONS</span><span class="p">(</span><span class="n">property_name</span><span class="o">=</span><span class="n">property_value</span><span class="p">,</span><span class="w"> </span><span class="p">...);</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0100__en-us_topic_0114776175_en-us_topic_0093946864_s0c845eb1e41148d79029d4f419b4fc4f"><h4 class="sectiontitle">Keywords</h4><ul id="dli_08_0100__ul85891167229"><li id="dli_08_0100__li145908169227">INPATH: path of data to be imported</li><li id="dli_08_0100__li14590101622219">OPTIONS: list of properties</li></ul>
</div>
<div class="section" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_se9decc2fedb94b0ea80dc860c79cd3f1"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_td8eb9ff5337945feb0b707d81a4acc90" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_ra973c320b2524745ab259dca3a46809f"><th align="left" class="cellrowborder" valign="top" width="25.2%" id="mcps1.3.4.2.2.3.1.1"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_a14c009dd6ca34d2caf68bc9c24fcf82b">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="74.8%" id="mcps1.3.4.2.2.3.1.2"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_abff9a2e7c9614f08973e2b8afb9d7abe">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_r1a16c5faa82047138ab4fae99da6b80a"><td class="cellrowborder" valign="top" width="25.2%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_aa30066b85f034fd0b4f9e4edfde93b43">folder_path</p>
</td>
<td class="cellrowborder" valign="top" width="74.8%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_aff3839c9a643497ab38dc650affa62de">OBS path of the file or folder used for storing the raw data.</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_re196701bd81944f3b77ddae2d89f4878"><td class="cellrowborder" valign="top" width="25.2%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_a3f6ef3ea3f764b27a3832c7fbf84654f">db_name</p>
</td>
<td class="cellrowborder" valign="top" width="74.8%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_ae3c5c3ed67574e0482fa3c69c38ce5f8">Enter the database name. If this parameter is not specified, the current database is used.</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_rf72baf2241cd40a7809184d7ff2e27de"><td class="cellrowborder" valign="top" width="25.2%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_afb004d9cdb34479c8b90dbf1ae8a60f7">table_name</p>
</td>
<td class="cellrowborder" valign="top" width="74.8%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p22002191134643">Name of the DLI table to which data is to be imported.</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p10428348174421">The following configuration options can be used during data import:</p>
<ul id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_ul20851259174428"><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li17583114016326">DATA_TYPE: specifies the type of data to be imported. Currently, <strong id="dli_08_0100__b197725215211">CSV</strong>, <strong id="dli_08_0100__b1477311220213">Parquet</strong>, <strong id="dli_08_0100__b1277412220213">ORC</strong>, <strong id="dli_08_0100__b117751328215">JSON</strong>, and <strong id="dli_08_0100__b176215136216">Avro</strong> are supported. The default value is <span class="parmvalue" id="dli_08_0100__parmvalue37761221725"><b>CSV</b></span>.<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1211114713417">The configuration item is <strong id="dli_08_0100__b6992114216147">OPTIONS</strong> ('DATA_TYPE' = 'CSV').</p>
<p id="dli_08_0100__en-us_topic_0114776194_p10823159101914">When importing a <strong id="dli_08_0100__b971775901420">CSV</strong> file or a <strong id="dli_08_0100__b58241716195014">JSON</strong> file, you can select one of the following modes:</p>
<ul id="dli_08_0100__en-us_topic_0114776194_ul0657104812312"><li id="dli_08_0100__en-us_topic_0114776194_li189261446102316"><strong id="dli_08_0100__b28004130158">PERMISSIVE</strong>: When the <strong id="dli_08_0100__b18824198151">PERMISSIVE</strong> mode is selected, the data of a column is set to <strong id="dli_08_0100__b46062461718">null</strong> if its data type does not match that of the target table column.</li><li id="dli_08_0100__en-us_topic_0114776194_li119531546102318"><strong id="dli_08_0100__b15182121191818">DROPMALFORMED</strong>: When the <strong id="dli_08_0100__b02928131817">DROPMALFORMED</strong> mode is selected, the data of a column s not imported if its data type does not match that of the target table column.</li><li id="dli_08_0100__en-us_topic_0114776194_li1095454602317"><strong id="dli_08_0100__b1789101312620">FAILFAST</strong>: When the <strong id="dli_08_0100__b1311581952611">FAILFAST</strong> mode is selected, exceptions might occur and the import may fail if a column type does not match.</li></ul>
<p id="dli_08_0100__en-us_topic_0114776194_p011155618249">You can set the mode by adding <strong id="dli_08_0100__b1544610189304">OPTIONS ('MODE' = 'PERMISSIVE')</strong> to the <strong id="dli_08_0100__b1740632317309">OPTIONS</strong> parameter.</p>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li1283316012322"><strong id="dli_08_0100__b273320113211">DELIMITER</strong>: You can specify a separator in the import statement. The default value is <span class="parmvalue" id="dli_08_0100__en-us_topic_0093946741_parmvalue3976890011539"><b>,</b></span>.<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p4571103843518">The configuration item is <strong id="dli_08_0100__en-us_topic_0093946741_b842352706124349">OPTIONS('DELIMITER'=',')</strong>.</p>
<p id="dli_08_0100__p15161732113516">For CSV data, the following delimiters are supported:</p>
<ul id="dli_08_0100__ul55224545357"><li id="dli_08_0100__li226614819353">Tab character, for example, <strong id="dli_08_0100__b6379164615506">'DELIMITER'='\t'</strong>.</li><li id="dli_08_0100__li6833163410550">Any binary character, for example, <strong id="dli_08_0100__b13277205775014">'DELIMITER'='\u0001(^A)'</strong>.</li><li id="dli_08_0100__li15171039125620">Single quotation mark ('). A single quotation mark must be enclosed in double quotation marks (" "). For example, <strong id="dli_08_0100__b196211613195110">'DELIMITER'= "'"</strong>.</li><li id="dli_08_0100__li725165103713"><strong id="dli_08_0100__b1526342895119">\001(^A)</strong> and <strong id="dli_08_0100__b8425173165116">\017(^Q)</strong> are also supported, for example, <strong id="dli_08_0100__b104993817512">'DELIMITER'='\001(^A)'</strong> and <strong id="dli_08_0100__b857517410514">'DELIMITER'='\017(^Q)'</strong>.</li></ul>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li177367331481"><strong id="dli_08_0100__b15364116113214">QUOTECHAR</strong>: You can specify quotation marks in the import statement. The default value is double quotation marks (<strong id="dli_08_0100__en-us_topic_0093946741_b1517187940124617">"</strong>).<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p287017571483">The configuration item is <strong id="dli_08_0100__b1507621517">OPTIONS('QUOTECHAR'='"')</strong>.</p>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li1185594314751"><strong id="dli_08_0100__b28581029163217">COMMENTCHAR</strong>: You can specify the comment character in the import statement. During the import operation, if a comment character is at the beginning of a row, the row is considered as a comment and will not be imported. The default value is a pound key (#).<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p3271113314753">The configuration item is <i><span class="varname" id="dli_08_0100__en-us_topic_0093946741_varname787033293193637">OPTIONS('COMMENTCHAR'='#')</span></i>.</p>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li3720302214743"><strong id="dli_08_0100__en-us_topic_0093946741_b84235270612499">HEADER</strong>: Indicates whether the source file contains a header. Possible values can be <span class="parmvalue" id="dli_08_0100__en-us_topic_0093946741_parmvalue6525743214743"><b>true</b></span> and <span class="parmvalue" id="dli_08_0100__en-us_topic_0093946741_parmvalue5044598014743"><b>false</b></span>. <span class="parmvalue" id="dli_08_0100__en-us_topic_0093946741_parmvalue5136063814743"><b>true</b></span> indicates that the source file contains a header, and <span class="parmvalue" id="dli_08_0100__en-us_topic_0093946741_parmvalue5959255914743"><b>false</b></span> indicates that the source file does not contain a header. The default value is <span class="parmvalue" id="dli_08_0100__en-us_topic_0093946741_parmvalue1139571418125019"><b>false</b></span>. If no header exists, specify the <strong id="dli_08_0100__en-us_topic_0093946741_b842352706125033">FILEHEADER</strong> parameter in the <strong id="dli_08_0100__en-us_topic_0093946741_b842352706125036">LOAD DATA</strong> statement to add a header.<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p391657814745">The configuration item is <i><span class="varname" id="dli_08_0100__en-us_topic_0093946741_varname1980173326125046">OPTIONS('HEADER'='true')</span></i>.</p>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li44070917174428"><strong id="dli_08_0100__en-us_topic_0093946741_b84235270612510">FILEHEADER</strong>: If the source file does not contain any header, add a header to the <strong id="dli_08_0100__en-us_topic_0093946741_b84235270612518">LOAD DATA</strong> statement.<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p61093938174428"><i><span class="varname" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_varname12974535174428">OPTIONS('FILEHEADER'='column1,column2')</span></i></p>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li2775514736"><strong id="dli_08_0100__en-us_topic_0093946741_b842352706125147">ESCAPECHAR</strong>: Is used to perform strict verification of the escape character on CSV files. The default value is a slash (<span class="parmvalue" id="dli_08_0100__parmvalue7400447131515"><b>\\</b></span>).<div class="p" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p6147484714738">The configuration item is OPTIONS. (ESCAPECHAR?=?\\?)<div class="note" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_note27940099174428"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p50134301174428">Enter <strong id="dli_08_0100__en-us_topic_0093946741_b842352706125233">ESCAPECHAR</strong> in the CSV data. <strong id="dli_08_0100__en-us_topic_0093946741_b842352706125236">ESCAPECHAR</strong> must be enclosed in double quotation marks (" "). For example, "a\b".</p>
</div></div>
</div>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li48555528174428"><strong id="dli_08_0100__en-us_topic_0093946741_b842352706125258">MAXCOLUMNS</strong>: This parameter is optional and specifies the maximum number of columns parsed by a CSV parser in a line.<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p34346574174428">The configuration item is <i><span class="varname" id="dli_08_0100__en-us_topic_0093946741_varname18805800112536">OPTIONS('MAXCOLUMNS'='400')</span></i>.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_table30609075174428" frame="border" border="1" rules="all"><caption><b>Table 2 </b>MAXCOLUMNS</caption><thead align="left"><tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row43224859174428"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.4.8.3.2.4.1.1"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p11552667174428"><strong id="dli_08_0100__en-us_topic_0093946741_b842352706125316">Name of the Optional Parameter</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.4.8.3.2.4.1.2"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p33286688174428"><strong id="dli_08_0100__en-us_topic_0093946741_b31144741174428">Default Value</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.4.8.3.2.4.1.3"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p39696127174428"><strong id="dli_08_0100__en-us_topic_0093946741_b842352706125326">Maximum Value</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row14556804174428"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.4.8.3.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p38250446174428">MAXCOLUMNS</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.4.8.3.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p11278404174428">2000</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.4.8.3.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p41135568174428">20000</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_note49471410174428"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p635289181499">After the value of <strong id="dli_08_0100__b1876913717346">MAXCOLUMNS Option</strong> is set, data import will require the memory of <strong id="dli_08_0100__b1382212618346">executor</strong>. As a result, data may fail to be imported due to insufficient <strong id="dli_08_0100__b19602105817341">executor</strong> memory.</p>
</div></div>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li415577174428"><strong id="dli_08_0100__en-us_topic_0093946741_b84235270618242">DATEFORMAT</strong>: Specifies the date format of a column.<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p33661782174428"><i><span class="varname" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_varname34520588174428">OPTIONS('DATEFORMAT'='dateFormat')</span></i></p>
<div class="note" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_note42249840174428"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_ul11139195324010"><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li1513911537402">The default value is yyyy-MM-dd.</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li91399538408">The date format is specified by the date mode string of <strong id="dli_08_0100__b21471922153713">Java</strong>. For the Java strings describing date and time pattern, characters <strong id="dli_08_0100__b1690294819124">A</strong> to <strong id="dli_08_0100__b5903184841213">Z</strong> and <strong id="dli_08_0100__b159043484123">a</strong> to <strong id="dli_08_0100__b790494812126">z</strong> without single quotation marks (') are interpreted as pattern characters , which are used to represent date or time string elements. If the pattern character is quoted by single quotation marks ('), text matching rather than parsing is performed. For the definition of pattern characters in Java, see <a href="#dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_table489265920252">Table 3</a>.</li></ul>
</div></div>
<div class="tablenoborder"><a name="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_table489265920252"></a><a name="en-us_topic_0114776194_en-us_topic_0093946741_table489265920252"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_table489265920252" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Definition of characters involved in the date and time patterns</caption><thead align="left"><tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row118943591258"><th align="left" class="cellrowborder" valign="top" width="14.291429142914291%" id="mcps1.3.4.4.9.4.2.4.1.1"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p18894185913259"><strong id="dli_08_0100__b84235270620307">Character</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25.61256125612561%" id="mcps1.3.4.4.9.4.2.4.1.2"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p5894359122519"><strong id="dli_08_0100__b842352706203013">Date or Time Element</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="60.0960096009601%" id="mcps1.3.4.4.9.4.2.4.1.3"><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1089445912519"><strong id="dli_08_0100__b75861933181615">Example</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row6894185912519"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p2089435919255">G</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p089405913251">Epoch ID</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p78941594258">AD</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row8894135911252"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p17894115915256">y</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1089435942516">Year</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p17894759172511">1996; 96</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row7802105310312"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1080217532315">M</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p19802125303118">Month</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p158021153133119">July; Jul; 07</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row1031413716327"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p133141037193213">w</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p6314337133214">Number of the week in a year</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p6314173712328">27 (the twenty-seventh week of the year)</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row926804183218"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p62686417327">W</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p427094112322">Number of the week in a month</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p9270941153213">2 (the second week of the month)</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row332544593218"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p12325184518327">D</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1432513457325">Number of the day in a year</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1632511452328">189 (the 189th day of the year)</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row17370115011358"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p14370350143517">d</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p63701150193513">Number of the day in a month</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p153721850193519">10 (the tenth day of the month)</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row636216557361"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p13621155183613">u</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p163621755183618">Number of the day in a week</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1136218553361">1 (Monday), ..., 7 (Sunday)</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row165104724711"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p851184724718">a</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p10511047114719">am/pm flag</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1751124714478">pm (12:00-24:00)</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row1525822744814"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p5258127204815">H</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p18258192794818">Hour time (0-23)</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1625882713487">2</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row727863518492"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p18278113518493">h</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p11278163511493">Hour time (1-12)</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p5278135174910">12</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row648417065119"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p24846085112">m</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p04841801511">Number of minutes</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p154841209516">30</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row1299152119511"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p599110215511">s</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p599112195119">Number of seconds</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p89301151125110">55</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row142275715311"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p822747155315">S</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p19227107115319">Number of milliseconds</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p622777205313">978</p>
</td>
</tr>
<tr id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_row11842184375419"><td class="cellrowborder" valign="top" width="14.291429142914291%" headers="mcps1.3.4.4.9.4.2.4.1.1 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p20844243145413">z</p>
</td>
<td class="cellrowborder" valign="top" width="25.61256125612561%" headers="mcps1.3.4.4.9.4.2.4.1.2 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p384454315420">Time zone</p>
</td>
<td class="cellrowborder" valign="top" width="60.0960096009601%" headers="mcps1.3.4.4.9.4.2.4.1.3 "><p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p1684464395413">Pacific Standard Time; PST; GMT-08:00</p>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li64273922174428"><strong id="dli_08_0100__en-us_topic_0093946741_b84235270618317">TIMESTAMPFORMAT</strong>: Specifies the timestamp format of a column.<p id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_p38805218174428"><i><span class="varname" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_varname13702648174428">OPTIONS('TIMESTAMPFORMAT'='timestampFormat')</span></i></p>
<div class="note" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_note56214971174428"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_ul13258146114112"><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li62584469419">Default value: yyyy-MM-dd HH:mm:ss.</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li1925815464416">The timestamp format is specified by the Java time pattern string. For details, see <a href="#dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_table489265920252">Table 3</a>.</li></ul>
</div></div>
</li></ul>
</div>
<ul id="dli_08_0100__en-us_topic_0114776194_ul333822315534"><li id="dli_08_0100__en-us_topic_0114776194_li318875095617"><strong id="dli_08_0100__b1528784093918">Mode</strong>: Specifies the processing mode of error records while importing. The options are as follows: <strong id="dli_08_0100__b1484042184414">PERMISSIVE</strong>, <strong id="dli_08_0100__b162141317442">DROPMALFORMED</strong>, and <strong id="dli_08_0100__b366611714417">FAILFAST</strong>.<p id="dli_08_0100__en-us_topic_0114776194_p17275175119562"><i><span class="varname" id="dli_08_0100__en-us_topic_0114776194_varname827895125616">OPTIONS('MODE'='permissive')</span></i></p>
<div class="note" id="dli_08_0100__en-us_topic_0114776194_note2067453817591"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_08_0100__en-us_topic_0114776194_ul262816211119"><li id="dli_08_0100__en-us_topic_0114776194_li85111231815"><strong id="dli_08_0100__b1566519574542">PERMISSIVE (default)</strong>: Parse bad records as much as possible. If a field cannot be converted, the entire row is null.</li><li id="dli_08_0100__en-us_topic_0114776194_li4631185118219"><strong id="dli_08_0100__b1785217512463">DROPMALFORMED</strong>: Ignore the <strong id="dli_08_0100__b14457227144718">bad records</strong> that cannot be parsed.</li><li id="dli_08_0100__en-us_topic_0114776194_li126212441035"><strong id="dli_08_0100__b48025485472">FAILFAST</strong>: If a record cannot be parsed, an exception is thrown and the job fails.</li></ul>
</div></div>
</li><li id="dli_08_0100__en-us_topic_0114776194_li1784019326537"><strong id="dli_08_0100__b126212418562">BADRECORDSPATH</strong>: Specifies the directory for storing error records during the import.<p id="dli_08_0100__en-us_topic_0114776194_p98431732175312"><i><span class="varname" id="dli_08_0100__en-us_topic_0114776194_varname18844632125311">OPTIONS('BADRECORDSPATH'='obs://bucket/path')</span></i></p>
<div class="note" id="dli_08_0100__en-us_topic_0114776194_note6847532185310"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0100__en-us_topic_0114776194_p6786013175519">It is recommended that this option be used together with the <strong id="dli_08_0100__b10638336135712">DROPMALFORMED</strong> pattern to import the records that can be successfully converted into the target table and store the records that fail to be converted to the specified error record storage directory.</p>
</div></div>
</li></ul>
<div class="section" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_s8f96768332ec4eefa9a34f76c15fdea0"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_ul131885284418"><li id="dli_08_0100__li13384853144911">When importing or creating an OBS table, you must specify a folder as the directory. If a file is specified, data import may be failed.</li><li id="dli_08_0100__en-us_topic_0114776194_li43426583119">Only the raw data stored in the OBS path can be imported.</li><li id="dli_08_0100__en-us_topic_0114776194_li17344358610">You are advised not to concurrently import data in to a table. If you concurrently import data into a table, there is a possibility that conflicts occur, leading to failed data import.</li><li id="dli_08_0100__en-us_topic_0114776194_li1834516581413">Only one path can be specified during data import. The path cannot contain commas (,).</li><li id="dli_08_0100__en-us_topic_0114776194_li18347205811114">If a folder and a file with the same name exist in the OBS bucket directory, the data is preferentially to be imported directed to the file rather than the folder.</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li41889289416">When importing data of the PARQUET, ORC, or JSON format, you must specify <i><span class="varname" id="dli_08_0100__varname10670182861319">DATA_TYPE</span></i>. Otherwise, the data is parsed into the default format <span class="parmvalue" id="dli_08_0100__parmvalue3671328131319"><b>CSV</b></span>. In this case, the format of the imported data is incorrect.</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li10904123614715">If the data to be imported is in the CSV or JSON format and contains the date and columns, you need to specify <i><span class="varname" id="dli_08_0100__varname717303318138">DATEFORMAT</span></i> and <i><span class="varname" id="dli_08_0100__varname10174183314136">TIMESTAMPFORMAT</span></i>. Otherwise, the data will be parsed into the default date and timestamp formats.</li></ul>
</div>
<div class="section" id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_s89feeae0ecc24152b81d87b9ab82de6e"><h4 class="sectiontitle">Example</h4><div class="note" id="dli_08_0100__note1149934611312"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0100__p161225010319">Before importing data, you must create a table. For details, see <a href="dli_08_0223.html">Creating an OBS Table</a> or <a href="dli_08_0224.html">Creating a DLI Table</a>.</p>
</div></div>
<ul id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_ul169541033171618"><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li16954533181613">To import a CSV file to a DLI table named <span class="parmname" id="dli_08_0100__parmname4773156165213"><b>t</b></span>, run the following statement:<div class="codecoloring" codetype="Sql" id="dli_08_0100__en-us_topic_0114776194_screen29241811616"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">LOAD</span><span class="w"> </span><span class="k">DATA</span><span class="w"> </span><span class="n">INPATH</span><span class="w"> </span><span class="s1">'obs://dli/data.csv'</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">t</span>
<span class="w"> </span><span class="k">OPTIONS</span><span class="p">(</span><span class="s1">'DELIMITER'</span><span class="o">=</span><span class="s1">','</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="s1">'QUOTECHAR'</span><span class="o">=</span><span class="s1">'&quot;'</span><span class="p">,</span><span class="s1">'COMMENTCHAR'</span><span class="o">=</span><span class="s1">'#'</span><span class="p">,</span><span class="s1">'HEADER'</span><span class="o">=</span><span class="s1">'false'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_08_0100__en-us_topic_0114776194_en-us_topic_0093946741_li8270135219163">To import a JSON file to a DLI table named <span class="parmname" id="dli_08_0100__parmname456265865117"><b>jsontb</b></span>, run the following statement:<div class="codecoloring" codetype="Sql" id="dli_08_0100__screen32607464581"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">LOAD</span><span class="w"> </span><span class="k">DATA</span><span class="w"> </span><span class="n">INPATH</span><span class="w"> </span><span class="s1">'obs://dli/alltype.json'</span><span class="w"> </span><span class="k">into</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">jsontb</span>
<span class="w"> </span><span class="k">OPTIONS</span><span class="p">(</span><span class="s1">'DATA_TYPE'</span><span class="o">=</span><span class="s1">'json'</span><span class="p">,</span><span class="s1">'DATEFORMAT'</span><span class="o">=</span><span class="s1">'yyyy/MM/dd'</span><span class="p">,</span><span class="s1">'TIMESTAMPFORMAT'</span><span class="o">=</span><span class="s1">'yyyy/MM/dd HH:mm:ss'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0221.html">Spark SQL Syntax Reference</a></div>
</div>
</div>
View Git Blame Copy Permalink