vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide/mrs_01_1438.html
Yang, Tong 6182f91ba8 MRS component operation guide_normal 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-09 14:55:21 +00:00

209 lines
28 KiB
HTML
Raw Blame History

<a name="mrs_01_1438"></a><a name="mrs_01_1438"></a>
<h1 class="topictitle1">LOAD DATA</h1>
<div id="body1595920212612"><div class="section" id="mrs_01_1438__sbcf3c5dfd5e64e3cbb83be9da9b7f66a"><h4 class="sectiontitle">Function</h4><p id="mrs_01_1438__a54d8e7a919cf4c899f49661ee944d6f0">This command is used to load user data of a particular type, so that CarbonData can provide good query performance.</p>
<div class="note" id="mrs_01_1438__na1d0a7be07f34416a4b60aa050aaad3c"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1438__p7805257161219">Only the raw data on HDFS can be loaded.</p>
</div></div>
</div>
<div class="section" id="mrs_01_1438__sf0c281125d99406bbd9c957a4eb70d58"><h4 class="sectiontitle">Syntax</h4><p id="mrs_01_1438__a0976c2c6528446199df1cb1d922ed609"><b><span class="cmdname" id="mrs_01_1438__cmdname17276125131614">LOAD DATA</span></b> <i><span class="varname" id="mrs_01_1438__v74f09271355e409bb7db4957b7c6e221">INPATH 'folder_path' INTO TABLE [db_name.]table_name OPTIONS(property_name=property_value, ...);</span></i></p>
</div>
<div class="section" id="mrs_01_1438__sb6649b1ad1954ca7b2f4f50ae35c8f08"><h4 class="sectiontitle">Parameter Description</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1438__t029f367c79f24ddeb167ed9c75f3201f" frame="border" border="1" rules="all"><caption><b>Table 1 </b>LOAD DATA parameters</caption><thead align="left"><tr id="mrs_01_1438__r221d06955aea4ec49666db5414d2fe20"><th align="left" class="cellrowborder" valign="top" width="25.2%" id="mcps1.3.3.2.2.3.1.1"><p id="mrs_01_1438__a97060f3d522c4cbd99809e842429d486"><strong id="mrs_01_1438__b142563105714">Parameter</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="74.8%" id="mcps1.3.3.2.2.3.1.2"><p id="mrs_01_1438__a95634856a4c944a48a805d6f3d693be0"><strong id="mrs_01_1438__b1317210218574">Description</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1438__r36726998738c4cb48b34d3cebc2f6581"><td class="cellrowborder" valign="top" width="25.2%" headers="mcps1.3.3.2.2.3.1.1 "><p id="mrs_01_1438__a0e157eac0bef487497105e3143e7d142">folder_path</p>
</td>
<td class="cellrowborder" valign="top" width="74.8%" headers="mcps1.3.3.2.2.3.1.2 "><p id="mrs_01_1438__a6389c1c42b3b4dcdaebdf8dd5e585d56">Path of the file or folder used for storing the raw CSV data.</p>
</td>
</tr>
<tr id="mrs_01_1438__rb2200cd0639149c68035fc51c8969231"><td class="cellrowborder" valign="top" width="25.2%" headers="mcps1.3.3.2.2.3.1.1 "><p id="mrs_01_1438__a19922fa0349d4570b371ea604ea82212">db_name</p>
</td>
<td class="cellrowborder" valign="top" width="74.8%" headers="mcps1.3.3.2.2.3.1.2 "><p id="mrs_01_1438__ae6b79564ef1543688ced3520dc1dc5ab">Database name. If this parameter is not specified, the current database is used.</p>
</td>
</tr>
<tr id="mrs_01_1438__rf0edeb6175da4a7eb8bc2c8eaa49d8cd"><td class="cellrowborder" valign="top" width="25.2%" headers="mcps1.3.3.2.2.3.1.1 "><p id="mrs_01_1438__a3c1ac51ba00f41f9abbb2093de0d80ff">table_name</p>
</td>
<td class="cellrowborder" valign="top" width="74.8%" headers="mcps1.3.3.2.2.3.1.2 "><p id="mrs_01_1438__aab65138225c749e69ad85cf4d190b22f">Name of a table in a database.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="mrs_01_1438__sd7720bec4e3a4c4f92ccaddcd133ab37"><h4 class="sectiontitle">Precautions</h4><p id="mrs_01_1438__afed07ecdbaa34bcc90f3d2d5b845be14">The following configuration items are involved during data loading:</p>
<ul id="mrs_01_1438__u2457a948e5394473b9588097c81d663f"><li id="mrs_01_1438__ld785869fa38447c6883f9374f9937c0a"><strong id="mrs_01_1438__b16289243205810">DELIMITER</strong>: Delimiters and quote characters provided in the load command. The default value is a comma (<strong id="mrs_01_1438__b18118145115583">,</strong>).<p id="mrs_01_1438__af463d4e147c145628e83f4f45f6592e6"><i><span class="varname" id="mrs_01_1438__v340dec3e84354b36b50cb057a1b4917e">OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"')</span></i></p>
<p id="mrs_01_1438__a10a5d569278c498cb12b2e4f2d9cc807">You can use <strong id="mrs_01_1438__b1659712579323">'DELIMITER'='\t'</strong> to separate CSV data using tabs.</p>
<p id="mrs_01_1438__ae2e8d46b87bc4a51ad46eb4d4ef8a91a">OPTIONS('DELIMITER'='\t')</p>
<p id="mrs_01_1438__ae4db0e40c32d49a08fcfe4a43d026e62">CarbonData also supports <strong id="mrs_01_1438__b181071852343">\001</strong> and <strong id="mrs_01_1438__b58641819342">\017</strong> as delimiters.</p>
<div class="note" id="mrs_01_1438__n8cab0868ed42489d805b7274769d70f2"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1438__ab4ec6d093b004466bd8b28c69157c564">When the delimiter of CSV data is a single quotation mark ('), the single quotation mark must be enclosed in double quotation marks (" "). For example, 'DELIMITER'= "'".</p>
</div></div>
</li><li id="mrs_01_1438__l41e56a2ee4d044f2988651eb0d1cb760"><strong id="mrs_01_1438__b1920017271342">QUOTECHAR</strong>: Delimiters and quote characters provided in the load command. The default value is double quotation marks (<strong id="mrs_01_1438__b939432915343">"</strong>).<p id="mrs_01_1438__a88b7486e1d274f64997bc6a08b908c29"><i><span class="varname" id="mrs_01_1438__vf670e396f78f46488b5358bcd563150d">OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"')</span></i></p>
</li><li id="mrs_01_1438__l75df21af6ecf46fba7057f3e45f9c59c"><strong id="mrs_01_1438__b164101537173417">COMMENTCHAR</strong>: Comment characters provided in the load command. During data loading, if there is a comment character at the beginning of a line, the line is regarded as a comment line and data in the line will not be loaded. The default value is a pound key (#).<p id="mrs_01_1438__a5e9d3271bbfb41179ef7d996c0488c52"><i><span class="varname" id="mrs_01_1438__v83ec23f1dd054061b083a7589cd588d7">OPTIONS('COMMENTCHAR'='#')</span></i></p>
</li><li id="mrs_01_1438__l2d5b1c55c8cf439d8174e7cca703ba06"><strong id="mrs_01_1438__b1685612440348">FILEHEADER</strong>: If the source file does not contain any header, add a header to the <strong id="mrs_01_1438__b148611844113415">LOAD DATA</strong> command.<p id="mrs_01_1438__aee2cf411d6fa451c94148d062b6d63f1"><i><span class="varname" id="mrs_01_1438__vc3df609d748f4cce8f8587bef3dbcc2e">OPTIONS('FILEHEADER'='column1,column2')</span></i></p>
</li><li id="mrs_01_1438__l91305ed8c9af49389e668e34d095db1d"><strong id="mrs_01_1438__b1177385011346">ESCAPECHAR</strong>: Is used to perform strict verification of the escape character on CSV files. The default value is backslash (<strong id="mrs_01_1438__b4736195603411">\</strong>).<p id="mrs_01_1438__a74ada0d1c39143bb88b87f1e3ff83729">OPTIONS('ESCAPECHAR'='\')</p>
<div class="note" id="mrs_01_1438__n13a1404a01ab4a3d95e468bfd5e4b66f"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1438__a7366672c52664ff8ad3e1b2969ef3222">Enter <strong id="mrs_01_1438__b41208033517">ESCAPECHAR</strong> in the CSV data. <strong id="mrs_01_1438__b71253017351">ESCAPECHAR</strong> must be enclosed in double quotation marks (" "). For example, "a\b".</p>
</div></div>
</li><li id="mrs_01_1438__lcf623574402c443e908646591898c2be"><a name="mrs_01_1438__lcf623574402c443e908646591898c2be"></a><a name="lcf623574402c443e908646591898c2be"></a>Bad records handling:<p id="mrs_01_1438__a036247a1cdc6431199b9472f70d43fe4"><a name="mrs_01_1438__lcf623574402c443e908646591898c2be"></a><a name="lcf623574402c443e908646591898c2be"></a>In order for the data processing application to provide benefits, certain data integration is required. In most cases, data quality problems are caused by data sources.</p>
<p id="mrs_01_1438__a0aa57ee1418448febc5842b87e867f58">Methods of handling bad records are as follows:</p>
<ul id="mrs_01_1438__ue49325eaebb84e4fa382191c116124f8"><li id="mrs_01_1438__l9bc44a395f3c4db69ef33c085d4f5df7">Load all of the data before dealing with the errors.</li><li id="mrs_01_1438__l0575e6a6753e4ed7a3159b1d22daf09a">Clean or delete bad records before loading data or stop the loading when bad records are found.</li></ul>
<p id="mrs_01_1438__a666d8da39ca440f5a23269fb3ba83e8c">There are many options for clearing source data during CarbonData data loading, as listed in <a href="#mrs_01_1438__t1d4d77614e2b4b92b0f334d52702013b">Table 2</a>.</p>
<div class="tablenoborder"><a name="mrs_01_1438__t1d4d77614e2b4b92b0f334d52702013b"></a><a name="t1d4d77614e2b4b92b0f334d52702013b"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1438__t1d4d77614e2b4b92b0f334d52702013b" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Bad Records Logger</caption><thead align="left"><tr id="mrs_01_1438__rac375bc3beb141db8304227817190f32"><th align="left" class="cellrowborder" valign="top" width="33.33%" id="mcps1.3.4.3.6.5.2.4.1.1"><p id="mrs_01_1438__aa16c745781d349b78d30a9bb6bdfdd9a"><strong id="mrs_01_1438__b144520585392">Configuration Item</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="18.77%" id="mcps1.3.4.3.6.5.2.4.1.2"><p id="mrs_01_1438__a0b1eb379a90949f1a3b6596ce6267998"><strong id="mrs_01_1438__b44971799412">Default Value</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="47.9%" id="mcps1.3.4.3.6.5.2.4.1.3"><p id="mrs_01_1438__a4d564d0654284f3bb1cbe5060b3ec559"><strong id="mrs_01_1438__b1630111018419">Description</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1438__r94385d596ea24afb9de0c09d53536e3c"><td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.4.3.6.5.2.4.1.1 "><p id="mrs_01_1438__a6b07e75ebea946f99dc998958034c7de">BAD_RECORDS_LOGGER_ENABLE</p>
</td>
<td class="cellrowborder" valign="top" width="18.77%" headers="mcps1.3.4.3.6.5.2.4.1.2 "><p id="mrs_01_1438__a396cd53d85964928991a68014fca76b7">false</p>
</td>
<td class="cellrowborder" valign="top" width="47.9%" headers="mcps1.3.4.3.6.5.2.4.1.3 "><p id="mrs_01_1438__a7ce11a48c72248898b52e15a32e46759">Whether to create logs with details about bad records</p>
</td>
</tr>
<tr id="mrs_01_1438__rf7cdfefabd8a44cc94d717c8344ac055"><td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.4.3.6.5.2.4.1.1 "><p id="mrs_01_1438__a10ed784040b04a459811ab9f204afd00">BAD_RECORDS_ACTION</p>
</td>
<td class="cellrowborder" valign="top" width="18.77%" headers="mcps1.3.4.3.6.5.2.4.1.2 "><p id="mrs_01_1438__a2b8095e81a1346f1b89e42e6c2cf1b6c">FAIL</p>
</td>
<td class="cellrowborder" valign="top" width="47.9%" headers="mcps1.3.4.3.6.5.2.4.1.3 "><p id="mrs_01_1438__a740daad1a2fa4c7babf23354e4974b5b">The four types of actions for bad records are as follows:</p>
<ul id="mrs_01_1438__u24927ebfb4aa4766b4c59048b36165d6"><li id="mrs_01_1438__l1b5f66b1f17e4ecbb0b41fec9bddede7"><strong id="mrs_01_1438__b138771837164111">FORCE</strong>: Auto-corrects the data by storing the bad records as NULL.</li><li id="mrs_01_1438__l268ae36c07d8479d9a52d0c2913236b1"><strong id="mrs_01_1438__b1062844194113">REDIRECT</strong>: Bad records are written to the raw CSV instead of being loaded.</li><li id="mrs_01_1438__lae6d364f7c6f4e1cba1940172bee671a"><strong id="mrs_01_1438__b74236487414">IGNORE</strong>: Bad records are neither loaded nor written to the raw CSV.</li><li id="mrs_01_1438__l4f5e85ac1ee7433ead84da06fef75f57"><strong id="mrs_01_1438__b19774115484110">FAIL</strong>: Data loading fails if any bad records are found.<div class="note" id="mrs_01_1438__n2bd09a5659d24631960c184c7256ffe9"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1438__ab2ca7ceebf5a4e9abfa75614362ccc57">In loaded data, if all records are bad records, <strong id="mrs_01_1438__b18719185816418">BAD_RECORDS_ACTION</strong> is invalid and the load operation fails.</p>
</div></div>
</li></ul>
</td>
</tr>
<tr id="mrs_01_1438__rf49f873c0aa044babb8650055198ae9d"><td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.4.3.6.5.2.4.1.1 "><p id="mrs_01_1438__a150a2a72d9fc4831bdb925caaf5a29e4">IS_EMPTY_DATA_BAD_RECORD</p>
</td>
<td class="cellrowborder" valign="top" width="18.77%" headers="mcps1.3.4.3.6.5.2.4.1.2 "><p id="mrs_01_1438__a6e2be65718414333a0fda67683210689">false</p>
</td>
<td class="cellrowborder" valign="top" width="47.9%" headers="mcps1.3.4.3.6.5.2.4.1.3 "><p id="mrs_01_1438__a1ea1d0479de541968812b4a8d73a5049">Whether empty data of a column to be considered as bad record or not. If this parameter is set to <span class="parmvalue" id="mrs_01_1438__p81aaa3308ebe4d61ab8461b145120dd6"><b>false</b></span>, empty data ("",', or,) is not considered as bad records. If this parameter is set to <span class="parmvalue" id="mrs_01_1438__pf785bd5b98104bad9cd11e022e3241a9"><b>true</b></span>, empty data is considered as bad records.</p>
</td>
</tr>
<tr id="mrs_01_1438__r5224c428d88047e9ac7fa3f05d9b90c7"><td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.4.3.6.5.2.4.1.1 "><p id="mrs_01_1438__ae647235d37f14915b409ec611fb406ee">BAD_RECORD_PATH</p>
</td>
<td class="cellrowborder" valign="top" width="18.77%" headers="mcps1.3.4.3.6.5.2.4.1.2 "><p id="mrs_01_1438__a4538aa2bab2c40eba646580cc80c9884">-</p>
</td>
<td class="cellrowborder" valign="top" width="47.9%" headers="mcps1.3.4.3.6.5.2.4.1.3 "><p id="mrs_01_1438__a8636145688ca4d7282f6673a8c836f23">HDFS path where bad records are stored. The default value is <strong id="mrs_01_1438__b14588143074410">Null</strong>. If bad records logging or bad records operation redirection is enabled, the path must be configured by the user.</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_1438__a03294fe1527a4eb18dd2c352c5725a17">Example:</p>
<p id="mrs_01_1438__a0018bb6b15f443359e9b0a7df5643245"><b><span class="cmdname" id="mrs_01_1438__cmdname1844319101612">LOAD DATA INPATH</span></b> <i><span class="varname" id="mrs_01_1438__ve4b0dc73413e49f285df227f4fe7c92c">'filepath.csv'</span></i> <b><span class="cmdname" id="mrs_01_1438__cmdname3529162510167">INTO TABLE</span></b> <i><span class="varname" id="mrs_01_1438__vdfcbf6ff74c94e488c0b2b1c762d5114">tablename</span></i> <i><span class="varname" id="mrs_01_1438__vc72b6aab30e54912ac034d2920b9dcf6">OPTIONS('BAD_RECORDS_LOGGER_ENABLE'='true',</span></i> <i><span class="varname" id="mrs_01_1438__v910d4197691c42e79ee55880aede57ae">'BAD_RECORD_PATH'='hdfs://hacluster/tmp/carbon', 'BAD_RECORDS_ACTION'='REDIRECT', 'IS_EMPTY_DATA_BAD_RECORD'='false');</span></i></p>
<div class="note" id="mrs_01_1438__n47e1cd685d3a437b96975c19dd3bcc29"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1438__p15116191113137">If <span class="parmvalue" id="mrs_01_1438__p4ee20d99489548d69a651fd271d2f816"><b>REDIRECT</b></span> is used, CarbonData will add all bad records into a separate CSV file. However, this file must not be used for subsequent data loading because the content may not exactly match the source record. You must clean up the source record for further data ingestion. This option is used to remind you which records are bad.</p>
</div></div>
</li><li id="mrs_01_1438__l8296a25adbe14022911a386ff9fd594a"><strong id="mrs_01_1438__b1258073104517">MAXCOLUMNS</strong>: (Optional) Specifies the maximum number of columns parsed by a CSV parser in a line.<p id="mrs_01_1438__ae25db3c38ed1440a821a165e70aa692a"><i><span class="varname" id="mrs_01_1438__v3f7f559b0809404b906620f4843cc94e">OPTIONS('MAXCOLUMNS'='400')</span></i></p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1438__t0a937ca7248e4cd79e2c94c3327810e2" frame="border" border="1" rules="all"><caption><b>Table 3 </b>MAXCOLUMNS</caption><thead align="left"><tr id="mrs_01_1438__r760c0409a0854dd6b927cd0ab19f2629"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.3.7.3.2.4.1.1"><p id="mrs_01_1438__a9b6edd038cf047faa424b8f75e96c72a"><strong id="mrs_01_1438__b3787912154514">Name of the Optional Parameter</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.3.7.3.2.4.1.2"><p id="mrs_01_1438__a7750b41d467f476d9dec3c3c17be4b87"><strong id="mrs_01_1438__b7136171412457">Default Value</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.3.7.3.2.4.1.3"><p id="mrs_01_1438__a82ea6a3abf7d4a0791943e46b75f5445"><strong id="mrs_01_1438__b19957101664511">Maximum Value</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1438__rba9bfdc6615445df90c4d0d9f3869a70"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.3.2.4.1.1 "><p id="mrs_01_1438__aee4463c3b6da4a2e874dcb238b1509d1">MAXCOLUMNS</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.3.2.4.1.2 "><p id="mrs_01_1438__a0290c5c0d57b4f7d9f315ff440392a9d">2000</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.3.2.4.1.3 "><p id="mrs_01_1438__aa900909704bd405d832c06a13121e2b1">20000</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1438__td0c56fed2a404209a82b6de6500b4f21" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Behavior chart of MAXCOLUMNS</caption><thead align="left"><tr id="mrs_01_1438__rd2a2e38a985042fe85f5d2a8a6bc82e6"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.3.7.4.2.4.1.1"><p id="mrs_01_1438__a44cf44ddb597428a9239c573b9a2b4e7"><strong id="mrs_01_1438__b1663113153466">MAXCOLUMNS Value</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.3.7.4.2.4.1.2"><p id="mrs_01_1438__a2c1824a7da8048439d05ebd440ed022f"><strong id="mrs_01_1438__b145841219174610">Number of Columns in the File Header</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.4.3.7.4.2.4.1.3"><p id="mrs_01_1438__a23c7a27ba57f4a00a7cf7722b5177807"><strong id="mrs_01_1438__b12598223467">Final Value Considered</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1438__r517c5eff92a24bb38895408dfc4f064c"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.1 "><p id="mrs_01_1438__a30b00dffca24437398f092cff6cf7e1d">Not specified in Load options</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.2 "><p id="mrs_01_1438__a142800ade2ac473cb53aa908df6b2c07">5</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.3 "><p id="mrs_01_1438__a2431de0d60e64f96a505a31991b16f39">2000</p>
</td>
</tr>
<tr id="mrs_01_1438__r4ff195434a7847e185208d61b46ff760"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.1 "><p id="mrs_01_1438__a0ca97ec98bc847deac0b7fc3db5a0777">Not specified in Load options</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.2 "><p id="mrs_01_1438__a722d03d860f44772b43e96bc42805ae7">6000</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.3 "><p id="mrs_01_1438__af417d1d7bf7c49dbb21fadd8f7305540">6000</p>
</td>
</tr>
<tr id="mrs_01_1438__r79b6016fb92c463c9817109a5d08e136"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.1 "><p id="mrs_01_1438__a708fc2d4d7a94ebcaf896e409b140538">40</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.2 "><p id="mrs_01_1438__a8c5aeeb4d089493183dd0edd8bf8623d">7</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.3 "><p id="mrs_01_1438__ae31d0d139b944fbda9699ee278ddaee3">Max (column count of file header, MAXCOLUMNS value)</p>
</td>
</tr>
<tr id="mrs_01_1438__r7e359a4b34f74fe787e495d301d8797c"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.1 "><p id="mrs_01_1438__aabd95f8d61df43c4b55687db18e6f68c">22000</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.2 "><p id="mrs_01_1438__a6a26db6aa6e54907bee756d3096c6fb0">40</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.3 "><p id="mrs_01_1438__aff6f2291795246999e4df4c7d6c4b39b">20000</p>
</td>
</tr>
<tr id="mrs_01_1438__rfd7545f821904187b1af686873985e66"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.1 "><p id="mrs_01_1438__a4381ca0409e24402b6055e3b0b55768b">60</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.2 "><p id="mrs_01_1438__a2287e653f16949ecb4e6103eeaed2837">Not specified in Load options</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.4.3.7.4.2.4.1.3 "><p id="mrs_01_1438__a6c87edebe8b94e529961aa40f17fd9d7">Max (Number of columns in the first line of the CSV file, MAXCOLUMNS value)</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="mrs_01_1438__n95e15395894e4d8ca04e931a9d7fd14d"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1438__accffdca1aa094ba9b725fdbb424c3c30">There must be sufficient executor memory for setting the maximum value of <strong id="mrs_01_1438__b1632885819463">MAXCOLUMNS Option</strong>. Otherwise, data loading will fail.</p>
</div></div>
</li></ul>
</div>
<ul id="mrs_01_1438__ul139104862312"><li id="mrs_01_1438__li183964819232">If <strong id="mrs_01_1438__b169111255161311">SORT_SCOPE</strong> is set to <strong id="mrs_01_1438__b1391212556135">GLOBAL_SORT</strong> during table creation, you can specify the number of partitions to be used when sorting data. If this parameter is not set or is set to a value less than <strong id="mrs_01_1438__b52596527483">1</strong>, the number of map tasks is used as the number of reduce tasks. It is recommended that each reduce task process 512 MB to 1 GB data.<p id="mrs_01_1438__p1891140162311"><i><span class="varname" id="mrs_01_1438__varname12840142918247">OPTIONS('GLOBAL_SORT_PARTITIONS'='2')</span></i></p>
<div class="note" id="mrs_01_1438__note26601516132617"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1438__p266141632612">To increase the number of partitions, you may need to increase the value of <span class="parmname" id="mrs_01_1438__parmname14601203518272"><b>spark.driver.maxResultSize</b></span>, as the sampling data collected in the driver increases with the number of partitions.</p>
</div></div>
</li></ul>
<ul id="mrs_01_1438__ul143711032132211"><li id="mrs_01_1438__li1210618571271"><strong id="mrs_01_1438__b19794154195313">DATEFORMAT</strong>: Specifies the date format of the table.<p id="mrs_01_1438__p1611614374224"><i><span class="varname" id="mrs_01_1438__varname1377716391297">OPTIONS('DATEFORMAT'='dateFormat')</span></i></p>
<div class="note" id="mrs_01_1438__note11118163716228"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1438__p38021112307">Date formats are specified by date pattern strings. The date pattern letters in Carbon are same as in JAVA.</p>
</div></div>
</li></ul>
<ul id="mrs_01_1438__ul488613213316"><li id="mrs_01_1438__li1784218400311"><strong id="mrs_01_1438__b126911746205314">TIMESTAMPFORMAT</strong>: Specifies the timestamp of a table.</li><li id="mrs_01_1438__li128871032133118"><i><span class="varname" id="mrs_01_1438__varname1189143213119">OPTIONS('TIMESTAMPFORMAT'='timestampFormat')</span></i></li></ul>
<ul id="mrs_01_1438__ul14322143713319"><li id="mrs_01_1438__li1230659113313"><strong id="mrs_01_1438__b15513555195711">SKIP_EMPTY_LINE</strong>: Ignores empty rows in the CSV file during data loading.<p id="mrs_01_1438__p9325143719317"><i><span class="varname" id="mrs_01_1438__varname12326437103113">OPTIONS('SKIP_EMPTY_LINE'='TRUE/FALSE')</span></i></p>
</li><li id="mrs_01_1438__li2087413919372"><strong>Optional: </strong><strong id="mrs_01_1438__b193241658195710">SCALE_FACTOR</strong>: Used to control the number of partitions for <strong id="mrs_01_1438__b175331919515">RANGE_COLUMN</strong>, <strong id="mrs_01_1438__b135345125114">SCALE_FACTOR</strong>. The formula is as follows:<pre class="codeblock" id="mrs_01_1438__codeblock14859427161116">splitSize = max(blocklet_size, (block_size - blocklet_size)) * scale_factor
numPartitions = total size of input data / splitSize</pre>
<p id="mrs_01_1438__p83411939964">The default value is <strong id="mrs_01_1438__b5982535135120">3</strong>. The value ranges from <strong id="mrs_01_1438__b12988135155118">1</strong> to <strong id="mrs_01_1438__b209891035105111">300</strong>.</p>
<p id="mrs_01_1438__p1531176151018"><i><span class="varname" id="mrs_01_1438__varname193217614103">OPTIONS('SCALE_FACTOR'='10')</span></i></p>
<div class="note" id="mrs_01_1438__note13651637267"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1438__ul52441448101112"><li id="mrs_01_1438__li3245348141117">If <strong id="mrs_01_1438__b65940105114">GLOBAL_SORT_PARTITIONS</strong> and <strong id="mrs_01_1438__b151119405512">SCALE_FACTOR</strong> are used at the same time, only <strong id="mrs_01_1438__b71310408515">GLOBAL_SORT_PARTITIONS</strong> is valid.</li><li id="mrs_01_1438__li7419194501219">The compaction on <strong id="mrs_01_1438__b8572945105118">RANGE_COLUMN</strong> will use <strong id="mrs_01_1438__b75781451511">LOCAL_SORT</strong> by default.</li></ul>
</div></div>
</li></ul>
<div class="section" id="mrs_01_1438__sf62f56a2f9df49a2b0e93cb0a76e5d90"><h4 class="sectiontitle">Scenarios</h4><p id="mrs_01_1438__a910c043ebc1a4404aeabd7e84c5f900d">To load a CSV file to a CarbonData table, run the following statement:</p>
<p id="mrs_01_1438__a2aec6f7d6ca4474f9a2174ef6ba78e5d"><b><span class="cmdname" id="mrs_01_1438__cmdname825616384169">LOAD DATA</span></b> <i><span class="varname" id="mrs_01_1438__vb66652ff6ba947e49afee78c5da00a99">INPATH 'folder path' INTO TABLE tablename OPTIONS(property_name=property_value, ...);</span></i></p>
</div>
<div class="section" id="mrs_01_1438__s0cae638894c146bc8a2f719647701445"><h4 class="sectiontitle">Examples</h4><p id="mrs_01_1438__p19305327184316">The data in the <strong id="mrs_01_1438__b2042312447462">data.csv</strong> file is as follows:</p>
<pre class="screen" id="mrs_01_1438__screen66949599437">ID,date,country,name,phonetype,serialname,salary
4,2014-01-21 00:00:00,city1,aaa4,phone2435,ASD66902,15003
5,2014-01-22 00:00:00,city1,aaa5,phone2441,ASD90633,15004
6,2014-03-07 00:00:00,city1,aaa6,phone294,ASD59961,15005</pre>
<p id="mrs_01_1438__p510951994412">CREATE TABLE carbontable(ID int, date Timestamp, country String, name String, phonetype String, serialname String,salary int) STORED AS carbondata;</p>
<p id="mrs_01_1438__a777819584e1146b999ca58d4dc4f18a2"><b><span class="cmdname" id="mrs_01_1438__cmdname126231427162">LOAD DATA</span></b> <i><span class="varname" id="mrs_01_1438__v6adcca080d2e4be29e2be1e4d293f360">inpath 'hdfs://hacluster/tmp/data.csv' INTO table carbontable</span></i></p>
<p id="mrs_01_1438__a5540be68735a4e9387435a10272b6440"><i><span class="varname" id="mrs_01_1438__v7295a9bcd2274969b37d3e795f137390">options('DELIMITER'=','</span></i><i><span class="varname" id="mrs_01_1438__v893432384991420f8d71bda79c680a4e">);</span></i></p>
</div>
<div class="section" id="mrs_01_1438__s4fbb207c33234984b0530e8ffe1609c3"><h4 class="sectiontitle">System Response</h4><p id="mrs_01_1438__a5f97b253cfc245c7bcb19802be0394ab">Success or failure will be recorded in the driver logs.</p>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1437.html">DML</a></div>
</div>
</div>
View Git Blame Copy Permalink