yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
0b3a43371a4bcc99577594de0a685249d676a567
doc-exports/docs/dataartsstudio/umn/dataartsstudio_01_0066.html
chenxiaoxiong f9e2808b7c DataArts UMN 20250810 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
2025-09-02 10:44:13 +00:00

181 lines
24 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<a name="dataartsstudio_01_0066"></a><a name="dataartsstudio_01_0066"></a>
<h1 class="topictitle1">To Hive</h1>
<div id="body8662426"><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p32842430161426">If the destination link of a job is a <a href="dataartsstudio_01_0026.html">Hive link</a>, configure the destination job parameters based on <a href="#dataartsstudio_01_0066__en-us_topic_0108275481_table31823995163953">Table 1</a>.</p>
<div class="tablenoborder"><a name="dataartsstudio_01_0066__en-us_topic_0108275481_table31823995163953"></a><a name="en-us_topic_0108275481_table31823995163953"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0066__en-us_topic_0108275481_table31823995163953" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row18653487163953"><th align="left" class="cellrowborder" valign="top" width="18.28%" id="mcps1.3.2.2.4.1.1"><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p15314298163953">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="62.43000000000001%" id="mcps1.3.2.2.4.1.2"><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p32498630163953">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="19.290000000000003%" id="mcps1.3.2.2.4.1.3"><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p15143370163953">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row1928353163953"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p28108739161524">Database Name</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p17702619155928">Database name. Click the icon next to the text box. The dialog box for selecting the database is displayed.</p>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p64647157163953">default</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row11687830163953"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p23518636161524">Table Name</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p17984995155931">Destination table name. Click the icon next to the text box. The dialog box for selecting the table is displayed.</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1210244910548">This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically. </p>
<div class="note" id="dataartsstudio_01_0066__en-us_topic_0108275481_note45791757134910"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275319_p52974484409">If you have configured a macro variable of date and time and schedule a CDM job through <span id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275319_text9997118203">DataArts Studio DataArts Factory</span>, the system replaces the macro variable of date and time with (<em id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275319_i799871152012">Planned start time of the data development job</em> – <em id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275319_i0998101192020">Offset</em>) rather than (<em id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275319_i299821132018">Actual start time of the CDM job</em> – <em id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275319_i129981917200">Offset</em>).</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p50683068163953">TBL_X</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row17946234105618"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p18627184215564">Auto Table Creation</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><div class="p" id="dataartsstudio_01_0066__en-us_topic_0108275481_p5627124216567">This parameter is displayed only when the source is a relational database. The options are as follows:<ul id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_ul1730585217292"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_li19305135216298"><strong id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_b84235270695756">Non-auto creation</strong>: CDM will not automatically create a table.</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_li193052523291"><strong id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_b84235270695832">Auto creation</strong>: If the destination database does not contain the table specified by <span class="parmname" id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_parmname1393810132155437"><b>Table Name</b></span>, CDM will automatically create the table. If the table specified by <span class="parmname" id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_parmname136853505353"><b>Table Name</b></span> already exists, no table is created and data is written to the existing table.</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_li5305125214295"><strong id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_b842352706155521">Deletion before creation</strong>: CDM deletes the table specified by <span class="parmname" id="dataartsstudio_01_0066__en-us_topic_0108275481_en-us_topic_0108275472_parmname517944180155536"><b>Table Name</b></span>, and then creates the table again.</li></ul>
</div>
<div class="note" id="dataartsstudio_01_0066__en-us_topic_0108275481_note26274422566"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dataartsstudio_01_0066__en-us_topic_0108275481_ul1022720451137"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li16227124510130">Only column comments are synchronized during automatic table creation. Table comments are not synchronized.</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li122784511134">Primary keys cannot be synchronized during automatic table creation.</li></ul>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p36271042135617">Non-auto creation</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row6452539133510"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p5452193919351">Source side null value conversion value</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1452839183520">Value to which the source null value is converted</p>
<ul id="dataartsstudio_01_0066__en-us_topic_0108275481_ul17237999360"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li1723749123619">TO_NULL</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li187741116163611">TO_EMPTY_STRRING</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li5317172603612">TO_NULL_STRING</li></ul>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p163891954163517">TO_NULL</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row43042085151531"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p21324079151542">Clear Data Before Import</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><div class="p" id="dataartsstudio_01_0066__en-us_topic_0108275481_p348084512021">Whether the data in the destination table is cleared before data import. The options are as follows:<ul id="dataartsstudio_01_0066__en-us_topic_0108275481_ua1819b93796044cd9f46f9dfb3e78f6b"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_l6ab161004da642a19cc2c8de077ee24e"><strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b842352706155658">Yes</strong>: The data is cleared.</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_l8659f1f4120241ee86b1c992eed6a627"><strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b842352706155739">No</strong>: The data is not cleared. Instead, it will be added to the existing table.</li></ul>
</div>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p23412518151542">Yes</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row33171030123711"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p13171130183718">Processing mode of newline characters</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p4317030163717">Policy for processing the newline characters in the data written to Hive textfile tables</p>
<ul id="dataartsstudio_01_0066__en-us_topic_0108275481_ul124021113143918"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li1240261310392">Delete</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li16402191319390">Replace with another string</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li1240213132397">Ignore</li></ul>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p113171230183710">Delete</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row1939111144817"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p123913134813">Hive Table Partition Field</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1879122216485">This parameter is unavailable when <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b819144445610">Auto Table Creation</strong> is set to <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b1865214586563">Non-auto Creation</strong>.</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p469611393480">Partition fields for creating a Hive table. Use commas (,) to separate multiple fields.</p>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p297320552487">A,B</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row183717284814"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p113714220488">Table Path</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p15851646165110">This parameter is unavailable when <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b12846115913563">Auto Table Creation</strong> is set to <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b484618594561">Non-auto Creation</strong>.</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p784419619497">It specifies the table path.</p>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1738722485">-</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row841212284814"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p741210216482">Storage Format</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p41171284510">This parameter is unavailable when <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b2261102817581">Auto Table Creation</strong> is set to <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b42612283584">Non-auto Creation</strong>.</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p297714211552">It specifies the storage format.</p>
<ul id="dataartsstudio_01_0066__en-us_topic_0108275481_ul14781101018508"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li1878114108504">Row-based storage format: TEXTFILE</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li3781161017504">Column-based storage formats: ORC, RCFILE, and PARQUET</li></ul>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p48477231494">TEXTFILE data is stored in plaintext. If data contains special characters, data may be written incorrectly. Exercise caution when using this format. The ORC format is recommended.</p>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p82531529194913">ORC</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row1879218503206"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p6792450142016">ClearDataMode</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p792655194212">This parameter is available when <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b64953135216">Clear Data Before Import</strong> is set to <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b2049510131429">Yes</strong>.</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p7381171795418">It specifies the mode for clearing data in the Hive table.</p>
<ul id="dataartsstudio_01_0066__en-us_topic_0108275481_ul233457164315"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li372616818437"><strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b11444351843">LOAD_OVERWRITE</strong>: A temporary data file directory is generated and loaded to the Hive table using the load overwrite syntax of Hive.</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li14700131615160"><strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b330965113610">TRUNCATE</strong>: Data files in partitions are deleted, but partitions are not deleted.<div class="note" id="dataartsstudio_01_0066__en-us_topic_0108275481_note164535569367"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p104531756193610">If the destination is a partitioned table, you are advised to select LOAD_OVERWRITE. Otherwise, the cluster memory or disks may be overloaded.</p>
</div></div>
</li></ul>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p20792115032012">TRUCATE</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row11284228135717"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1628472814572">Partitions info</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p6622439171414">This parameter is available when <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b115651719105713">Clear Data Before Import</strong> is set to <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b17243192320575">Yes</strong>. If the destination is a partitioned table, you must specify partitions.</p>
<ul id="dataartsstudio_01_0066__en-us_topic_0108275481_ul91471631122410"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li144504215271">If you select the <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b854615918812">TRUCATE</strong> mode, only the data files in the partitions are deleted.</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li44503216276">If you select the <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b1948814918911">LOAD_OVERWRITE</strong> mode, data is written to a specified partition and overwrites the existing data.<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1317102214269"></p>
</li></ul>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1368112410119">Single partition: <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b1688317579010">year=2020,location=sun</strong></p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1017720191219">Multiple partitions: <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b34430107111">['year=2020,location=sun', 'year=2021,location=earth']</strong></p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1464317221578">Partitions of the previous day:</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p9333840175719">day='${dateformat(yyyy-MM-dd HH:mm:ss, -1, DAY)}',</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row1769831204615"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p122792180469">Executing Analyze Statements</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1291763017568">After all data is written, the ANALYZE TABLE statement is asynchronously executed to accelerate query of data from Hive tables.</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p10504195811116">Run the following SQL statements:</p>
<ul id="dataartsstudio_01_0066__en-us_topic_0108275481_ul34181735101217"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li1841816353129">Non-partitioned table: <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b1997011751110">ANALYZE TABLE tablename COMPUTE STATISTICS</strong></li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li1841823511212">Partitioned table: <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b2706325137">ANALYZE TABLE tablename PARTITION(partcol1[=val1], partcol2[=val2], ...) COMPUTE STATISTICS</strong></li></ul>
<div class="note" id="dataartsstudio_01_0066__en-us_topic_0108275481_note18566181112518"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p714142416254">Parameter <strong id="dataartsstudio_01_0066__en-us_topic_0108275481_b12690171414415">Executing Analyze Statements</strong> applies only to the migration of a single table.</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p4970252124413">Running the ANALYZE statements may exert pressure on Hive.</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1469917120461">Yes</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row168928462459"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p208928461458">Maximum memory size of the internal write queue</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p206142519468">If the memory is insufficient, change the value of this parameter as needed. If the value is too small, the migration speed will be affected.</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p0892194613451">The value ranges from 1 to 128 MB. The default value is empty, indicating that there is no limit. If you set a value beyond the range, there is no limit.</p>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p19892346114512">16</p>
</td>
</tr>
<tr id="dataartsstudio_01_0066__en-us_topic_0108275481_row10985472451"><td class="cellrowborder" valign="top" width="18.28%" headers="mcps1.3.2.2.4.1.1 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p1798194712451">Maximum memory size of the internal conversion queue</p>
</td>
<td class="cellrowborder" valign="top" width="62.43000000000001%" headers="mcps1.3.2.2.4.1.2 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p117321043104619">If the memory is insufficient, change the value of this parameter as needed. If the value is too small, the migration speed will be affected.</p>
<p id="dataartsstudio_01_0066__en-us_topic_0108275481_p129844784510">The value ranges from 1 to 128 MB. The default value is empty, indicating that there is no limit. If you set a value beyond the range, there is no limit.</p>
</td>
<td class="cellrowborder" valign="top" width="19.290000000000003%" headers="mcps1.3.2.2.4.1.3 "><p id="dataartsstudio_01_0066__en-us_topic_0108275481_p298114713455">16</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="dataartsstudio_01_0066__en-us_topic_0108275481_note4345143122212"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dataartsstudio_01_0066__en-us_topic_0108275481_ul1868905105111"><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li18690125185113">If the source Hive contains both the array and map types of data, the destination table format can only be the ORC or parquet complex type. If the destination table format is RC or TEXT, the source data will be processed and can be successfully written.</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li17690145155115">As the map type is an unordered data structure, the data type may change after a migration.</li><li id="dataartsstudio_01_0066__en-us_topic_0108275481_li156901757513"><span id="dataartsstudio_01_0066__en-us_topic_0108275481_ph1126687105519">If Hive serves as the migration destination and the storage format is Textfile, delimiters must be explicitly specified in the statement for creating Hive tables. The following is an example:</span><pre class="screen" id="dataartsstudio_01_0066__en-us_topic_0108275481_screen765433135516">CREATE TABLE csv_tbl(
smallint_value smallint,
tinyint_value tinyint,
int_value int,
bigint_value bigint,
float_value float,
double_value double,
decimal_value decimal(9, 7),
timestmamp_value timestamp,
date_value date,
varchar_value varchar(100),
string_value string,
char_value char(20),
boolean_value boolean,
binary_value binary,
varchar_null varchar(100),
string_null string,
char_null char(20),
int_null int
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
"separatorChar" = "\t",
"quoteChar" = "'",
"escapeChar" = "\\"
)
STORED AS TEXTFILE;</pre>
</li></ul>
</div></div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0061.html">Configuring CDM Destination Job Parameters</a></div>
</div>
</div>
View Git Blame Copy Permalink