yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
0b3a43371a4bcc99577594de0a685249d676a567
doc-exports/docs/dataartsstudio/umn/dataartsstudio_01_0108.html
chenxiaoxiong f9e2808b7c DataArts UMN 20250810 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
2025-09-02 10:44:13 +00:00

371 lines
75 KiB
HTML
Raw Blame History

<a name="dataartsstudio_01_0108"></a><a name="dataartsstudio_01_0108"></a>
<h1 class="topictitle1">File Formats</h1>
<div id="body32001227"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p101750198439">When creating a CDM job, you need to specify <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname8860123183119"><b>File Format</b></span> in the job parameters of the migration source and destination in some scenarios. This section describes the application scenarios, subparameters, common parameters, and usage examples of the supported file formats.</p>
<ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul11248467589"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li152481363582"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section469044274219">CSV</a></li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1253178231"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section869714051913">JSON</a></li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li529555918264"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section39751230182317">Binary</a></li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1293586172719"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section148391468257">Common parameters</a></li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li11498161419273"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section5636174252511">Solutions to File Format Problems</a></li></ul>
<div class="section" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section469044274219"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section469044274219"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section469044274219"></a><h4 class="sectiontitle">CSV</h4><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p32662441802">To read or write a CSV file, set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname479418356428"><b>File Format</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue1135814134213"><b>CSV</b></span>. The CSV format can be used in the following scenarios:</p>
<ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul111371115"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1511278119">Import files to a database or NoSQL.</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li14952093114">Export data from a database or NoSQL to files.</li></ul>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p04076156419">After selecting the CSV format, you can also configure the following optional sub-parameters:</p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1896385615128"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li19945541171217">1. Line Separator</a></p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p12321622192020"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li2018172316145">2. Field Delimiter</a></p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p474773412203"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li969133717144">3. Encoding Type</a></p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1279642102018"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li183111219131712">4. Use Quote Character</a></p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p17131050122015"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1183030141819">5. Use RE to Separate Fields</a></p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p142147162110"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1425617365187">6. Use First Row as Header</a></p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1287617175216"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li207411446121814">7. File Size</a></p>
</div>
<ol id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ol1194594116121"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li19945541171217"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li19945541171217"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li19945541171217"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b2929170115011">Line Separator</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p840731584119">Character used to separate lines in a CSV file. The value can be a single character, multiple characters, or special characters. Special characters can be entered using the URL encoded characters. The following table lists the URL encoded characters of commonly used special characters.</p>
<div class="tablenoborder"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_table11215204115"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_table11215204115"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_table11215204115" frame="border" border="1" rules="all"><caption><b>Table 1 </b>URL encoded characters of special characters</caption><thead align="left"><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row1040751510415"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.4.1.3.2.3.1.1"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p204071115124113">Special Character</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.4.1.3.2.3.1.2"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1840721520413">URL Encoded Character</p>
</th>
</tr>
</thead>
<tbody><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row1540713155418"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p44072015114114">Space</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1440771510415">%20</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row14073155415"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1140771544120">Tab</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p18407815194117">%09</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row1940717151413"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p14407131514413">%</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p94071215154113">%25</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row104071515204115"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p740701511413">Enter</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p12407815204110">%0d</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row2407161514112"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p14071715104112">Newline character</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p8407215194111">%0a</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row1140712158416"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p13407415164111">Start of heading\u0001 (SOH)</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.1.3.2.3.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p54079158413">%01</p>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li2018172316145"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li2018172316145"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li2018172316145"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b1170617320589">Field Delimiter</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p340791534119">Character used to separate columns in a CSV file. The value can be a single character, multiple characters, or special characters. For details, see <a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_table11215204115">Table 1</a>.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li969133717144"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li969133717144"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li969133717144"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b7832147590">Encoding Type</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p64070152412">Encoding type of a CSV file. The default value is <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b126451343104413">UTF-8</strong>. </p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p14071415114113">If this parameter is specified at the migration source, the specified encoding type is used to parse the file. If this parameter is specified at the migration destination, the specified encoding type is used to write data to the file.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li183111219131712"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li183111219131712"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li183111219131712"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b128209421224">Use Quote Character</strong><ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul111611225553"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li416112285510">Exporting data from a database or NoSQL to CSV files (configuring <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b1345177887">Use Quote Character</strong> at the migration destination): If a field delimiter appears in the character string of a column of data at the migration source, set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname54265477125"><b>Use Quote Character</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue342518523123"><b>Yes</b></span> at the migration destination to quote the character string as a whole and write it into the CSV file. Currently, CDM uses double quotation marks ("") as the quote character only. <a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_fig87952010132518">Figure 1</a> shows that the value of the <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b242894719154">name</strong> field in the database contains a comma (,).<div class="fignone" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_fig87952010132518"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_fig87952010132518"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_fig87952010132518"></a><span class="figcap"><b>Figure 1 </b>Field value containing the field delimiter</span><br><span><img id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_image14077158418" src="en-us_image_0000002234236152.png"></span></div>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p13407111519418">If you do not use the quote character, the exported CSV file is displayed as follows:</p>
<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen3449185113411">3,hello,world,abc</pre>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p0407101512413">If you use the quote character, the exported CSV file is displayed as follows:</p>
<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen4450149174216">3,"hello,world",abc</pre>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p8407151534118">If the data in the database contains double quotation marks ("") and you set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname176145911204"><b>Use Quote Character</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue1876105918205"><b>Yes</b></span>, the quote character in the exported CSV file is displayed as three double quotation marks ("""). For example, if the value of a field is <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue1316493192312"><b>a"hello,world"c</b></span>, the exported data is as follows:</p>
<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen1515235773912">"""a"hello,world"c"""</pre>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li0601204620551">Exporting CSV files to a database or NoSQL (configuring <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname1370975518489"><b>Use Quote Character</b></span> at the migration source): If you want to import the CSV files with quoted values to a database correctly, set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname147097551484"><b>Use Quote Character</b></span> to <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b8709555174817">Yes</strong> at the migration source to write the quoted values as a whole.</li></ul>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1183030141819"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1183030141819"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1183030141819"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b12043113015">Use RE to Separate Fields</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p9407141518417">This function is used to parse complex semi-structured text, such as log files. For details, see <a href="dataartsstudio_01_0593.html">Using Regular Expressions to Separate Semi-structured Text</a>.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1425617365187"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1425617365187"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1425617365187"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b9636204743312">Use First Row as Header</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p114075156419">This parameter is used when CSV files are exported to other locations. If this parameter is specified at the migration source, CDM uses the first row as the header when extracting data. When the CSV files are transferred, the headers are skipped. The number of rows extracted from the migration source is more than the number of rows written to the migration destination. The log files will output the information that the header is skipped during the migration.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li207411446121814"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li207411446121814"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li207411446121814"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b52561344201411">File Size</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p840751511418">This parameter is used when data is exported from the database to a CSV file. If a table contains a large amount of data, a large CSV file is generated after migration, which is inconvenient to download or view. In this case, you can specify this parameter at the migration destination so that multiple CSV files with the specified size can be generated. The value of this parameter is an integer. The unit is MB.</p>
</li></ol>
<div class="section" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section869714051913"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section869714051913"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section869714051913"></a><h4 class="sectiontitle">JSON</h4><div class="p" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p139960194354">The following describes information about the JSON format:<ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul113916311362"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li111391831153617"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li17316535702">JSON Types Supported by CDM</a></li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1862283343619"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li131971342489">JSON Reference Node</a></li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li10513124143616"><a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li179647275105">Copying Data from a JSON File</a></li></ul>
</div>
<ol id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ol16316183514016"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li17316535702"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li17316535702"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li17316535702"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b182301355133216">JSON types supported by CDM: JSON object and JSON array</strong><ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul103161035905"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li12316935908">JSON object: A JSON file contains a single object or multiple objects separated/merged by rows.<ol type="a" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ol1631633513017"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1431616351602">The following is a single JSON object:<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen1431613351406">{
"took" : 190,
"timed_out" : false,
"total" : 1000001,
"max_score" : 1.0
}</pre>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li931610351106">The following are JSON objects separated by rows:<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen133162355012">{"took" : 188, "timed_out" : false, "total" : 1000003, "max_score" : 1.0 }
{"took" : 189, "timed_out" : false, "total" : 1000004, "max_score" : 1.0 }</pre>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li331615357010">The following are merged JSON objects:<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen113168351509">{
"took": 190,
"timed_out": false,
"total": 1000001,
"max_score": 1.0
}
{
"took": 191,
"timed_out": false,
"total": 1000002,
"max_score": 1.0
}</pre>
</li></ol>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li203162354019">JSON array: A JSON file is a JSON array consisting of multiple JSON objects.<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen19852111111617">[{
"took" : 190,
"timed_out" : false,
"total" : 1000001,
"max_score" : 1.0
},
{
"took" : 191,
"timed_out" : false,
"total" : 1000001,
"max_score" : 1.0
}]</pre>
</li></ul>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li131971342489"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li131971342489"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li131971342489"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b0242121354318">JSON Reference Node</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p184073155411">Root node that records data. The data corresponding to the node is a JSON array. CDM extracts data from the array in the same mode. Use periods (.) to separate multi-layer nested JSON nodes.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li179647275105"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li179647275105"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li179647275105"></a><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b1339172711452">Copying Data from a JSON File</strong><ol type="a" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ol6436378134"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li8987112853210">Example 1<div class="p" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p4644315329"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li8987112853210"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li8987112853210"></a>Extract data from multiple objects that are separated or merged. A JSON file contains multiple JSON objects. The following gives an example:<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen19921619171418"> {
"took": 190,
"timed_out": false,
"total": 1000001,
"max_score": 1.0
}
{
"took": 191,
"timed_out": false,
"total": 1000002,
"max_score": 1.0
}
{
"took": 192,
"timed_out": false,
"total": 1000003,
"max_score": 1.0
}</pre>
</div>
<div class="p" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p16423181515414">To extract data from the JSON object and write data to the database in the following formats, set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname788937163615"><b>File Format</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue13108141553616"><b>JSON</b></span> and <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname360912015361"><b>JSON Type</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue204051525163616"><b>JSON object</b></span>, and then map fields.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_table1033515144111" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Example</caption><thead align="left"><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row342391519414"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.1.2.5.2.5.1.1"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p842331524111">took</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.1.2.5.2.5.1.2"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p204232155416">timedOut</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.1.2.5.2.5.1.3"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p184231515204115">total</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.1.2.5.2.5.1.4"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1842311153412">maxScore</p>
</th>
</tr>
</thead>
<tbody><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row7423815204111"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p0423615114112">190</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p144237152413">false</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1842311584114">1000001</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.4 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p44231015144115">1.0</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row204231915134116"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p4423161517413">191</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p942318159415">false</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p94230156413">1000002</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.4 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p10423181584110">1.0</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row174237156417"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p164231715154112">192</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p342319156418">false</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p164238154417">1000003</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.1.2.5.2.5.1.4 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1042311534118">1.0</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li5228737183210"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li5228737183210"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li5228737183210"></a>Example 2<div class="p" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1710014387326"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li5228737183210"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li5228737183210"></a>Extract data from the reference node. A JSON file contains a single JSON object, but the valid data is on a data node. The following gives an example:<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen166582244185">{
"took": 190,
"timed_out": false,
"hits": {
"total": 1000001,
"max_score": 1.0,
"hits":
[{
"_id": "650612",
"_source": {
"name": "tom",
"books": ["book1","book2","book3"]
}
},
{
"_id": "650616",
"_source": {
"name": "tom",
"books": ["book1","book2","book3"]
}
},
{
"_id": "650618",
"_source": {
"name": "tom",
"books": ["book1","book2","book3"]
}
}]
}
}</pre>
</div>
<div class="p" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1442301564112">To write data to the database in the following formats, set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname14478144115566"><b>File Format</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue447813415569"><b>JSON</b></span>, <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname3478194135619"><b>JSON Type</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue1847864119565"><b>JSON object</b></span>, and <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname164462032175717"><b>JSON Reference Node</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue137216285571"><b>hits.hits</b></span>, and then map fields.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_table1897893910448" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Example</caption><thead align="left"><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row15978193913443"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.2.2.7.2.4.1.1"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1099385144412">ID</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.2.2.7.2.4.1.2"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p6993251104416">SourceName</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.5.3.3.2.2.2.7.2.4.1.3"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p2993551164411">SourceBooks</p>
</th>
</tr>
</thead>
<tbody><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row39781391440"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.2.2.7.2.4.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p137311210468">650612</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.2.2.7.2.4.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1497853944417">tom</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.3.2.2.2.7.2.4.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p597810391447">["book1","book2","book3"]</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row11978193964414"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.2.2.7.2.4.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1037391244619">650616</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.2.2.7.2.4.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p097893916447">tom</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.3.2.2.2.7.2.4.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p14978183915445">["book1","book2","book3"]</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row897823994417"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.2.2.7.2.4.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1037321254612">650618</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.2.2.7.2.4.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1197819395448">tom</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.5.3.3.2.2.2.7.2.4.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p129781839154420">["book1","book2","book3"]</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1561513478324">Example 3<div class="p" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p12291650193212"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1561513478324"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1561513478324"></a>Extract data from the JSON array. A JSON file is a JSON array consisting of multiple JSON objects. The following gives an example:<pre class="screen" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_screen689250142418">[{
"took" : 190,
"timed_out" : false,
"total" : 1000001,
"max_score" : 1.0
},
{
"took" : 191,
"timed_out" : false,
"total" : 1000002,
"max_score" : 1.0
}]</pre>
</div>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p14423815204110">To write data to the database in the following formats, set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname144961732115917"><b>File Format</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue1549611326598"><b>JSON</b></span> and <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname9511123216597"><b>JSON Type</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue3511123218596"><b>JSON array</b></span>, and then map fields.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_table465715154118" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Example</caption><thead align="left"><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row1542361519414"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.3.3.2.5.1.1"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p44231515184111">took</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.3.3.2.5.1.2"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p6423101594117">timedOut</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.3.3.2.5.1.3"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p7423101519413">total</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.5.3.3.2.3.3.2.5.1.4"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p12423151574120">maxScore</p>
</th>
</tr>
</thead>
<tbody><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row164236151413"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.3.3.2.5.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p19423101514120">190</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.3.3.2.5.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1242317157413">false</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.3.3.2.5.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p242371564116">1000001</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.3.3.2.5.1.4 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p0423915164111">1.0</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row242381504118"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.3.3.2.5.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p142311510411">191</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.3.3.2.5.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p94231915184114">false</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.3.3.2.5.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p154233151418">1000002</p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.5.3.3.2.3.3.2.5.1.4 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p4423415164116">1.0</p>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li12771135453216">Example 4<div class="p" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p095115543220"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li12771135453216"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li12771135453216"></a>Configure a converter when parsing the JSON file. On the premise of <a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li5228737183210">example 2</a>, to add the <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b898148194414">hits.max_score</strong> field to all records, that is, to write the data to the database in the following formats, perform the following operations:
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_table31723405471" frame="border" border="1" rules="all"><caption><b>Table 5 </b>Example</caption><thead align="left"><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row18172164024710"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.3.3.2.4.1.3.2.5.1.1"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p456274815477">ID</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.3.3.2.4.1.3.2.5.1.2"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1562184815473">SourceName</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="40%" id="mcps1.3.5.3.3.2.4.1.3.2.5.1.3"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1856214817475">SourceBooks</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.3.3.2.4.1.3.2.5.1.4"><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p15621148194720">MaxScore</p>
</th>
</tr>
</thead>
<tbody><tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row1417215407479"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p3317154044915">650612</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p61723402479">tom</p>
</td>
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1017211402473">["book1","book2","book3"]</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.4 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p21723408470">1.0</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row1817215400471"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1533434017495">650616</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p61721540174711">tom</p>
</td>
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p514618150496">["book1","book2","book3"]</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.4 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p617215403475">1.0</p>
</td>
</tr>
<tr id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_row71721540184719"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.1 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p2334940134915">650618</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.2 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p17172184012471">tom</p>
</td>
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.3 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p17162816184914">["book1","book2","book3"]</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.3.3.2.4.1.3.2.5.1.4 "><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1172740174715">1.0</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="p" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1142320158417">Set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname1329719438216"><b>File Format</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue13297104313210"><b>JSON</b></span>, <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname1429734314219"><b>JSON Type</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue8297164312216"><b>JSON object</b></span>, and <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname1729712430213"><b>JSON Reference Node</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue1029713433219"><b>hits.hits</b></span>, and then create a converter.<ol class="substepthirdol" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ol8423101594114"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li8423181584110">Click <span><img id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_image17131165625613" src="en-us_image_0000002234236140.png"></span> to add a field.<div class="fignone" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_fig118821833182616"><span class="figcap"><b>Figure 2 </b>Adding a field</span><br><span><img id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_image136204212494" src="en-us_image_0000002234076308.png" title="Click to enlarge" class="imgResize"></span></div>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li14928172484313">Click <span><img id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_image1313717134298" src="en-us_image_0000002269115513.png"></span> to create a converter for the new field.<div class="fignone" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_fig66941443192614"><span class="figcap"><b>Figure 3 </b>Creating a field converter</span><br><span><img id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_image19755114410263" src="en-us_image_0000002269115501.png" title="Click to enlarge" class="imgResize"></span></div>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1116315134312">Set <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b2201452144413">Converter</strong> to <span class="uicontrol" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_uicontrol120115234411"><b>Expression conversion</b></span>, enter <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b92035214410">"1.0"</strong> in the <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b921185218446">Expression</strong> text box, and click <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b122155234416">Save</strong>.<div class="fignone" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_fig168681459132613"><span class="figcap"><b>Figure 4 </b>Configuring a field converter</span><br><span><img id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_image1113381992916" src="en-us_image_0000002234076300.png" title="Click to enlarge" class="imgResize"></span></div>
</li></ol>
</div>
</li></ol>
</li></ol>
</div>
<div class="section" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section39751230182317"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section39751230182317"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section39751230182317"></a><h4 class="sectiontitle">Binary</h4><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p742318150418">If you want to copy files between file systems, you can select the binary format. Files can be transferred in binary format at a high speed and stable performance. In addition, field mapping is not required in the second step of the job.</p>
<ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul14206191811543"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1720616188545"><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b623783191820">Directory structure for file transfer</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p94231015124115">CDM can transfer a single file or all files in a directory at a time. After the files are transferred to the migration destination, the directory structure remains unchanged.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li750325185412"><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b1644395681918">Migrating incremental files</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p18423171512417">When you use CDM to transfer files in binary format, configure <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname22425372319"><b>Duplicate File Processing Method</b></span> at the migration destination for incremental file migration. For details, see <a href="dataartsstudio_01_0112.html">Incremental File Migration</a>.</p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p2423111594114">During incremental file migration, set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname19822721112416"><b>Duplicate File Processing Method</b></span> to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue1432214406249"><b>Skip</b></span>. If new files exist at the migration source or a failure occurs during the migration, run the job again, so that the migrated files will not be migrated repeatedly.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li11706833175418"><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b19981175511267">Write to Temporary File</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p34237156412">When migrating files in binary format, you can specify whether to write the files to a temporary file at the migration destination. If this parameter is specified, the file is written to a temporary file during file replication. After the file is successfully migrated, run the <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b1234421243015">rename</strong> or <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b19312614193012">move</strong> command to restore the file at the migration destination.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li8800183814541"><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b13784162014323">Generate MD5 Hash Value</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p24232151419">An MD5 hash value is generated for each transferred file, and the value is recorded in a new <span class="uicontrol" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_uicontrol10862821203220"><b>.md5</b></span> file. You can specify the directory where the MD5 value is generated.</p>
</li></ul>
</div>
<div class="section" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section148391468257"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section148391468257"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section148391468257"></a><h4 class="sectiontitle">Common parameters</h4><ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul173242208339"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li8323124163318"><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b188120362341">Start Job by Marker File</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p13423121513419">In automation scenarios, a scheduled task is configured on CDM to periodically read files from the migration source. However, files are being generated at the migration source. As a result, CDM reads data repeatedly or fails to read data from the migration source. You can specify the marker file for starting a job as <span class="uicontrol" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_uicontrol1454217593717"><b>ok.txt</b></span> in the job parameters of the migration source. After the file is successfully generated at the migration source, the <span class="uicontrol" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_uicontrol35731543113715"><b>ok.txt</b></span> file is generated in the file directory. In this way, CDM can read the complete file.</p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1210551211456">In addition, you can set the suspension period. Within the suspension period, CDM periodically queries whether the marker file exists. If the file does not exist after the suspension period expires, the job fails.</p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p151051312134518">The marker file will not be migrated.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1266819209465"><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b91381413184011">Job Success Marker File</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1365325224613">After data is successfully migrated to a file system, an empty file is generated in the destination directory. You can specify the file name. Generally, this parameter is used together with <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname12673192144314"><b>Start Job by Marker File</b></span>.</p>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p11653115215465">The name of the job success marker file cannot be the same as that of the transferred file, for example, finish.txt. If the two files have the same name, they will overwrite each other.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li140132843310"><strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_a7662e8b8eaec403eb65ca04f5aa281d3">Filter</strong><p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p17553945105615">When using CDM to migrate files, you can specify a filter to filter files. Files can be filtered by wildcard character or time filter.</p>
<ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul118979475710"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li10584180579">If you select <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b17938111412">Wildcard</strong>, CDM migrates only the paths or files that meet the filter condition.</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li69121045719">If you select <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b16197175394111">Time Filter</strong>, CDM migrates only the files modified after the specified time point.</li></ul>
<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p52251628185718">For example, the <span class="filepath" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_filepath98211585448"><b>/table/</b></span> directory stores a large number of data table directories divided by day. <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b18831258114412">DRIVING_BEHAVIOR_20180101</strong> to <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b178314588447">DRIVING_BEHAVIOR_20180630</strong> store all data of <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b1683105884416">DRIVING_BEHAVIOR</strong> from January to June. If you only want to migrate the table data of <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b11189622105119">DRIVING_BEHAVIOR</strong> in March, set the source directory to <span class="filepath" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_filepath787422458"><b>/table</b></span>, filter type to wildcard, and path filter to <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b8818135785215">DRIVING_BEHAVIOR_201803*</strong>.</p>
</li></ul>
</div>
<div class="section" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section5636174252511"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section5636174252511"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_section5636174252511"></a><h4 class="sectiontitle">Solutions to File Format Problems</h4><ol id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ol1149125213611"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1249165216612">When data in a database is exported to a CSV file, if the data contains commas (,), the data in the exported CSV file is disordered.<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p942316157414"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1249165216612"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1249165216612"></a>The following solutions are available:</p>
<ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul42010166391"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li92011416123910">Specify a field delimiter.<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p1842321513414"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li92011416123910"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li92011416123910"></a>Use a character that does not exist in the database or a rare non-printable character as the field delimiter. For example, you can set <span class="parmname" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmname5241144511543"><b>Field Delimiter</b></span> at the destination to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue12242184514547"><b>%01</b></span>. In this way, the exported field delimiter is <span class="uicontrol" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_uicontrol924216457547"><b>\u0001</b></span>. For details, see <a href="#dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_table11215204115">Table 1</a>.</p>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li14201121603920">Use a quote character.<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p13423615114112"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li14201121603920"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li14201121603920"></a>Set <span class="uicontrol" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_uicontrol35419293013"><b>Use Quote Character</b></span> to <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b102126351107">Yes</strong> at the migration destination. In this way, if the field in the database contains the field delimiter, CDM quotes the field using the quote character and write the field as a whole to the CSV file.</p>
</li></ul>
</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1495532213713">The data in the database contains line separators.<ul id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_ul1941415218398"><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li184144211393">Scenario: When you use CDM to export a table in the MySQL database (a field value contains the line separator <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b197481413839">\n</strong>) to a CSV file, and then use CDM to import the exported CSV file to MRS HBase, data in the exported CSV file is truncated.</li><li id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1387142453912">Solution: Specify a line separator.<p id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_p468663115391"><a name="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1387142453912"></a><a name="en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_li1387142453912"></a>When you use CDM to export MySQL table data to a CSV file, set <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b9767205918519">Line Separator</strong> at the migration destination to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue189233613611"><b>%01</b></span> (ensure that the value does not appear in the field value). In this way, the line separator in the exported CSV file is <span class="uicontrol" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_uicontrol191600171479"><b>%01</b></span>. Then use CDM to import the CSV file to MRS HBase. Set <strong id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_b201454527820">Line Separator</strong> at the migration source to <span class="parmvalue" id="dataartsstudio_01_0108__en-us_topic_0000001151779460_en-us_topic_0000001151619244_en-us_topic_0123434186_parmvalue766114810812"><b>%01</b></span>. This avoids data truncation.</p>
</li></ul>
</li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0110.html">Key Operation Guide</a></div>
</div>
</div>
<script language="JavaScript">
<!--
initImageViewer('.imgResize');
var msg_imageMax = "view original image";
var msg_imageClose = "close";
//--></script>
View Git Blame Copy Permalink