forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
59 lines
8.9 KiB
HTML
59 lines
8.9 KiB
HTML
<a name="dataartsstudio_03_0336"></a><a name="dataartsstudio_03_0336"></a>
|
|
|
|
<h1 class="topictitle1">How Do I Set the Number of Concurrent Extractors for a CDM Migration Job?</h1>
|
|
<div id="body8662426"><div class="p" id="dataartsstudio_03_0336__en-us_topic_0000001225868959_p178363436311">CDM migrates data through data migration jobs. It works in the following way:<ol id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001340006573_ol1095184992318"><li id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001340006573_li79541847152411">When data migration jobs are submitted, CDM splits each job into multiple tasks based on the <strong id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001340006573_b49451957175511">Concurrent Extractors</strong> parameter in the job configuration.<div class="note" id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001340006573_note636713355310"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001340006573_p1036843195312">Jobs for different data sources may be split based on different dimensions. Some jobs may not be split based on the <strong id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001340006573_b258662735711">Concurrent Extractors</strong> parameter.</p>
|
|
</div></div>
|
|
</li><li id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001340006573_li587861564119">CDM submits the tasks to the running pool in sequence. Tasks (defined by <strong id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001340006573_b169912243">Maximum Concurrent Extractors</strong>) run concurrently. Excess tasks are queued.</li></ol>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_03_0336__en-us_topic_0000001225868959_section496243431219"><h4 class="sectiontitle">Changing Concurrent Extractors</h4><ol id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_ol2522956103216"><li id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_li19522105683217">The maximum number of concurrent extractors for a cluster varies depending on the CDM cluster flavor. You are advised to set the maximum number of concurrent extractors to twice the number of vCPUs of the CDM cluster.
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_table1992816477328" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Maximum number of concurrent extractors for a CDM cluster</caption><thead align="left"><tr id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_row5928154743218"><th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.2.2.1.1.2.4.1.1"><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p109281147173219">Flavor</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.2.2.1.1.2.4.1.2"><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p7928247133214">vCPUs/Memory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.2.2.1.1.2.4.1.3"><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p392819472325">Maximum Concurrent Extractors</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_row19928247193213"><td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.1 "><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p79285478323">cdm.large</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.2 "><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p12928104793216">8 vCPUs, 16 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.3 "><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p19281247133218">16</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_row19928347123214"><td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.1 "><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p1792812479323">cdm.xlarge</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.2 "><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p49281847103210">16 vCPUs, 32 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.3 "><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p29281447203210">32</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_row7928154753217"><td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.1 "><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p19928174723213">cdm.4xlarge</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.2 "><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p19928164753217">64 vCPUs, 128 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.3 "><p id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_p12928447143210">128</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="fignone" id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_fig10797427201"><span class="figcap"><b>Figure 1 </b>Setting Maximum Concurrent Extractors for a CDM cluster</span><br><span><img id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_image1737265619202" src="en-us_image_0000002269115561.png" title="Click to enlarge" class="imgResize"></span></div>
|
|
</li><li id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_li1252215616327">Configure the number of concurrent extractors based on the following rules:<ol type="a" id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_ol195931755163315"><li id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_li852255615323">When data is to be migrated to files, CDM does not support multiple concurrent tasks. In this case, set a single process to extract data.</li><li id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_li12522145616329">If each row of the table contains less than or equal to 1 MB data, data can be extracted concurrently. If each row contains more than 1 MB data, it is recommended that data be extracted in a single thread.</li><li id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_li13833942349">Set <strong id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_b16578109101816">Concurrent Extractors</strong> for a job based on <strong id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_b1648672014182">Maximum Concurrent Extractors</strong> for the cluster. It is recommended that <strong id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_b317975281813">Concurrent Extractors</strong> is less than <strong id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_b148781559111818">Maximum Concurrent Extractors</strong>.</li><li id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_li97288231260">If the destination is DLI, you are advised to set the number of concurrent extractors to 1. Otherwise, data may fail to be written.</li></ol>
|
|
<div class="fignone" id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_fig165851910185417"><span class="figcap"><b>Figure 2 </b>Setting Concurrent Extractors for a job</span><br><span><img id="dataartsstudio_03_0336__en-us_topic_0000001225868959_en-us_topic_0000001287646722_image106111841143" src="en-us_image_0000002234236192.png" title="Click to enlarge" class="imgResize"></span></div>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_03_0027.html">DataArts Migration (CDM Jobs)</a></div>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<script language="JavaScript">
|
|
<!--
|
|
initImageViewer('.imgResize');
|
|
var msg_imageMax = "view original image";
|
|
var msg_imageClose = "close";
|
|
//--></script> |