forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
69 lines
12 KiB
HTML
69 lines
12 KiB
HTML
<a name="dataartsstudio_01_0076"></a><a name="dataartsstudio_01_0076"></a>
|
|
|
|
<h1 class="topictitle1">Performance Tuning</h1>
|
|
<div id="body8662426"><div class="section" id="dataartsstudio_01_0076__en-us_topic_0000001287646722_section10348239165916"><h4 class="sectiontitle">Overview</h4><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p1581214497594">In addition to increasing the source read speed, improving the destination write performance, and increasing the bandwidth, you can accelerate migration using the following methods:</p>
|
|
<ul id="dataartsstudio_01_0076__en-us_topic_0000001287646722_ul191773617334"><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li926242161719"><strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b2026362113170">Use a CDM cluster of higher specifications</strong><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p168895222172">The NIC bandwidth and maximum number of concurrent extractors vary depending on the CDM cluster specifications. If you want to migrate data faster, or the metrics of your CDM cluster (such as the CPU usage, disk usage, and memory usage) are often high, you may need a CDM cluster with higher specifications for data migration.</p>
|
|
</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li98671554203415"><strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b565515308179">Use multiple CDM clusters</strong><div class="p" id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p10730194103517">In some scenarios, you are advised to use multiple CDM clusters to share workloads to improve migration efficiency and stability. The following are some examples:<ul id="dataartsstudio_01_0076__en-us_topic_0000001287646722_ul1273024113512"><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li77307420359">Multiple CDM clusters are required for different purposes or by multiple business departments. For example, you may need one CDM cluster for running data migration jobs and another one as an agent for <span id="dataartsstudio_01_0076__en-us_topic_0000001287646722_text390431313817">DataArts Studio</span> Management Center.</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li673113414358">You want to migrate a large number of tables. In this case, you can use multiple CDM clusters to run jobs simultaneously to improve migration efficiency.</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li57312412354">The CPU usage, disk usage, and memory usage of the in-use CDM cluster are often high. In this case, you are advised to use multiple CDM clusters to shared workloads.</li></ul>
|
|
</div>
|
|
</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li1033794115912"><strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b5508153614175">Avoid running too many CDM jobs simultaneously</strong><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p11757920175911">If the number of CDM jobs that run concurrently exceeds the maximum concurrent extractors for the CDM cluster, some jobs will be queued, and the migration will be prolonged.</p>
|
|
<p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p6227171545913">Avoid running too many jobs simultaneously, which may cause slow migration due to insufficient resources.</p>
|
|
</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li246655171719"><strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b15252849181918">Change concurrent extractors</strong><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p26087149186">If the number of tasks is small, adjusting the number of concurrent extractors is the best way to improve performance. You can set the number of concurrent extractors for a job and the maximum number of concurrent extractors for a cluster.</p>
|
|
<div class="p" id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p5221242182218">CDM migrates data through data migration jobs. It works in the following way:<ol id="dataartsstudio_01_0076__en-us_topic_0000001287646722_en-us_topic_0000001340006573_ol1095184992318"><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_en-us_topic_0000001340006573_li79541847152411">When data migration jobs are submitted, CDM splits each job into multiple tasks based on the <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_en-us_topic_0000001340006573_b49451957175511">Concurrent Extractors</strong> parameter in the job configuration.<div class="note" id="dataartsstudio_01_0076__en-us_topic_0000001287646722_en-us_topic_0000001340006573_note636713355310"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_en-us_topic_0000001340006573_p1036843195312">Jobs for different data sources may be split based on different dimensions. Some jobs may not be split based on the <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_en-us_topic_0000001340006573_b258662735711">Concurrent Extractors</strong> parameter.</p>
|
|
</div></div>
|
|
</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_en-us_topic_0000001340006573_li587861564119">CDM submits the tasks to the running pool in sequence. Tasks (defined by <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_en-us_topic_0000001340006573_b169912243">Maximum Concurrent Extractors</strong>) run concurrently. Excess tasks are queued.</li></ol>
|
|
</div>
|
|
<p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p13951195014593">By setting appropriate values for parameters <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b459818118125">Concurrent Extractors</strong> and <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b19751182921119">Maximum Concurrent Extractors</strong>, you can accelerate migration. For details about how to change <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b1250910910133">Concurrent Extractors</strong>, see <a href="#dataartsstudio_01_0076__en-us_topic_0000001287646722_section6623192417013">Changing Concurrent Extractors</a>.</p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0076__en-us_topic_0000001287646722_section6623192417013"><a name="dataartsstudio_01_0076__en-us_topic_0000001287646722_section6623192417013"></a><a name="en-us_topic_0000001287646722_section6623192417013"></a><h4 class="sectiontitle">Changing Concurrent Extractors</h4><ol id="dataartsstudio_01_0076__en-us_topic_0000001287646722_ol2522956103216"><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li19522105683217">The maximum number of concurrent extractors for a cluster varies depending on the CDM cluster flavor. You are advised to set the maximum number of concurrent extractors to twice the number of vCPUs of the CDM cluster.
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0076__en-us_topic_0000001287646722_table1992816477328" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Maximum number of concurrent extractors for a CDM cluster</caption><thead align="left"><tr id="dataartsstudio_01_0076__en-us_topic_0000001287646722_row5928154743218"><th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.2.2.1.1.2.4.1.1"><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p109281147173219">Flavor</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.2.2.1.1.2.4.1.2"><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p7928247133214">vCPUs/Memory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.2.2.1.1.2.4.1.3"><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p392819472325">Maximum Concurrent Extractors</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0076__en-us_topic_0000001287646722_row19928247193213"><td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.1 "><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p79285478323">cdm.large</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.2 "><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p12928104793216">8 vCPUs, 16 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.3 "><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p19281247133218">16</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0076__en-us_topic_0000001287646722_row19928347123214"><td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.1 "><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p1792812479323">cdm.xlarge</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.2 "><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p49281847103210">16 vCPUs, 32 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.3 "><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p29281447203210">32</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0076__en-us_topic_0000001287646722_row7928154753217"><td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.1 "><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p19928174723213">cdm.4xlarge</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.2 "><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p19928164753217">64 vCPUs, 128 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.2.2.1.1.2.4.1.3 "><p id="dataartsstudio_01_0076__en-us_topic_0000001287646722_p12928447143210">128</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="fignone" id="dataartsstudio_01_0076__en-us_topic_0000001287646722_fig10797427201"><span class="figcap"><b>Figure 1 </b>Setting Maximum Concurrent Extractors for a CDM cluster</span><br><span><img id="dataartsstudio_01_0076__en-us_topic_0000001287646722_image1737265619202" src="en-us_image_0000002269116797.png" title="Click to enlarge" class="imgResize"></span></div>
|
|
</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li1252215616327">Configure the number of concurrent extractors based on the following rules:<ol type="a" id="dataartsstudio_01_0076__en-us_topic_0000001287646722_ol195931755163315"><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li852255615323">When data is to be migrated to files, CDM does not support multiple concurrent tasks. In this case, set a single process to extract data.</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li12522145616329">If each row of the table contains less than or equal to 1 MB data, data can be extracted concurrently. If each row contains more than 1 MB data, it is recommended that data be extracted in a single thread.</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li13833942349">Set <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b16578109101816">Concurrent Extractors</strong> for a job based on <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b1648672014182">Maximum Concurrent Extractors</strong> for the cluster. It is recommended that <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b317975281813">Concurrent Extractors</strong> is less than <strong id="dataartsstudio_01_0076__en-us_topic_0000001287646722_b148781559111818">Maximum Concurrent Extractors</strong>.</li><li id="dataartsstudio_01_0076__en-us_topic_0000001287646722_li97288231260">If the destination is DLI, you are advised to set the number of concurrent extractors to 1. Otherwise, data may fail to be written.</li></ol>
|
|
<div class="fignone" id="dataartsstudio_01_0076__en-us_topic_0000001287646722_fig165851910185417"><span class="figcap"><b>Figure 2 </b>Setting Concurrent Extractors for a job</span><br><span><img id="dataartsstudio_01_0076__en-us_topic_0000001287646722_image106111841143" src="en-us_image_0000002234077588.png" title="Click to enlarge" class="imgResize"></span></div>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0018.html">Improving Migration Performance</a></div>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<script language="JavaScript">
|
|
<!--
|
|
initImageViewer('.imgResize');
|
|
var msg_imageMax = "view original image";
|
|
var msg_imageClose = "close";
|
|
//--></script> |