forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
83 lines
12 KiB
HTML
83 lines
12 KiB
HTML
<a name="dataartsstudio_01_0025"></a><a name="dataartsstudio_01_0025"></a>
|
|
|
|
<h1 class="topictitle1">How Migration Jobs Work</h1>
|
|
<div id="body8662426"><div class="section" id="dataartsstudio_01_0025__en-us_topic_0000001340006573_section14018854920"><h4 class="sectiontitle">Data Migration Model</h4><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p111321034164314"><a href="#dataartsstudio_01_0025__en-us_topic_0000001340006573_fig165851910185417">Figure 1</a> shows the simplified migration model used by CDM.</p>
|
|
<div class="fignone" id="dataartsstudio_01_0025__en-us_topic_0000001340006573_fig165851910185417"><a name="dataartsstudio_01_0025__en-us_topic_0000001340006573_fig165851910185417"></a><a name="en-us_topic_0000001340006573_fig165851910185417"></a><span class="figcap"><b>Figure 1 </b>Migration model used by CDM</span><br><span><img class="imgResize" id="dataartsstudio_01_0025__en-us_topic_0000001340006573_image8585010145411" src="en-us_image_0000002269117481.png" title="Click to enlarge"></span></div>
|
|
<div class="p" id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p731010268319">CDM migrates data through data migration jobs. It works in the following way:<ol id="dataartsstudio_01_0025__en-us_topic_0000001340006573_ol1095184992318"><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li79541847152411">When data migration jobs are submitted, CDM splits each job into multiple tasks based on the <strong id="dataartsstudio_01_0025__en-us_topic_0000001340006573_b49451957175511">Concurrent Extractors</strong> parameter in the job configuration.<div class="note" id="dataartsstudio_01_0025__en-us_topic_0000001340006573_note636713355310"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1036843195312">Jobs for different data sources may be split based on different dimensions. Some jobs may not be split based on the <strong id="dataartsstudio_01_0025__en-us_topic_0000001340006573_b258662735711">Concurrent Extractors</strong> parameter.</p>
|
|
</div></div>
|
|
</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li587861564119">CDM submits the tasks to the running pool in sequence. Tasks (defined by <strong id="dataartsstudio_01_0025__en-us_topic_0000001340006573_b169912243">Maximum Concurrent Extractors</strong>) run concurrently. Excess tasks are queued.</li></ol>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0025__en-us_topic_0000001340006573_section649414854011"><h4 class="sectiontitle">Factors Affecting Migration Performance</h4><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p3193738202710">According to the migration model, the migration speed is affected by factors such as the source read speed, network bandwidth, destination write performance, and CDM cluster and job configuration.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0025__en-us_topic_0000001340006573_table191783304012" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Factors affecting migration performance</caption><thead align="left"><tr id="dataartsstudio_01_0025__en-us_topic_0000001340006573_row189182033144015"><th align="left" class="cellrowborder" colspan="2" valign="top" id="mcps1.3.2.3.2.4.1.1"><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p724717129368">Factor</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" id="mcps1.3.2.3.2.4.1.2"><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p291813337406">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0025__en-us_topic_0000001340006573_row831183012361"><td class="cellrowborder" rowspan="4" valign="top" width="6.99%" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p18231931522">Service-related factors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.879999999999999%" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p43123053616">Concurrent extractors of a job</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.13000000000001%" headers="mcps1.3.2.3.2.4.1.2 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p19978185215266">The number of concurrent extractors can be set for a CDM job during the job creation.</p>
|
|
<p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p13951195014593">Setting a proper value for this parameter can effectively improve the migration speed. If the value is too small, migration will be too slow. If the value is too large, the migration job is overloaded and may fail.</p>
|
|
<ul id="dataartsstudio_01_0025__en-us_topic_0000001340006573_ul33201814102416"><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li635771819246">When data is to be migrated to files, CDM does not support multiple concurrent tasks. In this case, set a single process to extract data.</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li19524634132415">If each row of the table contains less than or equal to 1 MB data, data can be extracted concurrently. If each row contains more than 1 MB data, it is recommended that data be extracted in a single thread.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0025__en-us_topic_0000001340006573_row4918333124012"><td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p59181433134019">Maximum concurrent extractors of a cluster</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p97141862273">Setting a proper value for this parameter can effectively improve the migration speed. If the value is too small, migration will be too slow. If the value is too large, the source is overloaded and the system may be unstable.</p>
|
|
<p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1958883811461">The maximum concurrent extractors vary depending on the CDM cluster flavor. The upper limit is twice the number of vCPUs. The following are the maximum concurrent extractors of some flavors:</p>
|
|
<ul id="dataartsstudio_01_0025__en-us_topic_0000001340006573_ul1062511772518"><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li56258712519">cdm.large: 16</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li141087167257">cdm.xlarge: 32</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li1169617261258">cdm.4xlarge: 128</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0025__en-us_topic_0000001340006573_row14998141610377"><td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p5998151693711">Service model</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1614419297502">If the number of CDM jobs that run concurrently exceeds the maximum concurrent extractors for the CDM cluster, some jobs will be queued, and the migration will be prolonged.</p>
|
|
<p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1755711268587">Avoid running too many jobs simultaneously, which may cause slow migration due to insufficient resources.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0025__en-us_topic_0000001340006573_row977220313535"><td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p157721438531">Data model</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1138153133519">The migration speed is also affected by the data structure. The following are some examples:</p>
|
|
<ul id="dataartsstudio_01_0025__en-us_topic_0000001340006573_ul1041917412365"><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li17419144163617">The wider a table is and the more string types the table has, the slower the migration is.</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li141924118364">A large file is migrated more quickly than multiple small files whose total size is the same as the large file.</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li476165713441">The more content a message has and the higher bandwidth it uses, the less transactions per second (TPS) are.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0025__en-us_topic_0000001340006573_row4555145241818"><td class="cellrowborder" colspan="2" valign="top" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1781875918188">Source read speed</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.4.1.2 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p11818135951817">It depends on the performance of the data source at the source.</p>
|
|
<p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1381885913189">For details about how to increase the read speed, see the documents of data sources at the source.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0025__en-us_topic_0000001340006573_row3669175417187"><td class="cellrowborder" colspan="2" valign="top" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1381820597181">Network bandwidth</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.4.1.2 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p4818105981814">The CDM cluster can communicate with the data source through an intranet, public network VPN, NAT, or Direct Connect.</p>
|
|
<ul id="dataartsstudio_01_0025__en-us_topic_0000001340006573_ul158186599183"><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li1781812597181">If they communicate through an intranet, the network bandwidth varies depending on the CDM instance flavor.<ul id="dataartsstudio_01_0025__en-us_topic_0000001340006573_ul1481895911818"><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li7819959131817">For cdm.large instances, the baseline and maximum bandwidths of the CDM cluster NIC are 0.8 and 3 Gbit/s, respectively.</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li16819659161819">For cdm.xlarge instances, the baseline and maximum bandwidths of the CDM cluster NIC are 4 and 10 Gbit/s, respectively.</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li781925915188">For cdm.4xlarge instances, the baseline and maximum bandwidths of the CDM cluster NIC are 36 and 40 Gbit/s, respectively.</li></ul>
|
|
</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li78191859141816">If they communicate through the Internet, the network bandwidth is subject to the Internet bandwidth. The bandwidth for the CDM cluster depends on the EIP bound to the CDM cluster, and the bandwidth for the data source depends on the Internet bandwidth.</li><li id="dataartsstudio_01_0025__en-us_topic_0000001340006573_li38190592187">If they communicate through a VPN, NAT, or Direct Connect, the network bandwidth is subject to the VPN, NAT, or Direct Connect bandwidth.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0025__en-us_topic_0000001340006573_row10430750201819"><td class="cellrowborder" colspan="2" valign="top" headers="mcps1.3.2.3.2.4.1.1 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1781935951818">Destination write performance</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.2.3.2.4.1.2 "><p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p1981995951819">It depends on the performance of the data source at the destination.</p>
|
|
<p id="dataartsstudio_01_0025__en-us_topic_0000001340006573_p3819859171815">For details about how to improve the performance, see the documents of data sources at the destination.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0018.html">Improving Migration Performance</a></div>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<script language="JavaScript">
|
|
<!--
|
|
initImageViewer('.imgResize');
|
|
var msg_imageMax = "view original image";
|
|
var msg_imageClose = "close";
|
|
//--></script> |