forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
121 lines
20 KiB
HTML
121 lines
20 KiB
HTML
<a name="dataartsstudio_01_0083"></a><a name="dataartsstudio_01_0083"></a>
|
|
|
|
<h1 class="topictitle1">Managing CDM Job Configuration</h1>
|
|
<div id="body8662426"><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p17414711172013">On the <span class="uicontrol" id="dataartsstudio_01_0083__en-us_topic_0173586861_uicontrol3594822416"><b>Settings</b></span> tab page, you can perform the following operations:</p>
|
|
<ul id="dataartsstudio_01_0083__en-us_topic_0173586861_ul1760181119217"><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li435163719516"><a href="#dataartsstudio_01_0083__en-us_topic_0173586861_section19611105617510">Maximum Concurrent Extractors</a></li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li194508254219"><a href="#dataartsstudio_01_0083__en-us_topic_0173586861_section11184152932110">Scheduled Backup/Restoration</a></li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li18761311122111"><a href="#dataartsstudio_01_0083__en-us_topic_0173586861_section10589151615203">Environment Variables of Job Parameters</a></li></ul>
|
|
<div class="section" id="dataartsstudio_01_0083__en-us_topic_0173586861_section19611105617510"><a name="dataartsstudio_01_0083__en-us_topic_0173586861_section19611105617510"></a><a name="en-us_topic_0173586861_section19611105617510"></a><h4 class="sectiontitle">Maximum Concurrent Extractors</h4><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p139651336123418">Maximum number of concurrent extraction tasks in a cluster</p>
|
|
<div class="note" id="dataartsstudio_01_0083__en-us_topic_0173586861_note47231345151219"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p14723204510121">This parameter is also available on the <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b16999173515361">Cluster Configuration</strong> page. You can change its value either on this page or the <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b8999335203610">Cluster Configuration</strong> page.</p>
|
|
</div></div>
|
|
<div class="p" id="dataartsstudio_01_0083__en-us_topic_0173586861_p178363436311">CDM migrates data through data migration jobs. It works in the following way:<ol id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001340006573_ol1095184992318"><li id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001340006573_li79541847152411">When data migration jobs are submitted, CDM splits each job into multiple tasks based on the <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001340006573_b49451957175511">Concurrent Extractors</strong> parameter in the job configuration.<div class="note" id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001340006573_note636713355310"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001340006573_p1036843195312">Jobs for different data sources may be split based on different dimensions. Some jobs may not be split based on the <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001340006573_b258662735711">Concurrent Extractors</strong> parameter.</p>
|
|
</div></div>
|
|
</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001340006573_li587861564119">CDM submits the tasks to the running pool in sequence. Tasks (defined by <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001340006573_b169912243">Maximum Concurrent Extractors</strong>) run concurrently. Excess tasks are queued.</li></ol>
|
|
</div>
|
|
<p id="dataartsstudio_01_0083__en-us_topic_0173586861_p43731113344">By setting appropriate values for the <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b884281514313">Concurrent Extractors</strong> and <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b113662210397">Maximum Concurrent Extractors</strong> parameters, you can accelerate migration.</p>
|
|
<ol id="dataartsstudio_01_0083__en-us_topic_0173586861_ol8645152312331"><li id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_li19522105683217">You are advised to set <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b25701429174215">Maximum Concurrent Extractors</strong> to twice the number of vCPUs. For details, see <a href="#dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_table1992816477328">Table 1</a>.
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_table1992816477328"></a><a name="en-us_topic_0173586861_en-us_topic_0000001287646722_table1992816477328"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_table1992816477328" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Recommended maximum number of concurrent extractors for a CDM cluster</caption><thead align="left"><tr id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_row5928154743218"><th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.3.6.1.3.2.4.1.1"><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p109281147173219">Flavor</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.3.6.1.3.2.4.1.2"><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p7928247133214">vCPUs/Memory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.333333333333336%" id="mcps1.3.3.6.1.3.2.4.1.3"><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p392819472325">Recommended Maximum Concurrent Extractors</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_row19928247193213"><td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.3.6.1.3.2.4.1.1 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p79285478323">cdm.large</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.3.6.1.3.2.4.1.2 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p12928104793216">8 vCPUs, 16 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.3.6.1.3.2.4.1.3 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p19281247133218">16</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_row19928347123214"><td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.3.6.1.3.2.4.1.1 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p1792812479323">cdm.xlarge</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.3.6.1.3.2.4.1.2 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p49281847103210">16 vCPUs, 32 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.3.6.1.3.2.4.1.3 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p29281447203210">32</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_row7928154753217"><td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.3.6.1.3.2.4.1.1 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p19928174723213">cdm.4xlarge</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.3.6.1.3.2.4.1.2 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p19928164753217">64 vCPUs, 128 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.333333333333336%" headers="mcps1.3.3.6.1.3.2.4.1.3 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_p12928447143210">128</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_li1252215616327">Configure the number of concurrent extractors based on the following rules:<ol type="a" id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_ol195931755163315"><li id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_li852255615323">When data is to be migrated to files, CDM does not support multiple concurrent tasks. In this case, set a single process to extract data.</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_li12522145616329">If each row of the table contains less than or equal to 1 MB data, data can be extracted concurrently. If each row contains more than 1 MB data, it is recommended that data be extracted in a single thread.</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_li13833942349">Set <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b1324393917449">Concurrent Extractors</strong> for a job based on <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b724483944413">Maximum Concurrent Extractors</strong> for the cluster. It is recommended that the value of <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b182441539194418">Concurrent Extractors</strong> is less than that of <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b142452039114419">Maximum Concurrent Extractors</strong>.</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li16926101543616">If the migration source is Hive and JDBC is used to read data, CDM does not support multi-concurrency. In this case, set the number of concurrent extractors to 1.</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_en-us_topic_0000001287646722_li97288231260">If the destination is DLI, you are advised to set the number of concurrent extractors to 1. Otherwise, data may fail to be written.</li></ol>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0083__en-us_topic_0173586861_section11184152932110"><a name="dataartsstudio_01_0083__en-us_topic_0173586861_section11184152932110"></a><a name="en-us_topic_0173586861_section11184152932110"></a><h4 class="sectiontitle">Scheduled Backup/Restoration</h4><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p98104311150">This function depends on the OBS service. Backup files cannot be automatically aged. You need to manually delete backup files on a regular basis.</p>
|
|
<ul id="dataartsstudio_01_0083__en-us_topic_0173586861_ul11483644193610"><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li1248318441368">Prerequisites<p id="dataartsstudio_01_0083__en-us_topic_0173586861_p52856063719"><a name="dataartsstudio_01_0083__en-us_topic_0173586861_li1248318441368"></a><a name="en-us_topic_0173586861_li1248318441368"></a>An OBS link has been created. For details, see <a href="dataartsstudio_01_0045.html">OBS Link Parameters</a>.</p>
|
|
</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li92584594375">Scheduled backup<p id="dataartsstudio_01_0083__en-us_topic_0173586861_p196448710388"><a name="dataartsstudio_01_0083__en-us_topic_0173586861_li92584594375"></a><a name="en-us_topic_0173586861_li92584594375"></a>On the <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b7512101717719">Job Management</strong> page, click <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b751316171074">Settings</strong> and configure <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b12897154889">Scheduled Backup</strong> and its related parameters.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0083__en-us_topic_0173586861_table1171092714397" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Scheduled backup parameters</caption><thead align="left"><tr id="dataartsstudio_01_0083__en-us_topic_0173586861_row197111127123910"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.4.3.2.2.2.4.1.1"><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p571112276391">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="60%" id="mcps1.3.4.3.2.2.2.4.1.2"><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p107111127113910">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="15%" id="mcps1.3.4.3.2.2.2.4.1.3"><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p7711527133912">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0083__en-us_topic_0173586861_row1871112717396"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.3.2.2.2.4.1.1 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p197111627143917">Scheduled Backup</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.4.3.2.2.2.4.1.2 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p0711142714395">Whether to enable automatic backup. This function is used to back up jobs but not links.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.4.3.2.2.2.4.1.3 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p7711172783913">Enable</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0083__en-us_topic_0173586861_row10711132712394"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.3.2.2.2.4.1.1 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p1671117278391">Backup Policy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.4.3.2.2.2.4.1.2 "><ul id="dataartsstudio_01_0083__en-us_topic_0173586861_ul92471011164220"><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li1224701194210"><strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b198462614105">All jobs</strong>: CDM backs up all table/file migration jobs and entire DB migration jobs regardless of the job statuses. However, historical jobs are not backed up.</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li2900141414218"><strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b942633116119">All jobs by groups</strong>: You select one or more job groups to back up.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.4.3.2.2.2.4.1.3 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p87115274391">All jobs</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0083__en-us_topic_0173586861_row1071112714399"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.3.2.2.2.4.1.1 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p15711172715393">Backup Cycle</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.4.3.2.2.2.4.1.2 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p81294539536">Select the backup cycle.</p>
|
|
<ul id="dataartsstudio_01_0083__en-us_topic_0173586861_ul1653311217541"><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li1987720317542"><strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b5268123216134">Day</strong>: The backup is performed daily at 00:00:00.</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li66165105411"><strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b6890936171317">Week</strong>: The backup is performed at 00:00:00 every Monday.</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li185334235410"><strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b110313404134">Month</strong>: The backup is performed at 00:00:00 on the first day of each month.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.4.3.2.2.2.4.1.3 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p571117272397">Day</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0083__en-us_topic_0173586861_row167111627113919"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.3.2.2.2.4.1.1 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p47111527173910">OBS Link for Writing Backups</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.4.3.2.2.2.4.1.2 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p43395323919">Link used to back up jobs to OBS buckets. Select a link you have created on the <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b36057376371">Links</strong> page. </p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.4.3.2.2.2.4.1.3 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p13711427183914">obslink</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0083__en-us_topic_0173586861_row157111827183917"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.3.2.2.2.4.1.1 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p3711827153911">OBS Bucket</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.4.3.2.2.2.4.1.2 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p117111127143910">OBS bucket where backup files are stored </p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.4.3.2.2.2.4.1.3 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p197111274394">cdm</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0083__en-us_topic_0173586861_row1773141612417"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.3.2.2.2.4.1.1 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p6774171614418">Backup Data Directory</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60%" headers="mcps1.3.4.3.2.2.2.4.1.2 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p67741316154117">Directory where backup files are stored </p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.4.3.2.2.2.4.1.3 "><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p8732759143520">/cdm-bk/</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="dataartsstudio_01_0083__en-us_topic_0173586861_li8322121412565">Restoring jobs<p id="dataartsstudio_01_0083__en-us_topic_0173586861_p1794517231567"><a name="dataartsstudio_01_0083__en-us_topic_0173586861_li8322121412565"></a><a name="en-us_topic_0173586861_li8322121412565"></a>If automatic backup has been performed, the backup list is displayed on the <span class="uicontrol" id="dataartsstudio_01_0083__en-us_topic_0173586861_uicontrol9401611172914"><b>Configuration Management</b></span> tab page. The OBS buckets where the backup files reside, backup paths, and backup time are displayed.</p>
|
|
<p id="dataartsstudio_01_0083__en-us_topic_0173586861_p1894093611466">You can click <span class="uicontrol" id="dataartsstudio_01_0083__en-us_topic_0173586861_uicontrol132182016314"><b>Restore Backup</b></span> in the <strong id="dataartsstudio_01_0083__en-us_topic_0173586861_b1932818579313">Operation</strong> column of the backup list to restore the CDM jobs.</p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0083__en-us_topic_0173586861_section10589151615203"><a name="dataartsstudio_01_0083__en-us_topic_0173586861_section10589151615203"></a><a name="en-us_topic_0173586861_section10589151615203"></a><h4 class="sectiontitle">Environment Variables of Job Parameters</h4><p id="dataartsstudio_01_0083__en-us_topic_0173586861_p79331334414">When creating a migration job on CDM, the parameter (such as the OBS bucket name or file path) that can be manually configured, a field in a parameter, or a character in a field can be configured as a global variable, so that you can change parameter values in batches, or batch replace certain characters after jobs are exported or imported.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0081.html">Creating a Job in a CDM Cluster</a></div>
|
|
</div>
|
|
</div>
|
|
|