forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
274 lines
36 KiB
HTML
274 lines
36 KiB
HTML
<a name="dataartsstudio_01_0075"></a><a name="dataartsstudio_01_0075"></a>
|
|
|
|
<h1 class="topictitle1">Creating an Entire Database Migration Job</h1>
|
|
<div id="body8662426"><div class="section" id="dataartsstudio_01_0075__en-us_topic_0108275370_sf6122b57876a439ca9c6d77777c1dc48"><h4 class="sectiontitle">Scenario</h4><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p2084965832210">CDM supports entire DB migration between homogeneous and heterogeneous data sources. The migration principles are the same as those in <a href="dataartsstudio_01_0046.html">Table/File Migration Jobs</a>. Each type of Elasticsearch, each key prefix of Redis, or each collection of MongoDB can be executed concurrently as a subtask.</p>
|
|
<div class="note" id="dataartsstudio_01_0075__en-us_topic_0108275370_note479638185918"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p12856337183814">Each time an entire DB migration job is executed, its subtasks are recreated based on the configuration of the migration job. You cannot modify the subtasks and then run the migration job again.</p>
|
|
</div></div>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_a6a05a4793fa04de799e48799f0efc9bb"><a href="dataartsstudio_01_0215.html">Supported Data Sources</a> lists the data sources supporting entire database migration.</p>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0075__en-us_topic_0108275370_section10128104561720"><h4 class="sectiontitle">Constraints</h4><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p156411937717">Field names of the source and destination parameters cannot contain ampersands (&) or number signs (%).</p>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0075__en-us_topic_0108275370_sfbab7a6f896a46d1be21cf46c8a3b8c3"><h4 class="sectiontitle">Prerequisites</h4><ul id="dataartsstudio_01_0075__en-us_topic_0108275370_ul163144187205"><li id="dataartsstudio_01_0075__en-us_topic_0108275370_l0f09c327f05f45e0a7aa0f61f0fe8d35">A link has been created. For details, see <a href="dataartsstudio_01_0024.html">Creating a Link Between CDM and a Data Source</a>.</li></ul>
|
|
<ul id="dataartsstudio_01_0075__en-us_topic_0108275370_u18c5f21ca2024c828bd83b7762870046"><li id="dataartsstudio_01_0075__en-us_topic_0108275370_l5038f44ed94b4d5c92d5b6d53eced26e">The CDM cluster can communicate with the data source.</li></ul>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0075__en-us_topic_0108275370_s466f96b7ba414d42aa49f9a9cfeec1c7"><h4 class="sectiontitle">Procedure</h4><ol id="dataartsstudio_01_0075__en-us_topic_0108275370_od1b27a0a1cf748ac921a0b7a3274f193"><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li4336350316596"><span>Log in to the management console and choose <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b7988165210257">Service List</strong> > <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b109881252192511">Cloud Data Migration</strong>. In the left navigation pane, choose <span class="uicontrol" id="dataartsstudio_01_0075__en-us_topic_0108275370_uicontrol59895524257"><b>Cluster Management</b></span>. Locate the target cluster and click <span class="uicontrol" id="dataartsstudio_01_0075__en-us_topic_0108275370_uicontrol209891452142513"><b>Job Management</b></span>.</span></li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li5835621161046"><span>Choose <span class="menucascade" id="dataartsstudio_01_0075__en-us_topic_0108275370_menucascade1077211812106"><b><span class="uicontrol" id="dataartsstudio_01_0075__en-us_topic_0108275370_uicontrol167725189103">Entire DB Migration</span></b> > <b><span class="uicontrol" id="dataartsstudio_01_0075__en-us_topic_0108275370_uicontrol177721418171020">Create Job</span></b></span>. The page for configuring the job is displayed.</span></li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li41220264163159"><span>Configure the related parameters of the source database according to <a href="#dataartsstudio_01_0075__en-us_topic_0108275370_table14973632102118">Table 1</a>.</span><p>
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0075__en-us_topic_0108275370_table14973632102118"></a><a name="en-us_topic_0108275370_table14973632102118"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0075__en-us_topic_0108275370_table14973632102118" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row20487414102118"><th align="left" class="cellrowborder" valign="top" width="27%" id="mcps1.3.4.2.3.2.1.2.5.1.1"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p37157660102118">Source Database</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="24%" id="mcps1.3.4.2.3.2.1.2.5.1.2"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p56980450102118">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="32%" id="mcps1.3.4.2.3.2.1.2.5.1.3"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p52013737102118">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="17%" id="mcps1.3.4.2.3.2.1.2.5.1.4"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p34345634143143">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row54909470102118"><td class="cellrowborder" rowspan="3" valign="top" width="27%" headers="mcps1.3.4.2.3.2.1.2.5.1.1 "><ul id="dataartsstudio_01_0075__en-us_topic_0108275370_ul21600492111710"><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li17262144713274">DWS</li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li3578132955719">MySQL</li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li64566498111710">PostgreSQL</li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li958515012579">SQL Server</li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li1562215814816">Oracle</li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li1265211572276">SAP HANA</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="24%" headers="mcps1.3.4.2.3.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p17535481105031">Schema/Tablespace</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32%" headers="mcps1.3.4.2.3.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p52647489105159">Name of the schema or tablespace from which data will be extracted. This parameter is displayed when <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275324_b317022014371">Use SQL Statement</strong> is set to <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275324_b678752710376">No</strong>. Click the icon next to the text box to go to the page for selecting a schema or directly enter a schema or tablespace.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_p2497091311218">If the desired schema or tablespace is not displayed, check whether the login account has the permissions required to query metadata.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17%" headers="mcps1.3.4.2.3.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p30532988143143">schema</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row7145183822214"><td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p914512383227">WHERE Clause</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p26422060162021">WHERE clause used to specify the tables to be extracted. This parameter applies to all subtables in the entire DB migration. If this parameter is not set, the entire table is extracted. If the table to be migrated does not contain the fields specified by the WHERE clause, the migration will fail.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_p39057470101222">You can set a date macro variable to extract data generated on a specific date. </p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p131451638192210">age > 18 and age <= 60</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row669875135320"><td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1569816525313">Null in Partition Column</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1569865125311">Whether a partition field can be null</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p17698254534">Yes</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row8101152492010"><td class="cellrowborder" valign="top" width="27%" headers="mcps1.3.4.2.3.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p11103142419207">Hive</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="24%" headers="mcps1.3.4.2.3.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p11701133102012">Database Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32%" headers="mcps1.3.4.2.3.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1124173812207">Name of the database to be migrated. The user configured in the source link must have the permission to read the database.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17%" headers="mcps1.3.4.2.3.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p11241938172014">hivedb</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1324173862014"></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row689892910317"><td class="cellrowborder" rowspan="2" valign="top" width="27%" headers="mcps1.3.4.2.3.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p48980298311">HBase</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275392_p1839020576516">CloudTable</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="24%" headers="mcps1.3.4.2.3.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1489832917320">Start Time</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32%" headers="mcps1.3.4.2.3.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p3589134610277">Start time (included). The format is <em id="dataartsstudio_01_0075__en-us_topic_0108275370_i1202184414">yyyy-MM-dd hh:mm:ss</em>. The dateformat time macro variable function is supported.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_p162261102041">Examples: <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b142478501414">2017-12-31 20:00:00</strong>, <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b20445459134119">${dateformat(yyyy-MM-dd, -1, DAY)} 02:00:00</strong>, and <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b184286112428">${dateformat(yyyy-MM-dd HH:mm:ss, -1, DAY)}</strong></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17%" headers="mcps1.3.4.2.3.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p149447514255">"2017-12-31 20:00:00"</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row101613497313"><td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p19275195710320">End Time</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1511734202812">End time (excluded) The format is <em id="dataartsstudio_01_0075__en-us_topic_0108275370_i5136193714468">yyyy-MM-dd hh:mm:ss</em>. The dateformat time macro variable function is supported.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_p13161184912317">Examples: <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b33182405468">2018-01-01 20:00:00</strong>, <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b5318184034615">${dateformat(yyyy-MM-dd, -1, DAY)} 02:00:00</strong>, and <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b73198406461">${dateformat(yyyy-MM-dd HH:mm:ss, -1, DAY)}</strong></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1769741072819">"2018-01-01 20:00:00"</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row8272123132715"><td class="cellrowborder" valign="top" width="27%" headers="mcps1.3.4.2.3.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p18922131405213">Redis</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="24%" headers="mcps1.3.4.2.3.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p122721432272">Key Filter Character</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32%" headers="mcps1.3.4.2.3.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1235241722816">Filter character used to determine the keys to be migrated</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_p32722311271">For example, if the value of this parameter is <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b784317488241">a*</strong>, all asterisks (*) will be migrated.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17%" headers="mcps1.3.4.2.3.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1727211310275">a*</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row71301631104116"><td class="cellrowborder" rowspan="2" valign="top" width="27%" headers="mcps1.3.4.2.3.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p411810172712">DDS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="24%" headers="mcps1.3.4.2.3.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p613014311413">Database Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="32%" headers="mcps1.3.4.2.3.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p131301531134113">Name of the database from which data is to be migrated. The user configured in the source link must have the permission to read the database.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17%" headers="mcps1.3.4.2.3.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1130831184112">ddsdb</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row123132291558"><td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p695453818511">Query Filter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1934683214289">Filter used to match documents.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_p331302920519">Example: <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b15421917546">{HTTPStatusCode:{$gt:"400",$lt:"500"},HTTPMethod:"GET"}</strong></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.3.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p531310297511">-</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li543483211856"><span>Configure the related parameters, from <a href="#dataartsstudio_01_0075__en-us_topic_0108275370_table17764626124115">Table 2</a>, for the destination cloud service.</span><p>
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0075__en-us_topic_0108275370_table17764626124115"></a><a name="en-us_topic_0108275370_table17764626124115"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0075__en-us_topic_0108275370_table17764626124115" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Destination job parameters</caption><thead align="left"><tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row176419264415"><th align="left" class="cellrowborder" valign="top" width="22.900000000000002%" id="mcps1.3.4.2.4.2.1.2.5.1.1"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p17644263413">Destination Database</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="17.61%" id="mcps1.3.4.2.4.2.1.2.5.1.2"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p8765172664113">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="46.52%" id="mcps1.3.4.2.4.2.1.2.5.1.3"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p127651926164117">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="12.97%" id="mcps1.3.4.2.4.2.1.2.5.1.4"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p47651226144115">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row117651926154112"><td class="cellrowborder" valign="top" width="22.900000000000002%" headers="mcps1.3.4.2.4.2.1.2.5.1.1 "><ul id="dataartsstudio_01_0075__en-us_topic_0108275370_ul13765426174118"><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li16765226124118">RDS for MySQL</li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li17659266414">RDS for PostgreSQL</li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li15765142604111">RDS for SQL Server</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.61%" headers="mcps1.3.4.2.4.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p16879184202717">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.52%" headers="mcps1.3.4.2.4.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p78791412278">For details about the destination job parameters required for entire DB migration to an RDS database, see <a href="dataartsstudio_01_0068.html">To MySQL/SQL Server/PostgreSQL</a>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.97%" headers="mcps1.3.4.2.4.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p77651268417">schema</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row1152119514156"><td class="cellrowborder" valign="top" width="22.900000000000002%" headers="mcps1.3.4.2.4.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p145221255157">DWS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.61%" headers="mcps1.3.4.2.4.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p152218531512">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.52%" headers="mcps1.3.4.2.4.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p2794111610150">For details about the destination job parameters required for entire DB migration to DWS, see <a href="dataartsstudio_01_1251.html">To DWS</a>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.97%" headers="mcps1.3.4.2.4.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p2522257154">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row137651926124113"><td class="cellrowborder" valign="top" width="22.900000000000002%" headers="mcps1.3.4.2.4.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p17765426154118">MRS Hive</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.61%" headers="mcps1.3.4.2.4.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p48691610192616">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.52%" headers="mcps1.3.4.2.4.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p386991032618">For details about the destination job parameters required for entire DB migration to MRS HIVE, see <a href="dataartsstudio_01_0066.html">To Hive</a>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.97%" headers="mcps1.3.4.2.4.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1276517265413">hivedb</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row11765126184113"><td class="cellrowborder" valign="top" width="22.900000000000002%" headers="mcps1.3.4.2.4.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p147658267411">MRS HBase</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1876552674112">CloudTable</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.61%" headers="mcps1.3.4.2.4.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p34959192252">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.52%" headers="mcps1.3.4.2.4.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p3495519142513">For details about the destination job parameters required for entire DB migration to MRS HBase or CloudTable, see <a href="dataartsstudio_01_0064.html">To HBase/CloudTable</a>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.97%" headers="mcps1.3.4.2.4.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1784219243165">Yes</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row7765192644117"><td class="cellrowborder" valign="top" width="22.900000000000002%" headers="mcps1.3.4.2.4.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p13557145155213">Redis</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.61%" headers="mcps1.3.4.2.4.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p937203082210">Clear Database</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.52%" headers="mcps1.3.4.2.4.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p43733307222">Clears the database data before data import.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.97%" headers="mcps1.3.4.2.4.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p7765326124116">Yes</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row6765112614116"><td class="cellrowborder" rowspan="2" valign="top" width="22.900000000000002%" headers="mcps1.3.4.2.4.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p10765182664114">DDS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.61%" headers="mcps1.3.4.2.4.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p15765126154117">Database Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.52%" headers="mcps1.3.4.2.4.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p576532664117">Name of the database from which data is to be migrated. The user configured in the source link must have the permission to read the database.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.97%" headers="mcps1.3.4.2.4.2.1.2.5.1.4 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p17765726174114">mongodb</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row876542619416"><td class="cellrowborder" valign="top" headers="mcps1.3.4.2.4.2.1.2.5.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1407173731318">Migration Behavior</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.4.2.1.2.5.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p4765112611414">Select <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b7401112303115">Add</strong> or <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b567412516313">Replace</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.4.2.4.2.1.2.5.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p11765152624114">-</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li1852210255336"><span>If you are migrating an entire relational database, click <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b102664571226">Next</strong> after configuring job parameters to select source and destination tables. Ensure that the destination table names are the same as the source table names. For example, if the source table name is <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b1661118249247">test</strong>, the destination table name must also be <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b632104116243">test</strong>.</span><p><div class="fignone" id="dataartsstudio_01_0075__en-us_topic_0108275370_fig851592533319"><span class="figcap"><b>Figure 1 </b>Field mapping</span><br><span><img id="dataartsstudio_01_0075__en-us_topic_0108275370_image426211715338" src="en-us_image_0000002234082252.png" title="Click to enlarge" class="imgResize"></span></div>
|
|
</p></li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li10112131265818"><span>Click <span class="uicontrol" id="dataartsstudio_01_0075__en-us_topic_0108275370_uicontrol1981952213019"><b>Next</b></span> and set job parameters.</span><p><div class="fignone" id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_fig42703541103949"><span class="figcap"><b>Figure 2 </b>Task parameters</span><br><span><img id="dataartsstudio_01_0075__en-us_topic_0108275370_image15511133218316" src="en-us_image_0000002269121457.png" title="Click to enlarge" class="imgResize"></span></div>
|
|
<div class="p" id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p1952722104238"><a href="#dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_table62790900104257">Table 3</a> describes related parameters.
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_table62790900104257"></a><a name="en-us_topic_0108275370_en-us_topic_0108275458_table62790900104257"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_table62790900104257" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Task configuration parameters</caption><thead align="left"><tr id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_row65473362104257"><th align="left" class="cellrowborder" valign="top" width="22.240000000000002%" id="mcps1.3.4.2.6.2.2.2.2.4.1.1"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p15679301104257">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="59.09000000000001%" id="mcps1.3.4.2.6.2.2.2.2.4.1.2"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p62063881104257">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="18.67%" id="mcps1.3.4.2.6.2.2.2.2.4.1.3"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p61118428104257">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row1367218544311"><td class="cellrowborder" valign="top" width="22.240000000000002%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p0755133114331">Concurrent Tables</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.09000000000001%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p13649152833516">Number of tables to be concurrently executed</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p13755031183312">3</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_row10953185715313"><td class="cellrowborder" valign="top" width="22.240000000000002%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p62157876104257">Concurrent Extractors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.09000000000001%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p1623176104257">Maximum number of threads of the job for reading data from the source</p>
|
|
<div class="note" id="dataartsstudio_01_0075__en-us_topic_0108275370_note710791615819"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p1810771616814">The number of concurrent threads may be less than or equal to the value of this parameter for some data sources that do not support concurrent extraction, for example, CSS and ClickHouse.</p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p64368425104257">1</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_row52808246153537"><td class="cellrowborder" valign="top" width="22.240000000000002%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p49609538153537">Write Dirty Data</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.09000000000001%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p58949638153537">Whether to record dirty data. By default, this parameter is set to <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b926613854011">No</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p10191403153537">Yes</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_row9411130214937"><td class="cellrowborder" valign="top" width="22.240000000000002%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p24104098214937">Write Dirty Data Link</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.09000000000001%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p62681934215229">This parameter is only displayed when <span class="parmname" id="dataartsstudio_01_0075__en-us_topic_0108275370_parmname12442183154115"><b>Write Dirty Data</b></span> is set to <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b744710324118">Yes</strong>.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p6274906214937">Only links to OBS support dirty data writes.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p38505354214937">obs_link</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_row22967022215317"><td class="cellrowborder" valign="top" width="22.240000000000002%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p48389470215317">OBS Bucket</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.09000000000001%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p59616852215319">This parameter is only displayed when <span class="parmname" id="dataartsstudio_01_0075__en-us_topic_0108275370_parmname123731739134116"><b>Write Dirty Data Link</b></span> is a link to OBS.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p4238450021543">Name of the OBS bucket to which the dirty data will be written.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p58389960215317">dirtydata</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_row39573864104257"><td class="cellrowborder" valign="top" width="22.240000000000002%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p51366450104257">Dirty Data Directory</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.09000000000001%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p1077787153637">This parameter is only displayed when <span class="parmname" id="dataartsstudio_01_0075__en-us_topic_0108275370_parmname103102055164111"><b>Write Dirty Data</b></span> is set to <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b1831585554120">Yes</strong>.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p67041813104257">Directory for storing dirty data on OBS. Dirty data is saved only when this parameter is configured.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p2836431711824">You can go to this directory to query data that fails to be processed or is filtered out during job execution, and check the source data that does not meet conversion or cleaning rules.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p61677752104257">/user/dirtydir</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_row45605180153618"><td class="cellrowborder" valign="top" width="22.240000000000002%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.1 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p46055690153628">Max. Error Records in a Single Shard</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.09000000000001%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.2 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p53760732164518">This parameter is only displayed when <span class="parmname" id="dataartsstudio_01_0075__en-us_topic_0108275370_parmname316231116439"><b>Write Dirty Data</b></span> is set to <strong id="dataartsstudio_01_0075__en-us_topic_0108275370_b171679114437">Yes</strong>.</p>
|
|
<p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p39523408153628">When the number of error records of a single map exceeds the upper limit, the job will automatically terminate and the imported data cannot be rolled back. You are advised to use a temporary table as the destination table. After the data is imported, rename the table or combine it into the final data table.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.4.2.6.2.2.2.2.4.1.3 "><p id="dataartsstudio_01_0075__en-us_topic_0108275370_en-us_topic_0108275458_p47279450153628">0</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</p></li><li id="dataartsstudio_01_0075__en-us_topic_0108275370_li1491312617320"><span>Click <span class="uicontrol" id="dataartsstudio_01_0075__en-us_topic_0108275370_uicontrol1021344466103311"><b>Save</b></span> or <span class="uicontrol" id="dataartsstudio_01_0075__en-us_topic_0108275370_uicontrol1926286298103311"><b>Save and Run</b></span>.</span><p><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p101821293216">When the job starts running, a sub-job will be generated for each table. You can click the job name to view the sub-job list.</p>
|
|
</p></li></ol>
|
|
<div class="note" id="dataartsstudio_01_0075__en-us_topic_0108275370_note18704193135911"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dataartsstudio_01_0075__en-us_topic_0108275370_p199811140143619">During the migration of an entire Oracle database to Hudi, if you select a view or a table that has no primary key at the source, automatic table creation is not supported.</p>
|
|
</div></div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0081.html">Creating a Job in a CDM Cluster</a></div>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<script language="JavaScript">
|
|
<!--
|
|
initImageViewer('.imgResize');
|
|
var msg_imageMax = "view original image";
|
|
var msg_imageClose = "close";
|
|
//--></script> |