doc-exports/docs/dataartsstudio/umn/dataartsstudio_07_005.html

<a name="dataartsstudio_07_005"></a><a name="dataartsstudio_07_005"></a>

<h1 class="topictitle1">Functions</h1>
<div id="body1555916447705"><div class="section" id="dataartsstudio_07_005__section17688828666"><h4 class="sectiontitle">DataArts Migration: Efficient Ingestion of Multiple Heterogeneous Data Sources</h4><p id="dataartsstudio_07_005__p71417513616">DataArts Migration can help you seamlessly migrate batch data between 30+ homogeneous or heterogeneous data sources. You can use it to ingest data from both on-premises and cloud-based data sources, including file systems, relational databases, data warehouses, NoSQL databases, big data services, and object storage.</p>
<p id="dataartsstudio_07_005__p52164437151344">DataArts Migration uses a distributed compute framework and concurrent processing techniques to help you migrate enterprise data in batches without any downtime and rapidly build desired data structures.</p>
<div class="fignone" id="dataartsstudio_07_005__fig112481851164813"><span class="figcap"><b>Figure 1 </b>DataArts Migration</span><br><span><img id="dataartsstudio_07_005__image18592192455313" src="en-us_image_0000002234068868.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_07_005__p2384479126">You can manage data on the wizard-based task management page. You can easily create data migration tasks that meet your requirements. DataArts Migration provides the following functions:</p>
<ul id="dataartsstudio_07_005__ul20174521145718"><li id="dataartsstudio_07_005__li51745219571"><strong id="dataartsstudio_07_005__b107791939153011">Table/File/Entire DB migration</strong><p id="dataartsstudio_07_005__p3767173325717">You can migrate tables or files in batches, and migrate an entire database between homogeneous and heterogeneous database systems. You can include hundreds of tables in a single job.</p>
</li><li id="dataartsstudio_07_005__li123531673310"><strong id="dataartsstudio_07_005__b761465517287">Incremental data migration</strong><p id="dataartsstudio_07_005__p666246937">You can migrate files, relational databases, and HBase in an incremental manner. You can perform incremental data migration by using WHERE clauses and variables of date and time.</p>
</li><li id="dataartsstudio_07_005__li18121153121410"><strong id="dataartsstudio_07_005__b4900115113302">Migration in transaction mode</strong><p id="dataartsstudio_07_005__p1196391814157">When a batch data migration job fails to be executed, data will be rolled back to the state before the job started and data in the destination table will be automatically deleted.</p>
</li><li id="dataartsstudio_07_005__li7819113613719"><strong id="dataartsstudio_07_005__b1820732491510">Field conversion</strong><p id="dataartsstudio_07_005__p15186527581">Field conversion includes anonymization, character string operations, and date operations.</p>
</li><li id="dataartsstudio_07_005__li18055051710"><strong id="dataartsstudio_07_005__b2069135111615">File encryption</strong><p id="dataartsstudio_07_005__p153840451179">You can encrypt files that are migrated to a cloud-based file system in batches.</p>
</li><li id="dataartsstudio_07_005__li163581165201"><strong id="dataartsstudio_07_005__b3183157151614">MD5 verification</strong><p id="dataartsstudio_07_005__p9320173318207">MD5 is used to check file consistency from end to end.</p>
</li><li id="dataartsstudio_07_005__li184661212911"><strong id="dataartsstudio_07_005__b113467108171">Dirty data archiving</strong><p id="dataartsstudio_07_005__p694465019919">Data that fails to be processed during migration, is filtered out and is not compliant with conversion or cleansing rules is recorded in dirty data logs. You can easily analyze abnormal data. You can also set a threshold for the dirty data ratio to determine whether a task is successful.</p>
</li></ul>
</div>
<div class="section" id="dataartsstudio_07_005__section1782414551778"><h4 class="sectiontitle">DataArts Factory: One-stop Collaborative Development</h4><p id="dataartsstudio_07_005__p145420574317">DataArts Factory provides an intuitive UI and built-in development methods for script and job development. DataArts Factory also supports fully hosted job scheduling, O&amp;M, and monitoring, and incorporates industry data processing pipelines. You can create data development jobs in a few steps, and the entire process is visual. Online jobs can be jointly developed by multiple users. You can use DataArts Factory to manage big data cloud services and quickly build a big data processing center.</p>
</div>
<div class="fignone" id="dataartsstudio_07_005__fig18728125014467"><span class="figcap"><b>Figure 2 </b>DataArts Factory architecture</span><br><span><img id="dataartsstudio_07_005__image18988432578" src="en-us_image_0000002269188109.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_07_005__p82514187274">DataArts Factory allows you to manage data, develop scripts, and schedule and monitor jobs. Data analysis and processing are easier than ever before.</p>
<ul id="dataartsstudio_07_005__ul13833141116"><li id="dataartsstudio_07_005__li77445520131"><strong id="dataartsstudio_07_005__b19414342296">Data management</strong><ul id="dataartsstudio_07_005__ul0791145915135"><li id="dataartsstudio_07_005__li438163171114">You can manage multiple types of data warehouses, such as GaussDB (DWS), <span id="dataartsstudio_07_005__text197321215105717">DLI</span>, and MRS Hive.</li><li id="dataartsstudio_07_005__li103813312118">You can use the graphical interface and data definition language (DDL) to manage database tables.</li></ul>
</li><li id="dataartsstudio_07_005__li137535531315"><strong id="dataartsstudio_07_005__b741117141959">Script development</strong><ul id="dataartsstudio_07_005__ul1716235191415"><li id="dataartsstudio_07_005__li1131013471417">Provides an online script editor that allows more than one operator to collaboratively develop and debug SQL, Python, and Shell scripts online.</li><li id="dataartsstudio_07_005__li6310143411419">You can use Variables.</li></ul>
</li></ul>
<ul id="dataartsstudio_07_005__ul174731038121214"><li id="dataartsstudio_07_005__li1475175513133"><strong id="dataartsstudio_07_005__b46815203134">Job development</strong><ul id="dataartsstudio_07_005__ul97393292151"><li id="dataartsstudio_07_005__li238032841512">DataArts Factory provides a graphical designer that allows you to rapidly develop workflows through drag-and-drop and build data processing pipelines.</li><li id="dataartsstudio_07_005__li338172881518">DataArts Factory is preset with multiple task types such as data integration, SQL, and Shell. Data is processed and analyzed based on task dependencies.</li><li id="dataartsstudio_07_005__li73821728131513">You can import and export jobs.</li></ul>
</li></ul>
<ul id="dataartsstudio_07_005__ul15489175213124"><li id="dataartsstudio_07_005__li177510553131"><strong id="dataartsstudio_07_005__b17767124317189">Resource management</strong><p id="dataartsstudio_07_005__p622735181415">You can centrally manage file, jar, and archive resources used during script and job development.</p>
</li><li id="dataartsstudio_07_005__li1175185515135"><strong id="dataartsstudio_07_005__b1920141461910">Job scheduling</strong><ul id="dataartsstudio_07_005__ul3960183219272"><li id="dataartsstudio_07_005__li13517133110278">You can schedule jobs to run once or recursively and use events to trigger scheduling jobs.</li><li id="dataartsstudio_07_005__li3517133152713">Job scheduling supports a variety of hybrid orchestration tasks. The high-performance scheduling engine has been tested by hundreds of applications.</li></ul>
</li><li id="dataartsstudio_07_005__li575155517135"><strong id="dataartsstudio_07_005__b7141122604712">O&amp;M and monitoring</strong><ul id="dataartsstudio_07_005__ul1270810395135"><li id="dataartsstudio_07_005__li1470873913131">You can run, suspend, restore, or terminate a job.</li><li id="dataartsstudio_07_005__li11708173951315">You can view the operation details of each job and each node in the job.</li><li id="dataartsstudio_07_005__li97086398136">You can use various methods to receive notifications when a job or task error occurs.</li></ul>
</li></ul>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_12_0001.html">Service Overview</a></div>
</div>
</div>


<script language="JavaScript">
<!--
initImageViewer('.imgResize');
var msg_imageMax = "view original image";
var msg_imageClose = "close";
//--></script>