forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
68 lines
8.5 KiB
HTML
68 lines
8.5 KiB
HTML
<a name="dataartsstudio_01_0401"></a><a name="dataartsstudio_01_0401"></a>
|
|
|
|
<h1 class="topictitle1">Overview</h1>
|
|
<div id="body1563885513904"><p id="dataartsstudio_01_0401__p1555234971516">DataArts Factory is a one-stop big data collaborative development platform that provides fully managed big data scheduling capabilities. It manages various big data services, making big data more accessible than ever before and helping you effortlessly build big data processing centers.</p>
|
|
<p id="dataartsstudio_01_0401__p163389259319">DataArts Factory used to be Data Lake Factory (DLF). Therefore, in this document, both Data Lake Factory and DLF can be used to refer to DataArts Factory.</p>
|
|
<div class="section" id="dataartsstudio_01_0401__section28271144115412"><h4 class="sectiontitle">Introduction to DataArts Factory</h4><p id="dataartsstudio_01_0401__p3963256215128">DataArts Factory enables a variety of operations such as data management, script development, job development, job scheduling, and monitoring, facilitating data analysis and processing.</p>
|
|
<div class="fignone" id="dataartsstudio_01_0401__fig18728125014467"><span class="figcap"><b>Figure 1 </b>DataArts Factory architecture</span><br><span><img id="dataartsstudio_01_0401__image6303573511" src="en-us_image_0000002234083444.png" title="Click to enlarge" class="imgResize"></span></div>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0401__section1214373435515"><h4 class="sectiontitle">Main Functions</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0401__table17951031183019" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Main functions of DataArts Factory</caption><thead align="left"><tr id="dataartsstudio_01_0401__row1996431113011"><th align="left" class="cellrowborder" valign="top" width="21.54%" id="mcps1.3.4.2.2.3.1.1"><p id="dataartsstudio_01_0401__p796193112308">Function</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="78.46%" id="mcps1.3.4.2.2.3.1.2"><p id="dataartsstudio_01_0401__p1096133115301">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0401__row69603183016"><td class="cellrowborder" valign="top" width="21.54%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dataartsstudio_01_0401__p796531193015">Data management</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.46%" headers="mcps1.3.4.2.2.3.1.2 "><ul id="dataartsstudio_01_0401__ul16418350123019"><li id="dataartsstudio_01_0401__li1732395693013">Manages multiple data warehouses, such as GaussDB(DWS), <span id="dataartsstudio_01_0401__text1332315693014">DLI</span> and MRS Hive.</li><li id="dataartsstudio_01_0401__li1124125710303">Manages data tables using the GUI or data definition language (DDL).</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0401__row5971831173011"><td class="cellrowborder" valign="top" width="21.54%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dataartsstudio_01_0401__p109710312305">Script development</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.46%" headers="mcps1.3.4.2.2.3.1.2 "><ul id="dataartsstudio_01_0401__ul59173063110"><li id="dataartsstudio_01_0401__li129203053114">Provides an online script editor that allows more than one operator to collaboratively develop and debug SQL, Python, and Shell scripts online.</li><li id="dataartsstudio_01_0401__li1910306312">Allows use of variables and functions.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0401__row5979317303"><td class="cellrowborder" valign="top" width="21.54%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dataartsstudio_01_0401__p1897631193020">Job development</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.46%" headers="mcps1.3.4.2.2.3.1.2 "><ul id="dataartsstudio_01_0401__ul09950193113"><li id="dataartsstudio_01_0401__li149550203116">Provides a graphical designer that allows you to quickly build a data processing workflow by drag-and-drop.</li><li id="dataartsstudio_01_0401__li12935043112">Presets multiple task types such as data integration, SQL, and Shell, and completes data analysis and processing by dependency between tasks.</li><li id="dataartsstudio_01_0401__li1991850123119">Supports job import and export.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0401__row59743193017"><td class="cellrowborder" valign="top" width="21.54%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dataartsstudio_01_0401__p59733143018">Resource management</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.46%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dataartsstudio_01_0401__p6339175133211">Supports unified management of file, jar, and archive resources used during script and job development.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0401__row139743117307"><td class="cellrowborder" valign="top" width="21.54%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dataartsstudio_01_0401__p109717318308">Job scheduling</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.46%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dataartsstudio_01_0401__p597153116308">Schedules jobs to run once or recursively and use events to trigger scheduling jobs. If the scheduling frequency is set to hour, the scheduling period can be based on interval hour or discrete hour.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0401__row02941014203211"><td class="cellrowborder" valign="top" width="21.54%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dataartsstudio_01_0401__p1429561473217">Monitoring</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.46%" headers="mcps1.3.4.2.2.3.1.2 "><ul id="dataartsstudio_01_0401__ul13385402325"><li id="dataartsstudio_01_0401__li143924015326">You can run, suspend, restore, or terminate a job.</li><li id="dataartsstudio_01_0401__li10396404325">You can view the operation details of each job and each node in the job.</li><li id="dataartsstudio_01_0401__li23954018323">You can use various methods to receive notifications when a job or task error occurs.</li></ul>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0401__section699015157442"><h4 class="sectiontitle">Objects in DataArts Factory</h4><ul id="dataartsstudio_01_0401__ul171489332172"><li id="dataartsstudio_01_0401__li11324113815179">Data connection: A data connection is a collection of information required for accessing data storage (computing) space, including the connection type, name, and login information.</li><li id="dataartsstudio_01_0401__li4171144651718">Solution: A solution provides users with convenient and systematic management operations to better meet service requirements and objectives. Each solution can contain one or more business-related jobs, and one job can be used by multiple solutions.</li></ul>
|
|
<ul id="dataartsstudio_01_0401__ul186588553172"><li id="dataartsstudio_01_0401__li5659195541716">Job: A job is composed of one or more nodes and can be executed to complete data operations.</li><li id="dataartsstudio_01_0401__li1265195133719">Script: A script is an extension of a batch processing file. It is a program that stores text. Generally, a computer script program is a combination of a series of operations that control computers to perform operations. In the script program, certain logic branches can be implemented.</li><li id="dataartsstudio_01_0401__li19306112201815">Node: A node defines the operations performed on data.</li><li id="dataartsstudio_01_0401__li8926115711719">Resource: Resources refer to self-defined codes or text files that are uploaded by users and scheduled when node tasks are executed.</li></ul>
|
|
<ul id="dataartsstudio_01_0401__ul135823751810"><li id="dataartsstudio_01_0401__li6513162818182">Expression: Node parameter values in a node job can be dynamically generated based on the running environment by using Expression Language (EL). EL uses simple arithmetic and logic to calculate and reference embedded objects, including job objects and tool objects.</li></ul>
|
|
<ul id="dataartsstudio_01_0401__ul149741072235"><li id="dataartsstudio_01_0401__li497416711230">Environment variable: An environment variable is an object with a specific name in the operating system. It contains information to be used by one or more applications.</li><li id="dataartsstudio_01_0401__li17572155222913">PatchData: PatchData refers to the instance that is generated in a period of time by a periodically scheduled job.</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0400.html">DataArts Factory</a></div>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<script language="JavaScript">
|
|
<!--
|
|
initImageViewer('.imgResize');
|
|
var msg_imageMax = "view original image";
|
|
var msg_imageClose = "close";
|
|
//--></script> |