Prerequisites
A DataArts Studio instance has been obtained. For details, see Creating a DataArts Studio Basic Package.
+Prerequisites
You have obtained a DataArts Studio instance. For details, see Creating a DataArts Studio Instance.
Procedure
- Log in to the cloud console. Choose DataArts Studio from the service list in the upper left corner to access the DataArts Studio instance console.
- If there are multiple DataArts Studio instances in the current region, the instance list is displayed by default. Locate an instance and click Access to access the DataArts Studio console.
+
Procedure
- Log in to the cloud console. Choose DataArts Studio from the service list in the upper left corner.
- If there are multiple DataArts Studio instances in the current region, the instance list is displayed by default. Locate an instance and click Access to access the DataArts Studio console homepage.
- If there is only one DataArts Studio instance in the current region, the DataArts Studio console homepage is displayed by default. @@ -14,7 +14,7 @@
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0003.html b/docs/dataartsstudio/umn/dataartsstudio_01_0003.html deleted file mode 100644 index 46f5709c3..000000000 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0003.html +++ /dev/null @@ -1,13 +0,0 @@ - - --Parent topic: Preparations+Parent topic: Creating and Configuring a DataArts Studio InstancePreparations
--Before using DataArts Studio, you need to create a a cloud platform account, create a DataArts Studio instance, access the DataArts Studio instance console, manage workspaces, and authorize users to use DataArts Studio.
-The following figure shows the preparations and operations, which will be described in detail in later sections.
-Figure 1 Preparations for using DataArts Studio-
-- diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0004.html b/docs/dataartsstudio/umn/dataartsstudio_01_0004.html index a4dfc6290..b18be2262 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0004.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0004.html @@ -1,27 +1,29 @@ ---Parent topic: Preparations-Creating an IAM User and Assigning DataArts Studio Permissions
-Identity and Access Management (IAM) can be used for fine-grained permissions management on your DataArts Studio resources. With IAM, you can:- Create IAM users for employees based on your enterprise's organizational structure. Each IAM user will have their own security credentials for accessing DataArts Studio resources.
- Assign users only the permissions required to perform a task.
- Entrust a a cloud platform account or cloud service to perform efficient O&M on your DataArts Studio resources.
Creating an IAM User and Assigning DataArts Studio Permissions
+Identity and Access Management (IAM) can be used for fine-grained permissions management on your DataArts Studio resources. With IAM, you can:-- Create IAM users for employees based on your enterprise's organizational structure. Each IAM user will have their own security credentials for accessing DataArts Studio resources.
- Grant users only the permissions required to perform a given task based on their job responsibilities.
- Entrust a cloud account or cloud service to perform efficient O&M on your DataArts Studio resources.
If you do not require individual IAM users for permissions management, skip this section.
--Background
- Before assigning permissions to a user group, you need to understand the DataArts Studio permission system and select permissions as needed.
Notes and Constraints
- The DARTS User role provides permissions related to instances, workspaces, and dependent services. Workspace roles provide service operation permissions in workspaces.
- IAM provides the following two authorization mechanisms. DataArts Studio supports IAM roles but not IAM policies.
- IAM roles: IAM initially provides a coarse-grained authorization mechanism to define permissions based on users' job responsibilities. Only a limited number of service-level roles are available for authorization. However, traditional IAM roles are not an ideal choice for fine-grained authorization and secure access control.
- IAM policies: A type of fine-grained authorization mechanism that defines permissions required to perform operations on specific cloud resources under certain conditions. This type of authorization is more flexible and meets the principle of least privilege (PoLP).
If you do not require individual IAM users for permissions management, skip this topic.
+This section describes the procedure for granting permissions. See Procedure for details.
+-Context
- Before assigning permissions to a user group, you need to understand the permission system of so that you can select the permissions that meet your needs. Figure 1 Permission system
Procedure
Figure 1 Authorization process-
- Create a user group and assign a system role to the group.
Log in to the IAM console using a cloud platform account, create a user group and assign a DataArts Studio system role to the group. For example, the system role can be DARTS Administrator or DARTS User.
-For details, see "User Groups and Authorization" > "Creating a User Group and Assigning Permissions" in Identity and Access Management User Guide.
-
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0005.html b/docs/dataartsstudio/umn/dataartsstudio_01_0005.html new file mode 100644 index 000000000..9e5a3d81a --- /dev/null +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0005.html @@ -0,0 +1,234 @@ + + +- When configuring DataArts Studio permissions for a user group, enter DARTS in the search box to search for the permissions and select the permissions to be granted to the user group, for example, DARTS User.
- DataArts Studio is a project-level service deployed in specific physical regions. If you select All resources for Scope, the permission takes effect in all projects of all regions. If you select Region-specific projects for Scope, the permission takes effect only for a specified project. When accessing DataArts Studio, the IAM user must switch to the region where they have been assigned the required permissions.
+Notes and Constraints
- The DARTS User system role provides the permissions related to instances, workspaces, and dependent services. The operation permissions in workspaces are provided by workspace roles.
- IAM provides the following two authorization mechanisms: Note that DataArts Studio supports only the IAM role-based authorization and does not support the IAM policy-based authorization.+
- IAM Roles: IAM initially provides a coarse-grained authorization mechanism to define permissions based on users' job responsibilities. Only a limited number of service-level roles are available. However, traditional IAM roles are not an ideal choice for fine-grained authorization and secure access control.
- IAM Policies: A type of fine-grained authorization mechanism that defines permissions required to perform operations on specific cloud resources under certain conditions. This type of authorization is more flexible and is ideal for least privilege access.
Procedure
- Create a user group and assign a system role to the group.
Log in to the IAM console using a cloud account, create a user group and assign a DataArts Studio system role to the group. For example, the system role can be DARTS Administrator or DARTS User.
+For details, see "User Groups and Authorization" > "Creating a User Group and Assigning Permissions" in Identity and Access Management User Guide.
+- - When configuring DataArts Studio permissions for a user group, enter DARTS in the search box to search for the permissions and select the permissions to be granted to the user group, for example, DARTS User.
- DataArts Studio is a project-level service deployed in specific physical regions. If you select All resources for Scope, the permission takes effect in all projects of all regions. If you select Region-specific projects for Scope, the permission takes effect only for a specified project. When accessing DataArts Studio, the IAM user must switch to the region where they have been assigned the required permissions.
- Create a user and add it to the user group.
Create users on the IAM console and add them to the group created in step 1.
-For details, see "IAM Users" > "Creating an IAM User" in Identity and Access Management User Guide.
-@@ -31,3 +33,10 @@ An IAM user can pass the authentication and access DataArts Studio through an API or SDK only if Programmatic access is selected for Access Type during the creation of the IAM user.
+- Create a user and add the user to the user group.
Create users on the IAM console and add them to the group created in Step 1.
+For details, see "IAM Users" > "Creating an IAM User" in Identity and Access Management User Guide.
+- An IAM user can pass the authentication and access DataArts Studio through an API or SDK only if Programmatic access is selected for Access Type during the creation of the IAM user.
- Create a custom workspace role for DARTS User, add it as a workspace member, and assign a role to the member.
DataArts Studio workspace roles determine the permissions of DARTS User in a workspace. There are five preset roles: admin, developer, deployer, operator, and viewer. If these preset roles can meet your requirements, you do not need to create a custom workspace role. Instead, you can add the user as a workspace member and assign a preset role to the member. Otherwise, you need to create a custom role, add the users as a workspace member, and assign the custom role to the member. For details about how to create a custom workspace role, see (Optional) Defining a Workspace Role. For details about how to add a workspace member and assign a role to the member, see Adding a Member and Assigning a Role.
-For details about the permissions of the roles, see "DataArts Studio Permissions" in Service Overview.
+- Create a custom workspace role for DARTS User, add it as a workspace member, and assign a role to the member.
DataArts Studio workspace roles determine the permissions of DARTS User in a workspace. There are five preset roles: admin, developer, deployer, operator, and viewer. If these preset roles can meet your requirements, you do not need to create a custom workspace role. Instead, you can add the user as a workspace member and assign a preset role to the member. Otherwise, you need to create a custom role, add the users as a workspace member, and assign a custom role to the member. For details about how to create a custom workspace role, see (Optional) Defining a Workspace RoleFor details about how to add a workspace member and assign a role, see Adding Workspace Members and Assigning Roles.
+For details about the permissions of the roles, see "Permissions" in Service Overview.
- Log in to the console and verify permissions.
Log in to the console using the created user and verify permissions of the user.
-- Choose Service List > DataArts Studio. Locate a DataArts Studio instance and click Access. Check whether the workspace list is displayed.
- Access a service module (for example, Management Center) to which your current user has been added and check whether you can perform the operations allowed for the workspace role assigned to you.
- Choose Service List > DataArts Studio. Locate a DataArts Studio instance and click Access. Check whether the workspace list is displayed.
- Access a service module (for example, Management Center) to which your current user has been added and check whether you can perform the operations allowed for the workspace role assigned to you.
- Create a user and add the user to the user group.
Data Sources Supported by DataArts Studio
++Before using DataArts Studio, you need to select cloud services or databases as the data foundation, which provides storage and compute capabilities. DataArts Studio provides one-stop data development, governance, and services based on the data foundation.
++Supported Data Sources
This section describes the data sources supported by DataArts Studio modules other than DataArts Migration. Table 1 lists the data sources supported by each module.
+Except DataArts Migration, all other modules use the data connections created in Management Center. (A data connection can be used in a module which was selected during the creation of the connection.) To connect to these data sources, go to the DataArts Studio console and choose Management Center to create data connections.
++ + The data sources supported by the migration jobs in DataArts Migration are different from those supported by other modules, and are described in the DataArts Migration chapter.
++
+Table 1 Data sources supported by DataArts Studio Data Source Type
+Management Center
+DataArts Factory
+DWS
+Supported
+Supported
+DLI
+Supported
+Supported
+MRS HBase
+Supported
+Not supported
+MRS Hive
+Supported
+Supported
+MRS Kafka
+Supported
+Supported
+MRS Spark
+Supported
+Supported
+MRS ClickHouse
+Supported
+Supported
+MRS Hetu
+Supported
+Supported
+MRS Impala
+Supported
+Supported
+MRS Doris
+Supported
+Supported
+RDS for MySQL
+Supported
+Supported
+RDS for PostgreSQL
+Supported
+Supported
+RDS for SQL Server
+Supported
+Not supported
+MySQL
+Supported
+Not supported
+Oracle
+Supported
+Not supported
+Host Connection
+Supported
+Supported
++ DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
++Overview
++
+Table 2 Data source overview Data Source Type
+Description
+DWS
+DWS employs the shared-nothing architecture and massively parallel processing (MPP) engine. It is compatible with ANSI SQL 99, SQL 2003, and the PostgreSQL or Oracle database ecosystem, providing competitive solutions for analyzing petabytes of data in various industries.
+DLI
+DLI is a serverless big data compute and analysis service that is fully compatible with Apache Spark and Apache Flink ecosystems. With multi-model engines supported by DLI, enterprises can use SQL statements or programs to easily complete batch processing, stream processing, in-memory computing, and machine learning of heterogeneous data sources.
+MRS HBase
+HBase undertakes data storage. It is an open-source, column-oriented, distributed storage system that is suitable for storing massive amounts of unstructured or semi-structured data. It features high reliability, high performance, and flexible scalability, and supports real-time data read/write.
+MRS HBase stores massive amount of data and supports data queries in milliseconds. MRS HBase can load and update logistics data in milliseconds, and query and analyze petabytes of time series data in seconds.
+MRS Hive
+Hive is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. Hive defines simple SQL-like query language, which is known as HiveQL. It allows a user familiar with SQL to query data.
+MRS Hive can be used to analyze terabytes or petabytes of data and quickly migrate on-premises Hadoop big data platforms (such as CDH and HDP) to the cloud without service interruption and service code modification.
+MRS Kafka
+MRS provides dedicated MRS Kafka clusters. Kafka is an open-source, distributed, partitioned, and replicated commit log service. Kafka is publish-subscribe messaging, rethought as a distributed commit log. It provides features similar to Java Message Service (JMS) but another design. It features message endurance, high throughput, distributed methods, multi-client support, and real time. It applies to both online and offline message consumption, such as regular message collection, website activeness tracking, aggregation of statistical system operation data (monitoring data), and log collection. These scenarios engage large amounts of data collection for Internet services.
+MRS Spark
+Spark is an open-source parallel data processing framework. It helps users easily develop unified big data applications and perform cooperative processing, stream processing, and interactive analysis on data.
+Spark provides a framework featuring fast calculation, write, and interactive query. Spark has obvious advantages over Hadoop in terms of performance. Spark provides the Spark SQL language similar to SQL statements to process structured data.
+MRS ClickHouse
+ClickHouse is an open-source columnar database oriented to online analysis and processing. It is independent of the Hadoop big data system and features ultimate compression rate and fast query performance. In addition, ClickHouse supports SQL query and provides good query performance, especially the aggregation analysis and query performance based on large and wide tables. The query speed is one order of magnitude faster than that of other analytical databases.
+ClickHouse is widely used in various fields such as Internet advertising, apps, web, telecommunications, finance, and IoT. It suits business intelligence ideally.
+MRS Impala
+Impala provides fast, interactive SQL queries directly on your Apache Hadoop data stored in HDFS, HBase, or the Object Storage Service (OBS). In addition to using the same unified storage platform, Impala also uses the same metadata, SQL syntax (Hive SQL), ODBC driver, and user interface (Impala query UI in Hue) as Apache Hive. This provides a familiar and unified platform for real-time or batch-oriented queries. Impala is an addition to tools available for querying big data. Impala does not replace the batch processing frameworks built on MapReduce such as Hive. Hive and other frameworks built on MapReduce are best suited for long running batch jobs.
+MRS Hudi
+Hudi is a data lake table format that provides the ability to update and delete data as well as consume new data on HDFS. It supports multiple compute engines and provides insert, update, and delete (IUD) interfaces and streaming primitives, including upsert and incremental pull, over datasets on HDFS.
+Hudi metadata is stored in Hive, and operations are performed using Spark.
+MRS Doris
+Doris is a high-performance, real-time analytical database. It can return query results of mass data in sub-seconds and can support high-concurrency point queries and high-throughput complex analysis. Apache Doris can meet requirements in report analysis, instant query, unified data warehouse building, and data lake federated query.
+RDS
+RDS is an online, out-of-the-box relational database service that is based on the cloud computing platform. It is stable, reliable, scalable, and easy to manage.
+MySQL
+MySQL is one of the most popular open-source databases. It features excellent performance, uses mature and stable architecture, supports popular applications, adapts to multiple fields and industries, and supports various web applications. It is cost-effective and preferred by small- and medium-sized enterprises.
+Oracle
+Oracle is a group of software that mainly applied to the distributed database. The Oracle database is one of the most popular Client/Server (C/S) and Browser/Server (B/S) databases.
+It is also the most widely used database management system in the world. As a general database system, the Oracle database provides complete data management functions. As a relational database, it provides complete relational models. As a distributed database, it implements distributed data processing.
+Rest Client
+The Rest Client can be used to execute RESTful requests that are authenticated using IAM tokens or usernames and passwords.
+Host Connection
+You can connect to a specified host during data development and execute shell or Python scripts on the host through script development and job development. If the host connection information changes, you only need to edit it on the Host Connections page, but do not need to edit it in scripts or jobs one by one.
+++ diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0005_0.html b/docs/dataartsstudio/umn/dataartsstudio_01_0005_0.html deleted file mode 100644 index 0cb6365cc..000000000 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0005_0.html +++ /dev/null @@ -1,228 +0,0 @@ - - -++Parent topic: Management Center+Data Sources
--Before using DataArts Studio, you need to select cloud services or databases as the data lake foundation, which provides storage and compute capabilities. DataArts Studio provides one-stop data development, governance, and services based on the data lake foundation.
--Data Sources Supported By DataArts Studio
DataArts Studio can interconnect with cloud services such as DWS, DLI, and MRS Hive as well as traditional databases such as MySQL and . For details, see Table 1.
-To connect to these data sources, go to the DataArts Studio console and choose Management Center to create a data connection.
-- - The data connections in DataArts Studio Management Center are used to connect to the data lake foundation. DataArts Studio provides one-stop data development, governance, and services based on the data lake foundation.
--
-Table 1 Data sources supported by DataArts Studio Data Source Type
-Management Center
-DataArts Factory
-DWS
-Supported
-Supported
-DLI
-Supported
-Supported
-MRS HBase
-Supported
-Not supported
-MRS Hive
-Supported
-Supported
-MRS Kafka
-Supported
-Supported
-MRS Spark
-Supported
-Supported
-MRS ClickHouse
-Supported
-Supported
-MRS Hetu
-Supported
-Supported
-MRS Impala
-Supported
-Supported
-MRS Doris
-Supported
-Supported
-RDS for MySQL
-Supported
-Supported
-RDS for PostgreSQL
-Supported
-Supported
-MySQL
-Supported
-Not supported
-Oracle
-Supported
-Not supported
-Host Connection
-Supported
-Supported
-- DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
--Overview
--
-Table 2 Data source overview Data Source Type
-Description
-DWS
-DWS employs the shared-nothing architecture and massively parallel processing (MPP) engine. It is compatible with ANSI SQL 99, SQL 2003, and the PostgreSQL or Oracle database ecosystem, providing competitive solutions for analyzing petabytes of data in various industries.
-DLI
-DLI is a serverless big data compute and analysis service that is fully compatible with Apache Spark and Apache Flink ecosystems. With multi-model engines supported by DLI, enterprises can use SQL statements or programs to easily complete batch processing, stream processing, in-memory computing, and machine learning of heterogeneous data sources.
-MRS HBase
-HBase undertakes data storage. It is an open-source, column-oriented, distributed storage system that is suitable for storing massive amounts of unstructured or semi-structured data. It features high reliability, high performance, and flexible scalability, and supports real-time data read/write.
-MRS HBase stores massive amount of data and supports data queries in milliseconds. MRS HBase can load and update logistics data in milliseconds, and query and analyze petabytes of time series data in seconds.
-MRS Hive
-Hive is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. Hive defines simple SQL-like query language, which is known as HiveQL. It allows users familiar with SQL to query data.
-MRS Hive can be used to analyze terabytes or petabytes of data and quickly migrate on-premises Hadoop big data platforms (such as CDH and HDP) to the cloud without service interruption and service code modification.
-MRS Kafka
-MRS provides dedicated MRS Kafka clusters. Kafka is an open-source, distributed, partitioned, and replicated commit log service. Kafka is publish-subscribe messaging, rethought as a distributed commit log. It provides features similar to Java Message Service (JMS) but another design. It features message endurance, high throughput, distributed methods, multi-client support, and real time. It applies to both online and offline message consumption, such as regular message collection, website activeness tracking, aggregation of statistical system operation data (monitoring data), and log collection. These scenarios engage large amounts of data collection for Internet services.
-MRS Spark
-Spark is an open-source parallel data processing framework. It helps users easily develop unified big data applications and perform cooperative processing, stream processing, and interactive analysis on data.
-Spark provides a framework featuring fast calculation, write, and interactive query. Spark has obvious advantages over Hadoop in terms of performance. Spark provides the Spark SQL language similar to SQL statements to process structured data.
-MRS ClickHouse
-ClickHouse is an open-source columnar database oriented to online analysis and processing. It is independent of the Hadoop big data system and features ultimate compression rate and fast query performance. In addition, ClickHouse supports SQL query and provides good query performance, especially the aggregation analysis and query performance based on large and wide tables. The query speed is one order of magnitude faster than that of other analytical databases.
-ClickHouse is widely used in various fields such as Internet advertising, apps, web, telecommunications, finance, and IoT. It suits business intelligence ideally.
-MRS Impala
-Impala provides fast, interactive SQL queries directly on your Apache Hadoop data stored in HDFS, HBase, or the Object Storage Service (OBS). In addition to using the same unified storage platform, Impala also uses the same metadata, SQL syntax (Hive SQL), ODBC driver, and user interface (Impala query UI in Hue) as Apache Hive. This provides a familiar and unified platform for real-time or batch-oriented queries. Impala is an addition to tools available for querying big data. Impala does not replace the batch processing frameworks built on MapReduce such as Hive. Hive and other frameworks built on MapReduce are best suited for long running batch jobs.
-MRS Hudi
-Hudi is a data lake table format that provides the ability to update and delete data as well as consume new data on HDFS. It supports multiple compute engines and provides insert, update, and delete (IUD) interfaces and streaming primitives, including upsert and incremental pull, over datasets on HDFS.
-Hudi metadata is stored in Hive, and operations are performed using Spark.
-MRS Doris
-Doris is a high-performance, real-time analytical database. It can return query results of mass data in sub-seconds and can support high-concurrency point queries and high-throughput complex analysis. Apache Doris can meet requirements in report analysis, instant query, unified data warehouse building, and data lake federated query.
-RDS
-RDS is an online, out-of-the-box relational database service that is based on the cloud computing platform. It is stable, reliable, scalable, and easy to manage.
-Currently, DataArts Studio supports only MySQL and PostgreSQL databases in RDS.
-MySQL
-MySQL is one of the most popular open-source databases. It features excellent performance, uses mature and stable architecture, supports popular applications, adapts to multiple fields and industries, and supports various web applications. It is cost-effective and preferred by small- and medium-sized enterprises.
-Oracle
-Oracle is a group of software that mainly applied to the distributed database. The Oracle database is one of the most popular Client/Server (C/S) and Browser/Server (B/S) databases.
-It is also the most widely used database management system in the world. As a general database system, the Oracle database provides complete data management functions. As a relational database, it provides complete relational models. As a distributed database, it implements distributed data processing.
-Rest Client
-The Rest Client can be used to execute RESTful requests that are authenticated using IAM tokens or usernames and passwords.
-Host Connection
-You can connect to a specified host during data development and execute shell or Python scripts on the host through script development and job development. If the host connection information changes, you only need to edit it on the Host Connections page, but do not need to edit it in scripts or jobs one by one.
--- diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0006.html b/docs/dataartsstudio/umn/dataartsstudio_01_0006.html deleted file mode 100644 index 254b2d28a..000000000 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0006.html +++ /dev/null @@ -1,28 +0,0 @@ - - ---Parent topic: Management Center-(Optional) Obtaining Authentication Information
--When using DataArts Studio, for example, when creating OBS links, calling APIs, or locating issues, you may need to obtain information such as access keys, project IDs, and endpoints. This section describes how to obtain such information.
--Obtaining an Access Key
To obtain an access key, perform the following steps:-- Log in to the TICS console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose . See Figure 1. -
- Click OK and save the access key file when prompted. The access key file is saved to the default download location in your browser. Open the credentials.csv file to view the access key with an AK and SK.-
- Only two access keys can be added for each user.
- For security purposes, access keys are automatically downloaded only when they are generated for the first time and cannot be obtained from the console later. Keep them properly.
-Obtaining a Project ID and Account ID
A project is a group of tenant resources, and an account ID corresponds to the current account. The IAM ID corresponds to the current user. You can view the project IDs, account IDs, and user IDs in different regions on the corresponding pages.-- Register with and log in to the management console.
- Hover the cursor on the username in the upper right corner and select My Credentials from the drop-down list.
- On the API Credentials page, obtain the account name, account ID, IAM username, and IAM user ID, and obtain the project ID from the project list.
-Obtaining the DataArts Studio Instance ID and Workspace ID
You can obtain them from the URI on the DataArts Studio console.-- On the homepage of the DataArts Studio console, select a workspace, and click a module, for example, Management Center.
- On the Management Center page, obtain the values of instanceId and workspace in the browser address bar, which are the DataArts Studio instance ID and workspace ID, respectively.As shown in Figure 2, the instance ID is 6b88…2688, and the workspace ID is 1dd3bc…d93f0. --
-Obtaining an Endpoint
An endpoint is the request address for calling an API. Endpoints vary depending on services and regions.
-You can obtain endpoints from Regions and Endpoints.
--- diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0008.html b/docs/dataartsstudio/umn/dataartsstudio_01_0008.html index a6c849a98..087e8b31f 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0008.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0008.html @@ -5,15 +5,17 @@--Parent topic: Preparations--
-
- Data Sources
+ - Data Sources Supported by DataArts Studio
- - Managing Data Connections
+ - Creating a DataArts Studio Data Connection
- - Migrating Resources
+ - Configuring DataArts Studio Data Connection Parameters
- - Configuring Environment Isolation for a Workspace in Enterprise Mode
+ - Configuring DataArts Studio Resource Migration
- - Tutorials
+ - Configuring Environment Isolation for a DataArts Studio Workspace in Enterprise Mode
+
+ - Typical Scenarios for Using Management Center
Managing Data Connections
+Configuring DataArts Studio Data Connection Parameters
-
-
- Creating a Data Connection
+ - DWS Connection Parameters
- - Configuring a DWS Connection
+ - DLI Connection Parameters
- - Configuring a DLI Connection
+ - MRS Hive Connection Parameters
- - Configuring an MRS Hive Connection
+ - MRS HBase Connection Parameters
- - Configuring an MRS HBase Connection
+ - MRS Kafka Connection Parameters
- - Configuring an MRS Kafka Connection
+ - MRS Spark Connection Parameters
- - Configuring an MRS Spark Connection
+ - MRS ClickHouse Connection Parameters
- - Configuring an MRS ClickHouse Connection
+ - MRS Hetu Connection Parameters
- - Configuring an MRS Hetu Connection
+ - MRS Impala Connection Parameters
- - Configuring an MRS Impala Connection
+ - Doris Connection Parameters
- - Configuring an MRS Doris Connection
+ - OpenSource ClickHouse Connection Parameters
- - Configuring an RDS Connection
+ - RDS Connection Parameters
- - Configuring an Oracle Connection
+ - Oracle Connection Parameters
- - Configuring a Host Connection
+ - Host Connection Parameters
+
+ - Rest Client Connection Parameters
+
+ - Redis Connection Parameters
+
+ - SAP HANA Connection Parameters
Migrating Resources
+Configuring DataArts Studio Resource Migration
To migrate resources in one workspace to another, you can use the resource migration function provided by DataArts Studio.
-Resources that can be migrated include the data connections in Management Center.
-Prerequisites
- Resources can be imported from OBS or a local path.
- There are resources that can be migrated. For details on how to create data connections, see Managing Data Connections.
Resources can be imported from OBS or a local path. Resources that can be migrated include the following service data:-- Data connections created in Management Center
- CDM jobs created in DataArts Migration, including the links in jobs
- Scripts and jobs that have been submitted in DataArts Factory. By default, when jobs are exported, their dependent scripts and resources are not exported.
Constraints
- Only an exported .zip file can be imported. During the import, the system verifies the resources in the file.
- For security concerns, passwords of connections are not exported when the connections are exported. You need to enter the passwords when importing the connections.
- The size of the file to be imported from an OBS bucket or local path cannot exceed 10 MB.
-Constraints
- Only an exported .zip file can be imported. During the import, the system verifies the resources in the file.
- For security concerns, passwords of connections are not exported when the connections are exported. You need to enter the passwords when importing the connections.
- The file to be imported from an OBS bucket or local path cannot be larger than 10 MB.
Exporting a Resource
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console.
- On the DataArts Studio console, locate a workspace and click Management Center.
- In the navigation pane, choose Migrate Resources.Figure 1 Migrating Resources+
-Exporting a Resource
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console.
- On the DataArts Studio console, locate a workspace and click Management Center.
- In the navigation pane, choose Migrate Resources.Figure 1 Migrating Resources-
- Click Create Export Task to configure the file name and the OBS path for saving the file. If OBS is unavailable, you only need to set the file name.Figure 2 Export Task+
- Click Export Task to configure the file name and the OBS path for saving the file. If OBS is unavailable, you only need to set the file name.Figure 2 Export Task
- Click Next and select the resources to export. -
- Click Next and wait until the export is complete. The resource package is exported to the OBS path you have set.Figure 3 Export completed+
- Click Next and wait until the export is complete. The resource package is exported to the OBS path you have set.Figure 3 Export completed
If no result is displayed in 1 minute, the export fails. Try again. If the failure persists, contact the customer service or technical support.
- - After the export is complete, you can click Download in the row of the corresponding migration task to download the exported resource package.Figure 4 Downloading the exported result+
- After the export is complete, you can click Download in the row of the corresponding migration task to download the exported resource package.Figure 4 Downloading the exported result
Importing a Resource
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console.
- On the DataArts Studio console, locate a workspace and click Management Center.
- In the navigation pane, choose Migrate Resources.Figure 5 Migrating Resources++ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0012.html b/docs/dataartsstudio/umn/dataartsstudio_01_0012.html index 559117984..5d4927075 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0012.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0012.html @@ -1,20 +1,22 @@ -
Importing a Resource
- In the navigation pane, choose Migrate Resources.Figure 5 Migrating Resources-
- Click Create Import Task. On the displayed page, select an import mode and set the OBS bucket and path or local path that stores resources. The resource to be imported must be a .zip file exported from the console.Figure 6 Configuring the path that stores the resources to be imported+
- Click Import File. On the displayed page, select an import mode and set the OBS bucket and path or local path that stores resources. The resource to be imported must be a .zip file exported from the console.Figure 6 Configuring the path that stores the resources to be imported-
- Click Create Import Task and upload a .zip resource file that you have exported.
- Click Next and select the resources to import. -
- If you select DataSource, click Next to configure a data connection.Figure 7 Configuring a data connection-
- Click Next and wait until the import task is delivered. When the import task is delivered successfully, the system displays message "Import task started."Figure 8 Import task started+
- Click Import File and upload resources. a .zip resource file that you have exported.
- Click Next and select the resources to import. +
- If you select DataSource, click Next to configure a data connection.Figure 7 Configuring a data connection+
- Click Next and wait until the import task is delivered. When the import task is delivered successfully, the system displays message "Import task started."Figure 8 Import task started-
- Click OK. You can view the import result in the resource migration task list.Subtasks that fail are marked in red. You can click their names to view the failure causes.Figure 9 Viewing the import result+
- Click OK. You can view the import result in the resource migration task list.
@@ -38,3 +37,10 @@Subtasks that fail are marked in red. You can click their names to view the failure causes.Figure 9 Viewing the import result
- Click OK. You can view the import result in the resource migration task list.
DataArts Migration
+DataArts Migration (CDM Jobs)
- Overview
- - Constraints
+ - Notes and Constraints
- Supported Data Sources
- - Managing Clusters
+ - Creating and Managing a CDM Cluster
- - Managing Links
+ - Creating a Link in a CDM Cluster
- - Managing Jobs
+ - Creating a Job in a CDM Cluster
+
+ - Using Macro Variables of Date and Time
- Improving Migration Performance
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0013.html b/docs/dataartsstudio/umn/dataartsstudio_01_0013.html
index b65bc72df..ac3f5aee6 100644
--- a/docs/dataartsstudio/umn/dataartsstudio_01_0013.html
+++ b/docs/dataartsstudio/umn/dataartsstudio_01_0013.html
@@ -4,7 +4,7 @@
- Table/file/entire DB migration
Tables or files can be migrated in batches. An entire database can be migrated between homogeneous and heterogeneous databases. A job can migrate hundreds of tables.
- Incremental data migration
CDM supports incremental migration of files, relational databases, and HBase/CloudTable, as well as with WHERE clauses and macro variables of date and time.
@@ -17,12 +17,19 @@
DataArts Migration is an efficient and easy-to-use data integration service. Based on the big data migration to the cloud and intelligent data lake solutions, CDM provides easy-to-use migration capabilities and can integrate various types of data sources into the data lake, which simplifies data source migration and integration and improves efficiency for you.
In this document, DataArts Migration refers to Cloud Data Migration (CDM).
Introduction to CDM
CDM uses a distributed compute framework and concurrent processing techniques to help you migrate enterprise data in batches without any downtime and rapidly build desired data structures.
-Figure 1 CDM+
Figure 1 CDM
Functions
Migration Principles
When a tenant uses CDM, the CDM system provisions a fully-managed CDM instance in the tenant's VPC. The instance allows only console and RESTful API access. Therefore the tenant cannot access the instance through other interfaces (such as SSH). This ensures data isolation between CDM tenants, prevents data leakage, and ensures transmission security during data migration between different cloud services in a VPC. Tenants can also use the VPN to migrate data from the on-premises data center to cloud services to ensure migration security.
CDM works in push-pull mode. CDM pulls data from the migration source and pushes the data to the migration destination. Data access operations are initiated by CDM. SSL will be used if the data source (such as RDS) supports it. During the migration, the usernames and passwords of the migration source and destination are required. Such information is stored in the database of the CDM instance. Protecting such information is critical to ensure CDM security.
-Figure 2 Migration principles+
Figure 2 Migration principles
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0014.html b/docs/dataartsstudio/umn/dataartsstudio_01_0014.html index b812aa8e8..0660d9da5 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0014.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0014.html @@ -7,23 +7,23 @@-Parent topic: DataArts Migration+Parent topic: DataArts Migration (CDM Jobs) - In the navigation pane, choose Migrate Resources.
Data Sources Supported by Table/File Migration
Table/File migration can migrate data in tables or files.
Table 1 describes the supported data sources. -Table 1 Supported data sources during table/file migration Category
+Table 1 Supported data sources during table/file migration Category
Source
+Source
Destination
+Destination
Description
+Description
Data warehouse
+Data warehouse
Data Warehouse Service
+Data Warehouse Service
- Data warehouse: GaussDB(DWS), Data Lake Insight (DLI), and MRS ClickHouse
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- Object-based storage: Object Storage Service (OBS)
- Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
- Data warehouse: GaussDB(DWS), Data Lake Insight (DLI), and MRS ClickHouse
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- Object-based storage: Object Storage Service (OBS)
- Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
The DWS physical machine management mode is not supported.
+The DWS physical machine management mode is not supported.
Data Lake Insight (DLI)
@@ -31,27 +31,28 @@- Data warehouse: GaussDB(DWS), Data Lake Insight (DLI), and MRS ClickHouse
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- Object-based storage: Object Storage Service (OBS)
- Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
- NoSQL: CloudTable and MongoDB
- Search: Elasticsearch and Cloud Search Service (CSS)
Recommended MongoDB version: 4.2
+You must have the SELECT permission (for querying tables) on all fields of the DLI data source.
MRS ClickHouse
Data warehouse: MRS ClickHouse and Data Lake Insight (DLI)
Hadoop
+Hadoop
MRS HDFS
+MRS HDFS
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- Object-based storage: Object Storage Service (OBS)
- Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- Object-based storage: Object Storage Service (OBS)
- Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
- Supported by local storage. Only MRS Hive and MRS Hudi are supported in storage-compute decoupling scenarios.
- Only MRS Hive is supported in Ranger scenarios.
- Not supported if SSL is enabled for ZooKeeper
- Recommended MRS HDFS versions:
- 2.8.X
- 3.1.X
- Supported by local storage. Only MRS Hive and MRS Hudi are supported in storage-compute decoupling scenarios.
- Only MRS Hive is supported in Ranger scenarios.
- Not supported if SSL is enabled for ZooKeeper
- Recommended MRS HDFS versions:
- 2.8.X
- 3.1.X
- Recommended MRS HBase versions:
- 2.1.X
- 1.3.X
- MRS Hive and MRS Hudi 2.x versions are not supported. The following versions are recommended:
- 1.2.X
- 3.1.X
MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
+MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
Apache HDFS
Object storage
+Object storage
Object Storage Service (OBS)
+Object Storage Service (OBS)
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
- Object Storage Migration Service (OMS) is recommended for migration between object storage services.
- Binary files cannot be imported to a database or NoSQL.
- Object Storage Migration Service (OMS) is recommended for migration between object storage services.
- Binary files cannot be imported to a database or NoSQL.
File system
+File system
FTP
+FTP
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
- Object-based storage: Object Storage Service (OBS)
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
- Object-based storage: Object Storage Service (OBS)
- The file system cannot serve as the destination.
- Only text files such as CSV files can be migrated from FTP or SFTP servers to search services. Binary files cannot.
- Only binary files can be migrated from FTP or SFTP servers to OBS.
- obsutil is recommended for migrating data from HTTP servers to OBS.
- The file system cannot serve as the destination.
- Only text files such as CSV files can be migrated from FTP or SFTP servers to search services. Binary files cannot.
- Only binary files can be migrated from FTP or SFTP servers to OBS.
- obsutil is recommended for migrating data from HTTP servers to OBS.
SFTP
@@ -127,13 +128,13 @@Hadoop: MRS HDFS
Relational database
+Relational database
RDS for MySQL
+RDS for MySQL
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, MRS Hive, and MRS Hudi
- Object-based storage: Object Storage Service (OBS)
- NoSQL: CloudTable
- Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
- Search: Elasticsearch and Cloud Search Service (CSS)
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, MRS Hive, and MRS Hudi
- Object-based storage: Object Storage Service (OBS)
- NoSQL: CloudTable
- Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
- Search: Elasticsearch and Cloud Search Service (CSS)
- You are advised to use Data Replication Service (DRS) to migrate data between OLTP databases.
- Recommended Microsoft SQL Server version: 2005 or later
- Recommended Microsoft SQL Server version: 2005 or later
RDS for SQL Server
@@ -184,13 +185,13 @@-
NoSQL
+NoSQL
Distributed Cache Service (DCS)
+Distributed Cache Service (DCS)
Hadoop: MRS HDFS, MRS HBase, and MRS Hive
+Hadoop: MRS HDFS, MRS HBase, and MRS Hive
NoSQL except CloudTable cannot serve as the destination.
+NoSQL except CloudTable cannot serve as the destination.
Redis
@@ -212,15 +213,15 @@- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- Object-based storage: Object Storage Service (OBS)
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
Message system
+Message system
Apache Kafka
+Apache Kafka
Search: Cloud Search Service (CSS)
+Search: Cloud Search Service (CSS)
The message system cannot serve as the destination.
+The message system cannot serve as the destination.
DMS Kafka
@@ -230,18 +231,18 @@- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- Object-based storage: Object Storage Service (OBS)
- Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
- MRS Kafka cannot serve as the destination.
- Supported only by local storage and not in storage-compute decoupling scenarios.
- Not supported by Ranger.
- Not supported if SSL is enabled for ZooKeeper.
MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
+- MRS Kafka cannot serve as the destination.
- Supported only by local storage and not in storage-compute decoupling scenarios
- Not supported by Ranger
- Not supported if SSL is enabled for ZooKeeper
MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
Search
+Search
Elasticsearch
+Elasticsearch
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- Object-based storage: Object Storage Service (OBS)
- Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
- Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
- Hadoop: MRS HDFS, MRS HBase, and MRS Hive
- Object-based storage: Object Storage Service (OBS)
- Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
- NoSQL: CloudTable
- Search: Elasticsearch and Cloud Search Service (CSS)
Only the non-security mode is supported.
+Only the non-security mode is supported.
Cloud Search Service (CSS)
@@ -294,7 +295,7 @@Entire DB migration only to MRS HBase
Recommended versions:-- 2.1.X
- 1.3.X
MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
+MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
MRS Hive
@@ -306,7 +307,7 @@Entire DB migration only to a relational database
2.x versions are not supported. The following versions are recommended:-- 1.2.X
- 3.1.X
MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
+MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
FusionInsight HBase
@@ -358,9 +359,9 @@Supported
Supported only by local storage and in storage-compute decoupling scenarios
-2.x versions are not supported. The following versions are recommended:
+2.x versions are not supported. The following versions are recommended:
- 1.2.X
- 3.1.X
MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
+MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
Relational database
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0015.html b/docs/dataartsstudio/umn/dataartsstudio_01_0015.html index 7786c8cc8..7c46faf87 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0015.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0015.html @@ -1,11 +1,11 @@ -Constraints
-CDM System Constraints
- You cannot modify the flavor of an existing cluster. If you require a higher flavor, create a cluster with your desired flavor.
- Arm CDM clusters do not support agents. The CDM cluster version (Arm or x86) is determined by the architecture of underlying resources.
- CDM does not support the function of controlling the data migration speed. Therefore, do not perform data migration during peak hours.
- During data migration, CDM imposes pressure on the data source. You are advised to create a database account for data migration and configure an account policy to reduce the resource consumption of the data source. For example, you can configure a policy to delete the connections of the account when the CPU usage exceeds 30% to prevent impact on services.
- The baseline and maximum bandwidths of the NIC of the cdm.large CDM instance is 0.8 Gbit/s and 3 Gbit/s, respectively. The theoretical maximum volume of data that can be transmitted per instance per day is about 8 TB. Similarly, the baseline and maximum bandwidths of the NIC of the cdm.xlarge instance are 4 Gbit/s and 10 Gbit/s, respectively, and the theoretical maximum volume of data that can be transmitted per instance per day is about 40 TB. The baseline and maximum bandwidths of the NIC of the cdm.4xlarge instance is 36 Gbit/s and 40 Gbit/s, respectively, and the theoretical maximum volume of data that can be transmitted per instance per day is about 360 TB. You can use multiple CDM instances if you want faster data transfer.
The actual amount of data that can be migrated in a day depends on the data source type, the read and write performance of the source and destination, and the actual available bandwidth. Typically you can migrate as much as 8 TB per day (large file migration to OBS) using the cdm.large instance. It is recommended that you test the speed with a small amount of data before migration.
+Notes and Constraints
+-CDM System Notes and Constraints
- You cannot modify the flavor of an existing cluster. If you require a higher flavor, create a cluster with your desired flavor.
- The CDM cluster version (Arm or x86) is determined by the architecture of underlying resources.
- CDM does not support the function of controlling the data migration speed. Therefore, do not perform data migration during peak hours.
- During data migration, CDM imposes pressure on the data source. You are advised to create a database account for data migration and configure an account policy to reduce the resource consumption of the data source. For example, you can configure a policy to delete the connections of the account when the CPU usage exceeds 30% to prevent impact on services.
- The baseline and maximum bandwidths of the NIC of the cdm.large CDM instance is 0.8 Gbit/s and 3 Gbit/s, respectively. The theoretical maximum volume of data that can be transmitted per instance per day is about 8 TB. Similarly, the baseline and maximum bandwidths of the NIC of the cdm.xlarge instance are 4 Gbit/s and 10 Gbit/s, respectively, and the theoretical maximum volume of data that can be transmitted per instance per day is about 40 TB. The baseline and maximum bandwidths of the NIC of the cdm.4xlarge instance is 36 Gbit/s and 40 Gbit/s, respectively, and the theoretical maximum volume of data that can be transmitted per instance per day is about 360 TB. You can use multiple CDM instances if you want faster data transfer.
The actual amount of data that can be migrated in a day depends on the data source type, the read and write performance of the source and destination, and the actual available bandwidth. Typically you can migrate as much as 8 TB per day (large file migration to OBS) using the cdm.large instance. It is recommended that you test the speed with a small amount of data before migration.
- CDM supports incremental file migration (by skipping repeated files), but does not support resumable transfer.
For example, if three files are to be migrated and the second file fails to be migrated due to the network fault. When the migration task is started again, the first file is skipped. The second file, however, cannot be migrated from the point where the fault occurs, but can only be migrated again.
- During file migration, a single task supports millions of files. If there are too many files in the directory to be migrated, you are advised to split the files into different directories and create multiple tasks.
- You can export links and jobs configured on CDM to a local directory. To ensure password security, CDM does not export the link password of the corresponding data source. Therefore, before importing job configurations to CDM, you need to manually input the password in the exported JSON file or configure the password in the import dialog box.
- The cluster cannot automatically upgrade to a new version. You need to use the job export and import functions to upgrade the cluster to the new version.
- If OBS is unavailable, CDM does not automatically back up users' job configurations. You need to export and back up configuration data using the export function.
- If VPC peering connection is configured, the peer VPC subnet may overlap with the CDM management network. As a result, data sources in the peer VPC cannot be accessed. You are advised to use the public network for cross-VPC data migration, or contact the administrator to add specific routes to the VPC peering connection in the CDM background.
- If the destination of a CDM job is a DWS or NewSQL database, constraints of the source end, such as the primary key and unique index, cannot be migrated together.
- When performing a CDM job, ensure that the JSON file formats of the two clusters are the same so that jobs can be imported from the source cluster to the destination cluster.
- If a running job is interrupted unexpectedly, the data that has been written to the destination will not be deleted. You must manually delete the data if needed.
- The size of a file to be transferred cannot exceed 1 TB.
General Constraints on Database Migration
- CDM is mainly used for batch migration. It supports only limited incremental migration but does not support real-time incremental migration.
- The entire DB migration of CDM supports only data table migration but not migration of database objects such as stored procedures, triggers, functions, and views.
CDM applies only to scenarios where databases are migrated to the cloud at a time, including homogeneous and heterogeneous database migrations. CDM is not applicable to data synchronization, for example, disaster recovery and real-time synchronization.
+@@ -14,7 +14,7 @@General Notes and Constraints on Database Migration
- CDM is mainly used for batch migration. It supports only limited incremental migration but does not support real-time incremental migration.
- The entire DB migration of CDM supports only data table migration but not migration of database objects such as stored procedures, triggers, functions, and views.
CDM applies only to scenarios where databases are migrated to the cloud at a time, including homogeneous and heterogeneous database migrations. CDM is not applicable to data synchronization, for example, disaster recovery and real-time synchronization.
- If CDM fails to migrate an entire database or table, the data that has been imported to the target table will not be rolled back automatically. If you want to perform migration in transaction mode, configure the Import to Staging Table parameter to enable a rollback upon a migration failure.
In extreme cases, the created stage table or temporary table cannot be automatically deleted. You need to manually clear the table (the name of the stage table ends with _cdm_stage), for example, cdmtet_cdm_stage).
- If CDM needs to access data sources in the on-premises data center (for example, the on-premises MySQL database), the data sources must support Internet access and the CDM instances must be bound with elastic IP addresses. In this case, the security practice is to configure the firewall or security policies to allow only the EIPs of the CDM instances to access the local data sources.
- Only common data types are supported, including character strings, digits, and dates. Object types are limited. If objects are too large, migration cannot be performed.
- Only the GBK and UTF-8 character sets are supported.
- A field name cannot contain & or %.
- jdbc2hive and hive2jdbc entire DB migration is implemented by field name mapping, and is unavailable if the source and destination field names are inconsistent.
-Constraints on FusionInsight HD and Apache Hadoop
If the FusionInsight HD and Apache Hadoop data sources are deployed in the on-premises data center, CDM must access all nodes in the cluster for reading and writing the Hadoop files. Therefore, the network access must be enabled for each node.
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0020_0.html b/docs/dataartsstudio/umn/dataartsstudio_01_0020_0.html deleted file mode 100644 index fb8e763d9..000000000 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0020_0.html +++ /dev/null @@ -1,20 +0,0 @@ - - -Constraints on DWS and FusionInsight LibrA
- If the DWS primary key or table contains only one field, the field type must be a common character string, value, or date. When data is migrated from another database to DWS, if automatic table creation is selected, the primary key must be of the following types. If no primary key is set, at least one of the following fields must be set. Otherwise, the table cannot be created and the CDM job fails.
- INTEGER TYPES: TINYINT, SMALLINT, INT, BIGINT, NUMERIC/DECIMAL
- CHARACTER TYPES: CHAR, BPCHAR, VARCHAR, VARCHAR2, NVARCHAR2, TEXT
- DATA/TIME TYPES: DATE, TIME, TIMETZ, TIMESTAMP, TIMESTAMPTZ, INTERVAL, SMALLDATETIME
Constraints on GaussDB(DWS)
- If the DWS primary key or table contains only one field, the field type must be a common character string, value, or date. When data is migrated from another database to DWS, if automatic table creation is selected, the primary key must be of the following types. If no primary key is set, at least one of the following fields must be set. Otherwise, the table cannot be created and the CDM job fails.
- INTEGER TYPES: TINYINT, SMALLINT, INT, BIGINT, NUMERIC/DECIMAL
- CHARACTER TYPES: CHAR, BPCHAR, VARCHAR, VARCHAR2, NVARCHAR2, TEXT
- DATA/TIME TYPES: DATE, TIME, TIMETZ, TIMESTAMP, TIMESTAMPTZ, INTERVAL, SMALLDATETIME
For clusters of version 2.9.1.200 or earlier, the NVARCHAR2 data type is not supported for DWS.
- In DWS, the character string '' is null. A null character string cannot be inserted into a field with non-null constraints. This is inconsistent with the MySQL behavior. MySQL does not consider that '' is null. Migration from MySQL to DWS may fail due to the preceding reason.
- When the Gauss Data Service (GDS) mode is used to quickly import data to DWS, you need to configure a security group or firewall policy to allow DataNodes of DWS or FusionInsight LibrA to access port 25000 of the CDM IP address.
- When data is imported to DWS in GDS mode, CDM automatically creates a foreign table for data import. The table name ends with a universally unique identifier (UUID), for example, cdmtest_aecf3f8n0z73dsl72d0d1dk4lcir8cd. If a job fails, it will be automatically deleted. In extreme cases, you may need to manually delete it.
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0017.html b/docs/dataartsstudio/umn/dataartsstudio_01_0017.html index c417f5c24..609b584e0 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0017.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0017.html @@ -1,27 +1,27 @@ --Parent topic: DataArts Migration+Parent topic: DataArts Migration (CDM Jobs)Managing Clusters
+Creating and Managing a CDM Cluster
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0018.html b/docs/dataartsstudio/umn/dataartsstudio_01_0018.html index 9134d7a01..06550090d 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0018.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0018.html @@ -15,7 +15,7 @@- Creating a CDM Cluster
- - Binding or Unbinding an EIP
+ - Binding or Unbinding an EIP
- - Restarting a Cluster
+ - Restarting a CDM Cluster
- - Deleting a Cluster
+ - Deleting a CDM Cluster
- - Downloading Cluster Logs
+ - Downloading CDM Cluster Logs
- - Viewing Basic Cluster Information and Modifying Cluster Configurations
+ - Viewing and Modifying CDM Cluster Configurations
- - Viewing Metrics
+ - Managing and Viewing CDM Metrics
-Parent topic: DataArts Migration+Parent topic: DataArts Migration (CDM Jobs)-Parent topic: DataArts Migration+Parent topic: DataArts Migration (CDM Jobs)
Binding or Unbinding an EIP
---Scenario
After creating a CDM cluster, you can bind an EIP to or unbind an EIP from the cluster.
-If CDM needs to access a local or Internet data source, or a cloud service in another VPC, bind an EIP to the CDM cluster or use a NAT gateway to enable the CDM cluster to share the EIP with ECSs to access the Internet.
-- If SSL encryption is configured for the access channel of a local data source, CDM cannot connect to the data source using the EIP.
--Prerequisites
- You have created a CDM cluster.
- Your EIP quota is sufficient.
-Procedure
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console. On the DataArts Studio console, locate a workspace and click DataArts Migration to access the CDM console.
- Bind an EIP to or unbind an EIP from a cluster.
- Binding an EIP: In the Operation column, click Bind EIP. The Bind EIP dialog box is displayed.
- Unbinding an EIP: In the Operation column, choose .
- Click Yes.
-- diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0021_0.html b/docs/dataartsstudio/umn/dataartsstudio_01_0021_0.html deleted file mode 100644 index 140f1422b..000000000 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0021_0.html +++ /dev/null @@ -1,27 +0,0 @@ - - ---Parent topic: Managing Clusters-Viewing Basic Cluster Information and Modifying Cluster Configurations
---Scenario
After creating a CDM cluster, you can view its basic information and modify its configurations.
-- You can view the following basic cluster information:
- Cluster information: cluster version, creation time, project ID, instance ID, and cluster ID
- Instance configuration: cluster flavor, CPU, and memory
- Network configuration
- You can modify the following cluster configurations:
- Notification: If a CDM migration job (only table/file migration) fails or the EIP is abnormal, CDM sends an SMS or email notification to the user.
- User isolation: determines whether other users can operate the migration jobs or links in the cluster.
- If this function is enabled, migration jobs and links in the cluster are isolated. Other IAM users of the a cloud platform account cannot operate the jobs and links.
- If this function is disabled, migration jobs and links in the cluster can be shared by users. All IAM users with the required permission in the a cloud platform account can view and perform operations on the jobs and links in the cluster.
After disabling User Isolation, restart the cluster VM for the settings to take effect.
-
-Prerequisites
You have created a CDM cluster.
--Viewing Basic Cluster Information
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console. On the DataArts Studio console, locate a workspace and click DataArts Migration to access the CDM console.
- Click the cluster name to view its basic information. -
-Modifying Cluster Configurations
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console. On the DataArts Studio console, locate a workspace and click DataArts Migration to access the CDM console.
- Click the name of a cluster and click the Cluster Configuration tab to modify Notification and User Isolation configuration. -
- Click Save. The Cluster Management page is displayed.
- If User Isolation is disabled, choose in the Operation column to restart the cluster VM for the settings to take effect.Figure 1 Restarting a cluster-
- Restart CDM service process: Only the CDM service process is restarted. The cluster VM will not be restarted.
- VM restart: The service process will be interrupted and VMs in the cluster will be restarted.
- Select VM restart and click Yes.
-- diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0022.html b/docs/dataartsstudio/umn/dataartsstudio_01_0022.html index 1b1994e1e..63e972e77 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0022.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0022.html @@ -1,17 +1,24 @@ ---Parent topic: Managing Clusters-Downloading Cluster Logs
+Downloading CDM Cluster Logs
Scenario
This section describes how to obtain cluster logs to view the job running history and locate job failure causes.
-Prerequisites
You have created a CDM cluster.
Procedure
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console. On the DataArts Studio console, locate a workspace and click DataArts Migration to access the CDM console.
- Locate the row that contains a cluster, click More in the Operation column, and select Download Log from the drop-down list.Figure 1 Download Log+
Procedure
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console. On the DataArts Studio console, locate a workspace and click DataArts Migration to access the CDM console.
- Locate the row that contains a cluster, click More in the Operation column, and select Download Log from the drop-down list.Figure 1 Download Log
- In the displayed dialog box, click OK to download logs to a local PC.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0023.html b/docs/dataartsstudio/umn/dataartsstudio_01_0023.html index d6852a722..2f21e2cd2 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0023.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0023.html @@ -1,73 +1,21 @@ --Parent topic: Managing Clusters+Parent topic: Creating and Managing a CDM ClusterManaging Links
+Creating a Link in a CDM Cluster
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0024.html b/docs/dataartsstudio/umn/dataartsstudio_01_0024.html index 0e3fca6b7..39e585336 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0024.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0024.html @@ -1,17 +1,17 @@ --
-
- Creating Links
+ - Creating a Link Between CDM and a Data Source
- - Managing Drivers
+ - Configuring Link Parameters
- - Managing Agents
+ - Uploading a CDM Link Driver
- - Managing Cluster Configurations
-
- - Link to OBS
-
- - Link to PostgreSQL/SQLServer
-
- - Link to DWS
-
- - Link to an RDS for MySQL/MySQL Database
-
- - Link to an Oracle Database
-
- - Link to DLI
-
- - Link to Hive
-
- - Link to HBase
-
- - Link to HDFS
-
- - Link to an FTP or SFTP Server
-
- - Link to Redis
-
- - Link to DDS
-
- - Link to CloudTable
-
- - Link to MongoDB
-
- - Link to Cassandra
-
- - Link to Kafka
-
- - Link to DMS Kafka
-
- - Link to CSS
-
- - Link to Elasticsearch
-
- - Link to SAP HANA
-
- - Link to a Database Shard
-
- - Link to MRS Hudi
-
- - Link to MRS ClickHouse
-
- - Link to a ShenTong Database
-
- - Link to CloudTable OpenTSDB
-
- - Link to Doris
+ - Creating a Hadoop Cluster Configuration
-Parent topic: DataArts Migration+Parent topic: DataArts Migration (CDM Jobs)Creating Links
+Creating a Link Between CDM and a Data Source
Scenario
Before creating a data migration job, create a link to enable the CDM cluster to read data from and write data to a data source. A migration job requires a source link and a destination link. For details on the data sources that can be exported (source links) and imported (destination links) in different migration modes (table/file migration), see Supported Data Sources.
The link configurations depend on the data source. This section describes how to create these links.
-Constraints
- If changes occur in the connected data source (for example, the MRS cluster capacity is expanded), you need to edit and save the connection.
- Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
Prerequisites
- A CDM cluster is available.
- The CDM cluster can communicate with the destination data source.
- If the destination data source is an on-premises database, you need the Internet or Direct Connect. When using the Internet, ensure that an EIP has been bound to the CDM cluster, the security group of CDM allows outbound traffic from the host where the off-cloud data source is located, the host where the data source is located can access the Internet, and the connection port has been enabled in the firewall rules.
- If the destination data source is a cloud service (such as DWS, MRS, and ECS), the following requirements must be met for network interconnection:
- If the CDM cluster and the cloud service are in different regions, a public network or a dedicated connection is required for enabling communication between the CDM cluster and the cloud service. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, the host where the data source is located can access the Internet, and the port has been enabled in the firewall rules.
- If the CDM cluster and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see "Adding a Custom Route" in Virtual Private Cloud (VPC) Usage Guide. For details about how to configure security group rules, see "Security Group" > "Adding a Security Group Rule" in Virtual Private Cloud (VPC) Usage Guide.
- The cloud service instance and the CDM cluster belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
-Prerequisites
- A CDM cluster is available.
- The CDM cluster can communicate with the destination data source.
- If the destination data source is an on-premises database, you need the Internet or Direct Connect. When using the Internet, ensure that an EIP has been bound to the CDM cluster, the security group of CDM allows outbound traffic from the host where the off-cloud data source is located, the host where the data source is located can access the Internet, and the connection port has been enabled in the firewall rules.
- If the destination data source is a cloud service (such as DWS, MRS, and ECS), the following requirements must be met for network interconnection:
- If the CDM cluster and the cloud service are in different regions, a public network or a dedicated connection is required for enabling communication between the CDM cluster and the cloud service. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, the host where the data source is located can access the Internet, and the port has been enabled in the firewall rules.
- If the CDM cluster and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see "Adding Routes to a Route Table" in Virtual Private Cloud (VPC) x.x.x User Guide in Virtual Private Cloud (VPC) x.x.x Usage Guide. For details about how to configure security group rules, see "Security Group" > "Adding a Security Group Rule" in Virtual Private Cloud (VPC) x.x.x User Guide in Virtual Private Cloud (VPC) x.x.x Usage Guide.
- The cloud service instance and the CDM cluster belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
- You have obtained the URL and the account for accessing the data source. The account is granted with the read and write permissions for the data source.
- When using the Agent, you need to use the main account to grant the CDM operation permission to the sub-account.
- You have obtained the URL and the account for accessing the data source. The account is granted with the read and write permissions for the data source.
Creating Links
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console. On the DataArts Studio console, locate a workspace and click DataArts Migration to access the CDM console.
- On the CDM console, choose Cluster Management in the left navigation pane. Locate the row that contains the target cluster and click Job Management in the Operation column. On the displayed Links page, click Create Link. On the displayed page, select a connector.
The connectors are classified based on the type of the data source to be connected. All supported data types are displayed.
-Figure 1 Selecting a connector type+
Creating Links
- Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console. On the DataArts Studio console, locate a workspace and click DataArts Migration to access the CDM console.
- On the CDM console, choose Cluster Management in the left navigation pane. Locate the row that contains the target cluster and click Job Management in the Operation column. On the displayed Links page, click Create Link. On the displayed page, select a connector.
The connectors are classified based on the type of the data source to be connected. All supported data types are displayed.
+Figure 1 Selecting a connector type
- Select a data source and click Next. The following describes how to create a MySQL link.The link parameters of different data sources vary. Table 1 describes the link parameters.@@ -55,7 +63,7 @@
Table 1 Link parameters Connector
- RDS for PostgreSQL
- RDS for SQL Server
- PostgreSQL
- Microsoft SQL Server
Because the JDBC drivers used by these relational databases are the same, the parameters to be configured are also the same and are described in Link to PostgreSQL/SQLServer.
+Because the JDBC drivers used by these relational databases are the same, the parameters to be configured are also the same and are described in PostgreSQL/SQLServer Link Parameters.
Data Warehouse Service
For details about the parameters, see Link to DWS.
+For details about the parameters, see GaussDB(DWS) Link Parameters.
SAP HANA
For details about the parameters, see Link to SAP HANA.
+For details about the parameters, see SAP HANA Link Parameters.
MySQL
For details about the parameters, see Link to an RDS for MySQL/MySQL Database.
+For details about the parameters, see RDS for MySQL/MySQL Database Link Parameters.
Oracle
For details about the parameters, see Link to an Oracle Database.
+For details about the parameters, see Oracle Database Link Parameters.
Database Sharding
For details about the parameters, see Link to a Database Shard.
+For details about the parameters, see Shard Link Parameters.
Object Storage Service (OBS)
For details about the parameters, see Link to OBS.
+For details about the parameters, see OBS Link Parameters.
- MRS HDFS
- FusionInsight HDFS
- Apache HDFS
If the data source is HDFS of MRS, Apache Hadoop, or FusionInsight HD, see Link to HDFS.
+If the data source is HDFS of MRS, Apache Hadoop, or FusionInsight HD, see HDFS Link Parameters.
- MRS HBase
- FusionInsight HBase
- Apache HBase
If the data source is HBase of MRS, Apache Hadoop, or FusionInsight HD, see Link to HBase.
+If the data source is HBase of MRS, Apache Hadoop, or FusionInsight HD, see HBase Link Parameters.
- MRS Hive
- FusionInsight Hive
- Apache Hive
If the data source is Hive on MRS, Apache Hadoop, or FusionInsight HD, see Link to Hive.
+If the data source is Hive on MRS, Apache Hadoop, or FusionInsight HD, see Hive Link Parameters.
CloudTable Service
If the data source is CloudTable, see Link to CloudTable.
+If the data source is CloudTable, see CloudTable Link Parameters.
- FTP
- SFTP
If the data source is an FTP or SFTP server, see Link to an FTP or SFTP Server.
+If the data source is an FTP or SFTP server, see FTP/SFTP Link Parameters.
HTTP
@@ -87,58 +87,65 @@MongoDB
If the data source is a local MongoDB, see Link to MongoDB.
+If the data source is a local MongoDB, see MongoDB Link Parameters.
Document Database Service (DDS)
If the data source is DDS, see Link to DDS.
+If the data source is DDS, see DDS Link Parameters.
- Redis
- Distributed Cache Service
If the data source is Redis or DCS, see Link to Redis.
+If the data source is Redis or DCS, see Redis Link Parameters.
- MRS Kafka
- Apache Kafka
If the data source is MRS Kafka or Apache Kafka, see Link to Kafka.
+If the data source is MRS Kafka or Apache Kafka, see Kafka Link Parameters.
Cloud Search Service (CSS)
Elasticsearch
If the data source is CSS or Elasticsearch, see Link to CSS.
+If the data source is CSS or Elasticsearch, see CSS Link Parameters.
Data Lake Insight
If the data source is DLI, see Link to DLI.
+If the data source is DLI, see DLI Link Parameters.
DMS Kafka
If the data source is DMS Kafka, see Link to DMS Kafka.
+If the data source is DMS Kafka, see DMS Kafka Link Parameters.
Cassandra
If the data source is Cassandra, see Link to Cassandra.
+If the data source is Cassandra, see Cassandra Link Parameters.
+NOTE:Cassandra is not supported in version 2.9.3.300 or later.
+MRS Hudi
For details about the parameters, see Link to MRS Hudi.
+For details about the parameters, see MRS Hudi Link Parameters.
MRS ClickHouse
For details about the parameters, see Link to MRS ClickHouse.
+For details about the parameters, see MRS ClickHouse Link Parameters.
+LogHub (SLS)
+For details about the parameters, see LogHub (SLS) Link Parameters.
Shentong database
For details about the parameters, see Link to a ShenTong Database.
+For details about the parameters, see ShenTong Database Link Parameters.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0025.html b/docs/dataartsstudio/umn/dataartsstudio_01_0025.html index 76ed24d7d..48bc46aab 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0025.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0025.html @@ -1,8 +1,8 @@-Parent topic: Managing Links+Parent topic: Creating a Link in a CDM ClusterHow Migration Jobs Work
-Data Migration Model
Figure 1 shows the simplified migration model used by CDM.
- ++ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0026.html b/docs/dataartsstudio/umn/dataartsstudio_01_0026.html index 5647f3a79..ab120dd9f 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0026.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0026.html @@ -1,13 +1,14 @@ -Data Migration Model
Figure 1 shows the simplified migration model used by CDM.
+CDM migrates data through data migration jobs. It works in the following way:- When data migration jobs are submitted, CDM splits each job into multiple tasks based on the Concurrent Extractors parameter in the job configuration.
Jobs for different data sources may be split based on different dimensions. Some jobs may not be split based on the Concurrent Extractors parameter.
- CDM submits the tasks to the running pool in sequence. Tasks (defined by Maximum Concurrent Extractors) run concurrently. Excess tasks are queued.
Link to Hive
+Hive Link Parameters
CDM supports the following Hive data sources:- MRS Hive
- FusionInsight Hive
- Apache Hive
Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
MRS Hive
You can view a table during field mapping only when you have the permission to access the table connected to MRS Hive.
MRS Hive links apply to the MapReduce Service (MRS) on cloud. Table 1 describes related parameters.
- - Before creating an MRS Hive link, you need to add an authenticated Kerberos user on MRS and log in to the MRS management page to change the initial password. Then use the new user to create an MRS link.
- To connect to an MRS 2.x cluster, create a CDM cluster of version 2.x first. CDM 1.8.x clusters cannot connect to MRS 2.x clusters.
- Currently, the Hive link obtains the core-site.xml configuration information from MRS HDFS. Therefore, if MRS Hive uses OBS as the underlying storage system, configure the AK/SK of OBS on MRS HDFS before creating the Hive link.
- Ensure that the MRS cluster and the DataArts Studio instance can communicate with each other. The following requirements must be met for network interconnection:
- If the CDM cluster in the DataArts Studio instance and the MRS cluster are in different regions, a public network or a dedicated connection is required. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, and the MRS cluster can access the Internet and the port has been enabled in the firewall rule.
- If the CDM cluster in the DataArts Studio instance and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see "Adding a Custom Route" in Virtual Private Cloud (VPC) Usage Guide. For details about how to configure security group rules, see Security Group > Adding a Security Group Rule in Virtual Private Cloud (VPC) Usage Guide.
- The MRS cluster and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
@@ -28,7 +29,7 @@ MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
+- Before creating an MRS Hive link, you need to add an authenticated Kerberos user on MRS and log in to the MRS management page to change the initial password. Then use the new user to create an MRS link.
- To connect to an MRS 2.x cluster, create a CDM cluster of version 2.x first. CDM 1.8.x clusters cannot connect to MRS 2.x clusters.
- Currently, the Hive link obtains the core-site.xml configuration information from MRS HDFS. Therefore, if MRS Hive uses OBS as the underlying storage system, configure the AK/SK of OBS on MRS HDFS before creating the Hive link.
- Ensure that the MRS cluster and the DataArts Studio instance can communicate with each other. The following requirements must be met for network interconnection:
- If the CDM cluster in the DataArts Studio instance and the MRS cluster are in different regions, a public network or a dedicated connection is required. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, and the MRS cluster can access the Internet and the port has been enabled in the firewall rule.
- If the CDM cluster in the DataArts Studio instance and the MRS cluster are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see "Adding Routes to a Route Table" in Virtual Private Cloud (VPC) x.x.x User Guide in Virtual Private Cloud (VPC) x.x.x Usage Guide. For details about how to configure security group rules, see "Security Group" > "Adding a Security Group Rule" in Virtual Private Cloud (VPC) x.x.x User Guide in Virtual Private Cloud (VPC) x.x.x Usage Guide.
- The MRS cluster and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
Manager IP
Floating IP address of MRS Manager. Click Select next to the Manager IP text box to select an MRS cluster. CDM automatically fills in the authentication information.NOTE:DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
+DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
ldapUsername
This parameter is mandatory when Enable ldap is enabled.
+This parameter is mandatory when Enable ldap is enabled.
Enter the username configured when LDAP authentication was enabled for MRS Hive.
-
@@ -85,7 +86,7 @@ldapPassword
This parameter is mandatory when Enable ldap is enabled.
+This parameter is mandatory when Enable ldap is enabled.
Enter the password configured when LDAP authentication was enabled for MRS Hive.
-
@@ -102,7 +103,7 @@This parameter is mandatory when OBS storage support is enabled. The account corresponding to the AK/SK pair must have the OBS Buckets Viewer permission. Otherwise, OBS cannot be accessed and the "403 AccessDenied" error is reported.
You need to create an access key for the current account and obtain an AK/SK pair.
-- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.
+
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.NOTE:
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
This parameter is mandatory when OBS storage support is enabled. The account corresponding to the AK/SK pair must have the OBS Buckets Viewer permission. Otherwise, OBS cannot be accessed and the "403 AccessDenied" error is reported.
You need to create an access key for the current account and obtain an AK/SK pair.
-- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.Figure 2 Clicking Create Access Key-
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.NOTE:
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 2. +
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.NOTE:
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
This parameter is mandatory when OBS storage support is enabled. The account corresponding to the AK/SK pair must have the OBS Buckets Viewer permission. Otherwise, OBS cannot be accessed and the "403 AccessDenied" error is reported.
You need to create an access key for the current account and obtain an AK/SK pair.
-- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.Figure 3 Clicking Create Access Key-
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.NOTE:
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 3. +
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.NOTE:
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
Run Mode
This parameter is used only when the Hive version is HIVE_3_X. Possible values are:
-- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- Standalone: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, Standalone prevails.NOTE:
The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
+- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- Standalone: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, Standalone prevails.NOTE:
The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0027.html b/docs/dataartsstudio/umn/dataartsstudio_01_0027.html index d1672dead..af589d236 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0027.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0027.html @@ -1,6 +1,6 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to CloudTable
+CloudTable Link Parameters
When connecting CDM to CloudTable, configure the parameters as described in Table 1.
@@ -80,7 +80,7 @@ Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0028.html b/docs/dataartsstudio/umn/dataartsstudio_01_0028.html index bc9226973..29442a76d 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0028.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0028.html @@ -1,6 +1,6 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to an FTP or SFTP Server
+FTP/SFTP Link Parameters
The FTP/SFTP link is used to migrate files from the on-premises file server or ECS to a database.
@@ -48,6 +48,14 @@ - Only FTP servers running Linux are supported.
- Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
-
FTP File Name controlEncoding
+This parameter is available for a FTP link.
+It indicates the controlEncoding file name encoding configuration of ftp-client. The value can be ISO-8859-1 or UFT8. The default value is ISO-8859-1.
+ISO-8859-1
+
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0030.html b/docs/dataartsstudio/umn/dataartsstudio_01_0030.html index 674abaefe..fdd9b05b9 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0030.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0030.html @@ -1,9 +1,9 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to MongoDB
+MongoDB Link Parameters
This link is used to transfer data from a third-party cloud MongoDB service or MongoDB created in the on-premises data center or ECS to a big data platform.
When connecting CDM to an on-premises MongoDB database, configure the parameters as described in Table 1.
- Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
+ - MongoDB data sources with SSL enabled are not supported.
- Do not change the password or user when a job is running. If you do so, the password will not take effect immediately and the job will fail.
Table 1 MongoDB link parameters Parameter
@@ -72,7 +72,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0031.html b/docs/dataartsstudio/umn/dataartsstudio_01_0031.html index 41f19f56c..c91717417 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0031.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0031.html @@ -1,9 +1,9 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to DDS
+DDS Link Parameters
The DDS link is used to synchronize data from Document Database Service (DDS) on the cloud to a big data platform.
When connecting CDM to DDS, configure the parameters as described in Table 1.
- Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
+ - DDS data sources with SSL enabled are not supported.
- Do not change the password or user when a job is running. If you do so, the password will not take effect immediately and the job will fail.
Table 1 DDS link parameters Parameter
@@ -64,7 +64,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0032.html b/docs/dataartsstudio/umn/dataartsstudio_01_0032.html index 7b512ccdf..cc0f8ff42 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0032.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0032.html @@ -1,6 +1,6 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to Redis
+Redis Link Parameters
The Redis link is applicable to data migration of Redis created in the local data center or ECS. It is used to load data in the database or files to Redis.
Links to Redis data encrypted using SSL are not supported.
When connecting CDM to an on-premises Redis database, configure the parameters as described in Table 1.
@@ -84,7 +84,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0033.html b/docs/dataartsstudio/umn/dataartsstudio_01_0033.html index 428c4e94d..8e04cac37 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0033.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0033.html @@ -1,6 +1,6 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to Kafka
+Kafka Link Parameters
MRS Kafka
When connecting CDM to Kafka of MRS, configure the parameters as described in Table 1.
@@ -22,7 +22,7 @@ Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
Manager IP
Floating IP address of MRS Manager. Click Select next to the Manager IP text box to select an MRS cluster. CDM automatically fills in the authentication information.NOTE:DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
+DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
Table 2 Parameter description Parameter
Description
+Description
Example Value
+Example Value
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0035.html b/docs/dataartsstudio/umn/dataartsstudio_01_0035.html index 3110c2d36..fb3ca8f78 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0035.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0035.html @@ -1,6 +1,6 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to CSS
+CSS Link Parameters
Cloud Search Service (CSS) is a fully hosted distributed search service powered by open-source Elasticsearch. CSS links can be used to migrate log files and database records to CSS for search and analysis using Elasticsearch.
@@ -52,7 +52,7 @@ - Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
HTTPS Access
This parameter is displayed when Security Mode Authentication is set to Yes. This parameter specifies whether to enable HTTPS access. HTTPS access is more secure than HTTP access.
+This parameter is displayed when Security Mode Authentication is set to Yes. This parameter specifies whether to enable HTTPS access. HTTPS access is more secure than HTTP access.
Yes
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0036.html b/docs/dataartsstudio/umn/dataartsstudio_01_0036.html index 40a243425..9b2b8dd5a 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0036.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0036.html @@ -1,8 +1,9 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to DLI
+DLI Link Parameters
When connecting CDM to DLI, configure the parameters as described in Table 1.
- Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
+ Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
+- When data is migrated to DLI, DLI generates data files in the dli-trans* temporary OBS bucket. Therefore, you need to grant the user who uses the AK/SK the permissions to read and write the dli-trans* bucket and create directories. Otherwise, the migration will fail. For details about how to add permission policies for temporary bucket dli-trans*, see Adding an Authorization Policy for the dli-trans* Temporary Bucket.
+Table 1 DLI link parameters Parameter
@@ -24,7 +25,7 @@AK/SK required for authentication during access to the DLI database.
You need to create an access key for the current account and obtain an AK/SK pair.
-- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.
+
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.NOTE:
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
Project ID
Project ID in the region where DLI resides
-A project is a group of tenant resources, and an account ID corresponds to the current account. The IAM ID corresponds to the current user. You can view the project IDs, account IDs, and user IDs in different regions on the corresponding pages.- Register with and log in to the management console.
- Hover the cursor on the username in the upper right corner and select My Credentials from the drop-down list.
- On the API Credentials page, obtain the account name, account ID, IAM username, and IAM user ID, and obtain the project ID from the project list.
A project is a group of tenant resources, and an account ID corresponds to the current account. The IAM ID corresponds to the current user. You can view the project IDs, account IDs, and user IDs in different regions on the corresponding pages.- Register with and log in to the management console.
- Hover the cursor on the username in the upper right corner and select My Credentials from the drop-down list.
- On the API Credentials page, obtain the account name, account ID, IAM username, and IAM user ID, and obtain the project and its ID from the project list.
-
Batch Size
+Number of rows written each time. When the number of rows written reaches the value of Commit Size, the rows will be committed to the database.
+50000
+
Adding an Authorization Policy for the dli-trans* Temporary Bucket
- Log in to the IAM console.
- In the navigation pane, choose Permissions > Policies/Roles and click Create Custom Policy in the upper right corner.Figure 2 Creating a custom policy+
- On the Create Custom Policy page, select JSON for Policy View and create custom policy obs_dli-trans.
{ + "Version": "1.1", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "obs:object:GetObject", + "obs:object:DeleteObjectVersion", + "obs:bucket:GetBucketLocation", + "obs:object:GetAccessLabel", + "obs:bucket:PutEncryptionConfiguration", + "obs:bucket:PutBucketStoragePolicy", + "obs:object:DeleteAccessLabel", + "obs:bucket:PutBucketCustomDomainConfiguration", + "obs:bucket:GetLifecycleConfiguration", + "obs:bucket:PutBucketInventoryConfiguration", + "obs:bucket:DeleteDirectColdAccessConfiguration", + "obs:object:AbortMultipartUpload", + "obs:bucket:PutBucketLogging", + "obs:bucket:DeleteBucketWebsite", + "obs:object:DeleteObject", + "obs:bucket:PutBucketVersioning", + "obs:bucket:GetBucketWebsite", + "obs:bucket:GetBucketLogging", + "obs:bucket:DeleteBucketCustomDomainConfiguration", + "obs:object:PutObject", + "obs:object:RestoreObject", + "obs:bucket:PutReplicationConfiguration", + "obs:bucket:GetBucketQuota", + "obs:object:GetObjectVersionAcl", + "obs:bucket:DeleteBucket", + "obs:bucket:CreateBucket", + "obs:bucket:GetDirectColdAccessConfiguration", + "obs:bucket:PutDirectColdAccessConfiguration", + "obs:bucket:GetBucketAcl", + "obs:bucket:GetBucketVersioning", + "obs:bucket:GetBucketInventoryConfiguration", + "obs:bucket:GetBucketStoragePolicy", + "obs:bucket:GetEncryptionConfiguration", + "obs:bucket:PutBucketCORS", + "obs:bucket:PutBucketTagging", + "obs:bucket:GetBucketTagging", + "obs:bucket:PutLifecycleConfiguration", + "obs:bucket:GetBucketCustomDomainConfiguration", + "obs:object:ListMultipartUploadParts", + "obs:object:ModifyObjectMetaData", + "obs:bucket:ListBucketVersions", + "obs:bucket:PutBucketQuota", + "obs:object:PutAccessLabel", + "obs:bucket:ListBucket", + "obs:bucket:GetBucketCORS", + "obs:bucket:DeleteBucketInventoryConfiguration", + "obs:object:GetObjectVersion", + "obs:bucket:PutBucketWebsite", + "obs:bucket:DeleteReplicationConfiguration", + "obs:object:GetObjectAcl", + "obs:bucket:GetBucketNotification", + "obs:bucket:PutBucketNotification", + "obs:bucket:GetReplicationConfiguration", + "obs:bucket:GetBucketPolicy", + "obs:bucket:DeleteBucketTagging", + "obs:bucket:GetBucketStorage" + ], + "Resource": [ + "OBS:*:*:object:*", + "OBS:*:*:bucket:dli-trans*" + ] + } + ] +}+Figure 3 Creating custom policy obs_dli-trans+ +
- Click OK.
- In the navigation pane, choose User Groups, locate the user group to which the DLI link user using the AK/SK belongs, and click Authorize to assign the custom obs_dli-trans policy to the user.Figure 4 Assigning the custom obs_dli-trans policy to a user group+ +
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0037.html b/docs/dataartsstudio/umn/dataartsstudio_01_0037.html index 591b9ebb8..419955d29 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0037.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0037.html @@ -1,6 +1,6 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to CloudTable OpenTSDB
+CloudTable OpenTSDB Link Parameters
When connecting CDM to CloudTable OpenTSDB, configure the parameters as described in Table 1.
@@ -69,7 +80,14 @@Table 1 CloudTable OpenTSDB link parameters Parameter
@@ -36,7 +36,7 @@Project ID
Project ID in the region where CloudTable resides
-A project is a group of tenant resources, and an account ID corresponds to the current account. The IAM ID corresponds to the current user. You can view the project IDs, account IDs, and user IDs in different regions on the corresponding pages.- Register with and log in to the management console.
- Hover the cursor on the username in the upper right corner and select My Credentials from the drop-down list.
- On the API Credentials page, obtain the account name, account ID, IAM username, and IAM user ID, and obtain the project ID from the project list.
A project is a group of tenant resources, and an account ID corresponds to the current account. The IAM ID corresponds to the current user. You can view the project IDs, account IDs, and user IDs in different regions on the corresponding pages.- Register with and log in to the management console.
- Hover the cursor on the username in the upper right corner and select My Credentials from the drop-down list.
- On the API Credentials page, obtain the account name, account ID, IAM username, and IAM user ID, and obtain the project and its ID from the project list.
-
@@ -53,7 +53,7 @@AK and SK for accessing CloudTable.
You need to create an access key for the current account and obtain an AK/SK pair.
-- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.
+
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.NOTE:
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0038.html b/docs/dataartsstudio/umn/dataartsstudio_01_0038.html index 585921cfe..b16f140ce 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0038.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0038.html @@ -1,6 +1,6 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to DMS Kafka
+DMS Kafka Link Parameters
When connecting CDM to DMS Kafka, configure the parameters as described in Table 1.
@@ -72,7 +72,7 @@ Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0039.html b/docs/dataartsstudio/umn/dataartsstudio_01_0039.html index 6c6de3cfa..05aa23976 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0039.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0039.html @@ -1,12 +1,16 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to HBase
+HBase Link Parameters
CDM supports the following HBase data sources:- MRS HBase
- FusionInsight HBase
- Apache HBase
Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
MRS HBase
When connecting CDM to HBase of MRS, configure the parameters as described in Table 1.
- - Before creating an MRS link, you need to add an authenticated Kerberos user on MRS and log in to the MRS management page to change the initial password. Then use the new user to create an MRS link.
- To connect to an MRS 2.x cluster, create a CDM cluster of version 2.x first. CDM 1.8.x clusters cannot connect to MRS 2.x clusters.
- If the connection fails after you select a cluster, check whether the MRS cluster can communicate with the CDM instance which functions as the agent. They can communicate with each other in the following scenarios:
- If the CDM cluster in the DataArts Studio instance and the MRS cluster are in different regions, a public network or a dedicated connection is required. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, and the MRS cluster can access the Internet and the port has been enabled in the firewall rule.
- If the CDM cluster in the DataArts Studio instance and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see "Adding a Custom Route" in Virtual Private Cloud (VPC) Usage Guide. For details about how to configure security group rules, see "Security Group" > "Adding a Security Group Rule" in Virtual Private Cloud (VPC) Usage Guide.
- The MRS cluster and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
@@ -27,7 +31,7 @@ MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
+- Before creating an MRS link, you need to add an authenticated Kerberos user on MRS and log in to the MRS management page to change the initial password. Then use the new user to create an MRS link.
- To connect to an MRS 2.x cluster, create a CDM cluster of version 2.x first. CDM 1.8.x clusters cannot connect to MRS 2.x clusters.
- If the connection fails after you select a cluster, check whether the MRS cluster can communicate with the CDM instance which functions as the agent. They can communicate with each other in the following scenarios:
- If the CDM cluster in the DataArts Studio instance and the MRS cluster are in different regions, a public network or a dedicated connection is required. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, and the MRS cluster can access the Internet and the port has been enabled in the firewall rule.
- If the CDM cluster in the DataArts Studio instance and the MRS cluster are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see "Adding Routes to a Route Table" in Virtual Private Cloud (VPC) x.x.x User Guide in Virtual Private Cloud (VPC) x.x.x Usage Guide. For details about how to configure security group rules, see "Security Group" > "Adding a Security Group Rule" in Virtual Private Cloud (VPC) x.x.x User Guide in Virtual Private Cloud (VPC) x.x.x Usage Guide.
- The MRS cluster and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
+ If an agent is connected to multiple MRS clusters and one of the MRS clusters is deleted or abnormal, connections to the other MRS clusters will be affected. Therefore, you are advised to connect an agent to only one MRS cluster.
+
Manager IP
Floating IP address of MRS Manager. Click Select next to the Manager IP text box to select an MRS cluster. CDM automatically fills in the authentication information.NOTE:DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
+DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
Authentication Method
Authentication method used for accessing the cluster:- SIMPLE: Select this for non-security mode.
- KERBEROS: Select this for security mode.
Authentication method used for accessing the cluster:- SIMPLE: Select this for non-security mode.
- KERBEROS: Select this for security mode.
Kerberos
@@ -219,7 +223,7 @@Authentication Method
- SIMPLE: Select this for non-security mode.
- KERBEROS: Select this for security mode.
Authentication method used for accessing the cluster:- SIMPLE: Select this for non-security mode.
- KERBEROS: Select this for security mode.
Kerberos
@@ -244,7 +248,7 @@Run Mode
Run mode of the HBase link. This parameter is used only when the HBase version is HBASE_2_X.
-- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- Standalone: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, Standalone prevails.NOTE:
The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
+- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- Standalone: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, Standalone prevails.NOTE:
The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0040.html b/docs/dataartsstudio/umn/dataartsstudio_01_0040.html index 033b4fd7a..8980eb24b 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0040.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0040.html @@ -1,12 +1,16 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to HDFS
+HDFS Link Parameters
CDM supports the following HDFS data sources:- MRS HDFS
- FusionInsight HDFS
- Apache HDFS
Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
MRS HDFS
When connecting CDM to HDFS of MRS, configure the parameters as described in Table 1.
- - Before creating an MRS link, you need to add an authenticated Kerberos user on MRS and log in to the MRS management page to change the initial password. Then use the new user to create an MRS link.
- To connect to an MRS 2.x cluster, create a CDM cluster of version 2.x first. CDM 1.8.x clusters cannot connect to MRS 2.x clusters.
- If the connection fails after you select a cluster, check whether the MRS cluster can communicate with the CDM instance which functions as the agent. They can communicate with each other in the following scenarios:
- If the CDM cluster in the DataArts Studio instance and the MRS cluster are in different regions, a public network or a dedicated connection is required. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, and the MRS cluster can access the Internet and the port has been enabled in the firewall rule.
- If the CDM cluster in the DataArts Studio instance and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see "Adding a Custom Route" in Virtual Private Cloud (VPC) Usage Guide. For details about how to configure security group rules, see "Security Group" > "Adding a Security Group Rule" in Virtual Private Cloud (VPC) Usage Guide.
- The MRS cluster and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
@@ -27,7 +31,7 @@ MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.
+- Before creating an MRS link, you need to add an authenticated Kerberos user on MRS and log in to the MRS management page to change the initial password. Then use the new user to create an MRS link.
- To connect to an MRS 2.x cluster, create a CDM cluster of version 2.x first. CDM 1.8.x clusters cannot connect to MRS 2.x clusters.
- If the connection fails after you select a cluster, check whether the MRS cluster can communicate with the CDM instance which functions as the agent. They can communicate with each other in the following scenarios:
- If the CDM cluster in the DataArts Studio instance and the MRS cluster are in different regions, a public network or a dedicated connection is required. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, and the MRS cluster can access the Internet and the port has been enabled in the firewall rule.
- If the CDM cluster in the DataArts Studio instance and the MRS cluster are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see "Adding Routes to a Route Table" in Virtual Private Cloud (VPC) x.x.x User Guide in Virtual Private Cloud (VPC) x.x.x Usage Guide. For details about how to configure security group rules, see "Security Group" > "Adding a Security Group Rule" in Virtual Private Cloud (VPC) x.x.x User Guide in Virtual Private Cloud (VPC) x.x.x Usage Guide.
- The MRS cluster and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
+ If an agent is connected to multiple MRS clusters and one of the MRS clusters is deleted or abnormal, connections to the other MRS clusters will be affected. Therefore, you are advised to connect an agent to only one MRS cluster.
+
Manager IP
Floating IP address of MRS Manager. Click Select next to the Manager IP text box to select an MRS cluster. CDM automatically fills in the authentication information.NOTE:DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
+DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
Run Mode
Run mode of the HDFS link. The options are as follows:- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- STANDALONE: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, select STANDALONE or configure different agents.
Note: The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
- - Agent: The link instance runs on an agent.
Run mode of the HDFS link. The options are as follows:-- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- STANDALONE: The link instance runs in an independent process. If you want to connect CDM to multiple Hadoop data sources (MRS, Hadoop, or CloudTable), and both KERBEROS and SIMPLE authentication modes are available, you must select STANDALONE for this parameter.
Note: The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
+
If Agent is not used, and the CDM cluster connects to two or more clusters with Kerberos authentication enabled and the same realm, only one cluster can be connected in EMBEDDED mode, and the other clusters must be in STANDALONE mode.
+If a CDM cluster connects to two or more clusters with Kerberos authentication enabled and the same realm, only one cluster can be connected in EMBEDDED mode, and the other clusters must be connected in STANDALONE mode.
STANDALONE
Agent
Click Select and select the agent created in Connecting to an Agent. This parameter is displayed when Run Mode is set to Agent.
+The agent function will be unavailable soon and does not need to be configured.
-
Run Mode
Run mode of the HDFS link. The options are as follows:- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- STANDALONE: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, select STANDALONE or configure different agents.
Note: The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
- - Agent: The link instance runs on an agent.
Run mode of the HDFS link. The options are as follows:- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- STANDALONE: The link instance runs in an independent process. If you want to connect CDM to multiple Hadoop data sources (MRS, Hadoop, or CloudTable), and both KERBEROS and SIMPLE authentication modes are available, you must select STANDALONE for this parameter.
Note: The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
+
STANDALONE
@@ -168,7 +172,7 @@Agent
Click Select and select the agent created in Connecting to an Agent. This parameter is displayed when Run Mode is set to Agent.
+The agent function will be unavailable soon and does not need to be configured.
-
Run Mode
Run mode of the HDFS link. The options are as follows:- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- STANDALONE: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, select STANDALONE or configure different agents.
Note: The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
- - Agent: The link instance runs on an agent. For Apache HDFS, you can select Agent only if Authentication Method is set to SIMPLE.
Run mode of the HDFS link. The options are as follows:- EMBEDDED: The link instance runs with CDM. This mode delivers better performance.
- STANDALONE: The link instance runs in an independent process. If you want to connect CDM to multiple Hadoop data sources (MRS, Hadoop, or CloudTable), and both KERBEROS and SIMPLE authentication modes are available, you must select STANDALONE for this parameter.
Note: The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
+
STANDALONE
@@ -245,7 +249,7 @@Agent
This parameter is required when Authentication Method is set to SIMPLE and Run Mode is set to Agent. Select the agent created in Connecting to an Agent.
+The agent function will be unavailable soon and does not need to be configured.
-
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0044.html b/docs/dataartsstudio/umn/dataartsstudio_01_0044.html index 9b7344f9c..e07473d23 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0044.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0044.html @@ -1,7 +1,7 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to PostgreSQL/SQLServer
-Table 1 lists the parameters for creating a link to PostgreSQL/SQLServer.
+PostgreSQL/SQLServer Link Parameters
+Table 1 lists the parameters for creating a link to PostgreSQL/SQLServer. Greenplum, Kingbase, and GaussDB can be connected through the PostgreSQL connector. The source and destination data sources supported by migration jobs are the same as those for PostgreSQL.
@@ -60,16 +60,31 @@ Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
Use Agent
Whether to extract data from the data source through an agent
+The agent function will be unavailable soon and does not need to be configured.
Yes
+-
Agent
Click Select and select the agent created in Managing Agents.
+The agent function will be unavailable soon and does not need to be configured.
-
+-
+Driver Class Name
+Class name of the uploaded driver
+Select org.postgresql.Driver or com.kingbase8.Driver.
+-
+Reference Sign
+(Optional) Delimiter between the names of the referenced tables or columns. For details, see the product documentation of the corresponding database.
+"
Driver Version
@@ -87,20 +102,36 @@1000
Commit Size
+(Optional) Displayed when you click Show Advanced Attributes.
+Number of records submitted each time. Set this parameter based on the destination and data size of the job. If the value is too large or too small, the job execution time may be affected.
+10000
+SSL Encryption
+Whether to connect to the database in SSL mode
+Yes
+Link Attributes
(Optional) Click Add to add the JDBC connector attributes of multiple specified data sources. For details, see the JDBC connector document of the corresponding database.
-The following are some examples:- connectTimeout=60 and socketTimeout=300: When a large amount of data needs to be migrated or the entire table is retrieved using query statements, the migration fails due to connection timeout. In this case, you can customize the connection timeout interval (s) and socket timeout interval (s) to prevent failures caused by timeout.
- useCursorFetch=false: By default, useCursorFetch is enabled, indicating that the JDBC connector communicates with relational databases using a binary protocol. Some third-party systems may have compatibility issues, causing migration time conversion errors. In this case, you can disable this function.
- trustServerCertificate=true: A PKIX error may be reported during the creation of a secure connection. You are advised to set this parameter to true.
The following are some examples:- connectTimeout=60 and socketTimeout=300: When a large amount of data needs to be migrated or the entire table is retrieved using query statements, the migration fails due to connection timeout. In this case, you can customize the connection timeout interval (s) and socket timeout interval (s) to prevent failures caused by timeout.
- useCursorFetch=false: By default, useCursorFetch is enabled, indicating that the JDBC connector communicates with relational databases using a binary protocol. Some third-party systems may have compatibility issues, causing migration time conversion errors. In this case, you can disable this function.
- trustServerCertificate=true: A PKIX error may be reported during the creation of a secure connection. You are advised to set this parameter to true.
- sslmode=require: The link to PostgreSQL may fail when SSL authentication is enabled. Set this parameter to require.
sslmode=require
Reference Sign
+Link Secret Attributes
(Optional) Delimiter between the names of the referenced tables or columns. For details, see the product documentation of the corresponding database.
+(Optional) Displayed when you click Show Advanced Attributes.
+Custom secret attributes of the link
"
+sk=09fUgD5WOF1L6f
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0045.html b/docs/dataartsstudio/umn/dataartsstudio_01_0045.html index e2d990928..465407c8c 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0045.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0045.html @@ -1,6 +1,6 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to OBS
+OBS Link Parameters
When connecting CDM to the destination OBS bucket, you need to add the read and write permissions to the destination OBS bucket, and file authentication is not required.
@@ -27,7 +27,7 @@ - If the CDM cluster and OBS bucket are not in the same region, the CDM cluster cannot access the OBS bucket.
- Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
NOTE:- If the CDM cluster and OBS bucket are not in the same region, the CDM cluster cannot access the OBS bucket.
- Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
-
+obs.myregion.mycloud.com
Port
@@ -48,7 +48,7 @@AK and SK are used to log in to the OBS server.
You need to create an access key for the current account and obtain an AK/SK pair.
-To obtain an access key, perform the following steps:- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.
+To obtain an access key, perform the following steps:
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 1.
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.NOTE:
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
-
Link Attributes
+(Optional) Displayed when you click Show Advanced Attributes.
+You can click Add to add custom attributes for the link.
+Only connectionTimeout, socketTimeout, and idleConnectionTime are supported.
+The following are some examples:
+- socketTimeout: timeout interval for data transmission at the socket layer, in milliseconds
- connectionTimeout: timeout interval for establishing an HTTP/HTTPS connection, in milliseconds
-
++ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_004501.html b/docs/dataartsstudio/umn/dataartsstudio_01_004501.html index a09a208aa..758110eba 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_004501.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_004501.html @@ -1,7 +1,7 @@ --Parent topic: Managing Links+Parent topic: Configuring Link ParametersLink to Cassandra
- Do not change the password or user when the job is running. If you do so, the password will not take effect immediately and the job will fail.
+Cassandra Link Parameters
+ - Cassandra is not supported in version 2.9.3.300 or later.
- Do not change the password or user when a job is running. If you do so, the password will not take effect immediately and the job will fail.
Table 1 Parameter description Parameter
@@ -69,7 +69,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0046.html b/docs/dataartsstudio/umn/dataartsstudio_01_0046.html index ee5b2e959..649ec8e99 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0046.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0046.html @@ -5,9 +5,9 @@-Parent topic: Managing Links+Parent topic: Configuring Link Parameters-Constraints
- The dirty data recording function depends on OBS.
- The JSON file of a job to be imported cannot exceed 1 MB.
- The size of a file to be transferred cannot exceed 1 TB.
- Field names of the source and destination parameters cannot contain ampersands (&) or number signs (%).
Prerequisites
- A link has been created. For details, see Creating Links.
- The CDM cluster can communicate with the data source.
-Prerequisites
- A link has been created. For details, see Creating a Link Between CDM and a Data Source.
- The CDM cluster can communicate with the data source.
Procedure
- Log in to the management console and choose Service List > Cloud Data Migration. In the left navigation pane, choose Cluster Management. Locate the target cluster and click Job Management.
- Choose . The page for configuring the job is displayed.Figure 1 Creating a migration job+
Procedure
- Log in to the management console and choose Service List > Cloud Data Migration. In the left navigation pane, choose Cluster Management. Locate the target cluster and click Job Management.
- Choose . The page for configuring the job is displayed.Figure 1 Creating a migration job
- Select the source and destination links.
- Job Name: Enter a string consisting of 1 to 240 characters. The name can contain digits, letters, hyphens (-), underscores (_), and periods (.), and cannot start with a hyphen (-) or period (.). An example value is oracle2rds_t.
- Source Link Name: Select the data source from which data will be exported.
- Destination Link Name: Select the data source to which data will be imported.
- Configure the source link parameters.
The parameters vary with data sources. For details about the job parameters of other types of data sources, see Table 1 and Table 2.
@@ -52,7 +52,7 @@Data can be exported from DLI.
For details, see From DLI.
+For details, see From DLI.
- FTP
- SFTP
- MongoDB
- Document Database Service
Data can be exported from MongoDB or DDS.
+NOTE:MongoDB and DDS data sources with SSL enabled are not supported.
+For details, see From MongoDB/DDS.
For details, see From MRS ClickHouse.
LogHub (SLS)
+Data can be exported from LogHub (SLS).
+For details, see From LogHub (SLS).
+ShenTong database
Data can be exported from a ShenTong database.
@@ -255,6 +264,8 @@MongoDB
Data can be rapidly imported to MongoDB.
+NOTE:MongoDB data sources with SSL enabled are not supported.
+For details, see To MongoDB.
- After the parameters are configured, click Next. The Map Field tab page is displayed.
If files are migrated between FTP, SFTP, OBS, and HDFS and the migration source's File Format is set to Binary, files will be directly transferred, free from field mapping.
In other scenarios, CDM automatically maps fields of the source table and the destination table. You need to check whether the mapping and time format are correct. For example, check whether the source field type can be converted into the destination field type.
-Figure 2 Field mapping-
- If the fields from the source and destination do not match, you can drag the fields to make adjustments.
- On the Map Field tab page, if CDM fails to obtain all columns by obtaining sample values (for example, when data is exported from HBase, CloudTable, or MongoDB, or when data is migrated from SFTP/FTP to DLI, there is a high probability that CDM failed to obtain all columns), you can click
and select Add a new field to add new fields to ensure that the data imported to the migration destination is complete. - When a relational database, Hive, DLI, or MRS Hudi is used as the migration source, sample values cannot be obtained.
- On the Map Field page, you can click
to add custom constants, variables, and expressions. - Column names are displayed when the source of the migration job is OBS, CSV files are to be migrated, and parameter Extract first row as columns is set to Yes.
- When SQLServer is the destination, fields of the timestamp type cannot be written. You must change their type (for example, to datatime) so that they can be written.
- When Hive serves as the source, data of the array and map types can be read.
- Field mapping is not involved when the binary format is used to migrate files to files.
- If the data is imported to DWS, you need to select the distribution columns in the destination fields. You are advised to select the distribution columns according to the following principles:
- Use the primary key as the distribution column.
- If multiple data segments are combined as primary keys, specify all primary keys as the distribution column.
- In the scenario where no primary key is available, if no distribution column is selected, DWS uses the first column as the distribution column by default. As a result, data skew risks exist.
Figure 2 Field mapping+
- - If the fields from the source and destination do not match, you can drag the fields to make adjustments.
- On the Map Field tab page, if CDM fails to obtain all columns by obtaining sample values (for example, when data is exported from HBase, CloudTable, or MongoDB, or when data is migrated from SFTP/FTP to DLI, there is a high probability that CDM failed to obtain all columns), you can click
and select Add a new field to add new fields to ensure that the data imported to the migration destination is complete. - When a relational database, Hive, DLI, or MRS Hudi is used as the migration source, sample values cannot be obtained.
- On the Map Field page, you can click
to add custom constants, variables, and expressions. - Column names are displayed when the source of the migration job is OBS, CSV files are to be migrated, and parameter Extract first row as columns is set to Yes.
- When SQLServer is the destination, fields of the timestamp type cannot be written. You must change their type (for example, to datatime) so that they can be written.
- When Hive serves as the source, data of the array and map types can be read.
- Field mapping is not involved when the binary format is used to migrate files to files.
- If the data is imported to DWS, you need to select the distribution columns in the destination fields. You are advised to select the distribution columns according to the following principles:
- Use the primary key as the distribution column.
- If multiple data segments are combined as primary keys, specify all primary keys as the distribution column.
- In the scenario where no primary key is available, if no distribution column is selected, DWS uses the first column as the distribution column by default. As a result, data skew risks exist.
- CDM supports field conversion. Click
and then click Create Converter.Figure 3 Creating a converter+
- CDM supports field conversion. Click
and then click Create Converter.Figure 3 Creating a converter+
CDM supports the following converters:
- Anonymization: hides key data in the character string.For example, if you want to convert 12345678910 to 123****8910, configure the parameters as follows:
- Set Reserve Start Length to 3.
- Set Reserve End Length to 4.
- Set Replace Character to *.
- Trim automatically deletes the spaces before and after the character string.
- Reverse string automatically reverses a character string. For example, reverse ABC into CBA.
- Replace string replaces the specified character string.
- Expression conversion uses the JSP expression language (EL) to convert the current field or a row of data.
- Remove line break deletes the newline characters, such as \n, \r, and \r\n from the field.
- If Use SQL Statement is set to Yes in the source job configuration, converters cannot be created.
- Anonymization: hides key data in the character string.
- Click Next, set job parameters, and click Show Advanced Attributes to display and configure optional parameters.Figure 4 Task parameters+
- Click Next, set job parameters, and click Show Advanced Attributes to display and configure optional parameters.Figure 4 Task parameters
Table 3 describes related parameters.-Table 3 Parameter description Parameter
Schedule Execution
If you select Yes, you can set the start time, cycle, and validity period of a job. For details, see Scheduling Job Execution.
+If you select Yes, you can set the start time, cycle, and validity period of a job. For details, see Configuring a Scheduled CDM Job.
NOTE:If you use DataArts Studio DataArts Factory to schedule the CDM migration job and configure this parameter, both configurations take effect. To ensure unified service logic and avoid scheduling conflicts, enable job scheduling in DataArts Factory and do not configure a scheduled task for the job in DataArts Migration.
Concurrent Extractors
Configure the number of tasks to be split from a CDM job.
+Maximum number of threads of the job for reading data from the source
+NOTE:The number of concurrent threads may be less than or equal to the value of this parameter for some data sources that do not support concurrent extraction, for example, CSS and ClickHouse.
+CDM migrates data through data migration jobs. It works in the following way:- When data migration jobs are submitted, CDM splits each job into multiple tasks based on the Concurrent Extractors parameter in the job configuration.NOTE:
Jobs for different data sources may be split based on different dimensions. Some jobs may not be split based on the Concurrent Extractors parameter.
- CDM submits the tasks to the running pool in sequence. Tasks (defined by Maximum Concurrent Extractors) run concurrently. Excess tasks are queued.
Dirty Data Directory
This parameter is displayed only when Write Dirty Data is set to Yes.
+This parameter is displayed only when Write Dirty Data is set to Yes.
Dirty data is stored in the directory for storing dirty data on OBS. Dirty data is saved only when this parameter is configured.
You can go to this directory to query data that fails to be processed or is filtered out during job execution, and check the source data that does not meet conversion or cleaning rules.
Max. Error Records in a Single Shard
This parameter is displayed only when Write Dirty Data is set to Yes.
+This parameter is displayed only when Write Dirty Data is set to Yes.
When the number of error records of a single map exceeds the upper limit, the job will automatically terminate and the imported data cannot be rolled back. You are advised to use a temporary table as the destination table. After the data is imported, rename the table or combine it into the final data table.
0
@@ -393,20 +407,29 @@Yes
Max. error records in a single shard
+byteRate(MB/s)
Maximum rate for a job. To configure throttling for multiple jobs, multiply the rate by the number of concurrent jobs.
+Maximum read/write speed of the job
+Throttling can be enabled for a job for migrating data to Hive, DLI, JDBC, OBS, or HDFS. If multiple concurrent jobs are allowed, the actual maximum speed can be calculated by the value of this parameter multiplied by the number of concurrent jobs.
NOTE:The rate is an integer greater than 1.
20
Intermediate Queue Cache Size (MB)
+Amount of data that the intermediate queue can cache. The value ranges from 1 to 500. The default value is 64.
+If the amount of data of a row exceeds the value of this parameter, the migration may fail. If the value of this parameter is too large, the cluster may not run properly. Set an appropriate value for this parameter and use the default value (64) unless otherwise specified.
+64
+ - When data migration jobs are submitted, CDM splits each job into multiple tasks based on the Concurrent Extractors parameter in the job configuration.
- Click Save or Save and Run. On the page displayed, you can view the job status.
The job status can be New, Pending, Booting, Running, Failed, Succeeded, or Stopped.
+- Click Save or Save and Run. On the displayed page, you can view the job status.
@@ -414,7 +437,14 @@ The job status can be New, Pending, Booting, Running, Failed, Succeeded, or Stopped.
Pending indicates that the job is waiting to be scheduled by the system, and Booting indicates that the data to be migrated is being analyzed.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0047.html b/docs/dataartsstudio/umn/dataartsstudio_01_0047.html index ab2c4066f..971b697a1 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0047.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0047.html @@ -1,6 +1,6 @@ --Parent topic: Managing Jobs+Parent topic: Creating a Job in a CDM ClusterSource Job Parameters
+Configuring CDM Source Job Parameters
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0048.html b/docs/dataartsstudio/umn/dataartsstudio_01_0048.html index 17f385050..c5f564994 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0048.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0048.html @@ -29,7 +29,7 @@-
@@ -12,7 +12,7 @@
- From Hive
- - From DLI
+ - From DLI
- From FTP/SFTP
@@ -44,14 +44,16 @@
- From MRS ClickHouse
+ - From LogHub (SLS)
+ - From a ShenTong Database
- - From Doris
+ - From YASHAN
-Parent topic: Managing Jobs+Parent topic: Creating a Job in a CDM ClusterDirectory or file path from which data will be extracted. You can enter a maximum of 50 file paths. By default, the file paths are separated by vertical bars (|). You can also customize a file separator.
Directory from which data is to be migrated. All files (including all nested subdirectories and their subfiles) in the directory will be migrated.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
FROM/example.csv
@@ -77,7 +77,7 @@ - Click Save or Save and Run. On the displayed page, you can view the job status.
JSON Type
This parameter is displayed only when File Format is set to JSON. Type of a JSON object stored in a JSON file. The options are JSON object and JSON array.
+This parameter is displayed only when File Format is set to JSON. Type of a JSON object stored in a JSON file. The options are JSON object and JSON array.
JSON object
Use First N Rows as Header
This parameter is displayed only when File Format is set to CSV. When you migrate a CSV file to a table, CDM writes all data to the table by default. If you set this parameter to Yes, CDM uses the first N rows of the CSV file as the heading row and does not write the row to the destination table.
+This parameter is displayed only when File Format is set to CSV. When you migrate a CSV file to a table, CDM writes all data to the table by default. If you set this parameter to Yes, CDM uses the first N rows of the CSV file as the heading row and does not write the row to the destination table.
No
Directory Filter
If you set Filter Type to Wildcard or Regex, enter a wildcard character to filter paths. The paths that meet the filtering condition are migrated. You can configure multiple paths separated by commas (,).
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
*input
@@ -231,7 +231,7 @@File Filter
If you set Filter Type to Wildcard or Regex, you can enter a wildcard character to search for files in a specified path. The files that meet the search criteria are migrated. You can configure multiple files separated by commas (,).
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
*.csv,*.txt
@@ -248,7 +248,7 @@If you set Filter Type to Time Filter, and specify a point in time for this parameter, only the files modified at or after the specified time are transferred. The time format must be yyyy-MM-dd HH:mm:ss.
This parameter can be set to a macro variable of date and time. For example, ${timestamp(dateformat(yyyy-MM-dd HH:mm:ss,-90,DAY))} indicates that only files generated within the latest 90 days are migrated.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
2019-06-01 00:00:00
@@ -258,7 +258,7 @@If you set Filter Type to Time Filter, and specify a point in time for this parameter, only the files modified before the specified time are transferred. The time format must be yyyy-MM-dd HH:mm:ss.
This parameter can be set to a macro variable of date and time. For example, ${timestamp(dateformat(yyyy-MM-dd HH:mm:ss))} indicates that only the files whose modification time is earlier than the current time are migrated.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
2019-07-01 00:00:00
@@ -288,7 +288,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0049.html b/docs/dataartsstudio/umn/dataartsstudio_01_0049.html index 8d45b002c..07243fabf 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0049.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0049.html @@ -28,7 +28,7 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job ParametersDirectory or file path from which data will be extracted.
Directory from which data is to be migrated. All files (including all nested subdirectories and their subfiles) in the directory will be migrated.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
/user/cdm/
@@ -91,7 +91,7 @@Use First Row as Header
This parameter is displayed only when File Format is set to CSV. When you migrate a CSV file to a table, CDM writes all data to the table by default. If you set this parameter to Yes, CDM uses the first N rows of the CSV file as the heading row and does not write the row to the destination table.
+This parameter is displayed only when File Format is set to CSV. When you migrate a CSV file to a table, CDM writes all data to the table by default. If you set this parameter to Yes, CDM uses the first N rows of the CSV file as the heading row and does not write the row to the destination table.
No
Directory Filter
If you set Filter Type to Wildcard or Regex, enter a wildcard character to filter paths. The paths that meet the filtering condition are migrated. You can configure multiple paths separated by commas (,).
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
*input
@@ -129,7 +129,7 @@File Filter
If you set Filter Type to Wildcard or Regex, you can enter a wildcard character to search for files in a specified path. The files that meet the search criteria are migrated. You can configure multiple files separated by commas (,).
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
*.csv
@@ -146,7 +146,7 @@If you set Filter Type to Time Filter, and specify a point in time for this parameter, only the files modified at or after the specified time are transferred. The time format must be yyyy-MM-dd HH:mm:ss.
This parameter can be set to a macro variable of date and time. For example, ${timestamp(dateformat(yyyy-MM-dd HH:mm:ss,-90,DAY))} indicates that only files generated within the latest 90 days are migrated.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
2019-07-01 00:00:00
@@ -156,7 +156,7 @@If you set Filter Type to Time Filter, and specify a point in time for this parameter, only the files modified before the specified time are transferred. The time format must be yyyy-MM-dd HH:mm:ss.
This parameter can be set to a macro variable of date and time. For example, ${timestamp(dateformat(yyyy-MM-dd HH:mm:ss))} indicates that only the files whose modification time is earlier than the current time are migrated.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
2019-07-30 00:00:00
@@ -181,7 +181,7 @@DEK
This parameter is displayed only when Encryption is set to AES-256-GCM. The key consists of 64 hexadecimal numbers and must be the same as the DEK configured during encryption. If the encryption and decryption keys are inconsistent, the system does not report an exception, but the decrypted data is incorrect.
+This parameter is displayed only when Encryption is set to AES-256-GCM. The key consists of 64 hexadecimal numbers and must be the same as the DEK configured during encryption. If the encryption and decryption keys are inconsistent, the system does not report an exception, but the decrypted data is incorrect.
DD0AE00DFECD78BF051BCFDA25BD4E320DB0A7AC75A1F3FC3D3C56A457DCDC1B
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0050.html b/docs/dataartsstudio/umn/dataartsstudio_01_0050.html index a6ffc3a21..c79d36902 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0050.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0050.html @@ -22,7 +22,7 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job ParametersName of the HBase table that data will be exported from
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
TBL_2
@@ -55,7 +55,7 @@(Optional) Start time (including the value) for extracting data. The format is yyyy-MM-dd HH:mm:ss. Only the data generated at the specified time and later is extracted.
This parameter can be set to a macro variable of date and time. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
2019-01-01 20:00:00
@@ -65,7 +65,7 @@(Optional) End time (excluding the value) for extracting data. The format is yyyy-MM-dd HH:mm:ss. Only the data generated before the time point is extracted.
This parameter can be set to a macro variable of date and time.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
2019-02-01 20:00:00
@@ -77,7 +77,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0051.html b/docs/dataartsstudio/umn/dataartsstudio_01_0051.html index 8d83be724..b1afc1a25 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0051.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0051.html @@ -26,7 +26,7 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job ParametersHive table name. Click the icon next to the text box. The dialog box for selecting the table is displayed.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
TBL_E
@@ -35,7 +35,9 @@Read Mode
Two read modes are available: HDFS and JDBC. By default, the HDFS mode is used. If you do not need to use the WHERE condition to filter data or add new fields on the field mapping page, select the HDFS mode.
-- The HDFS mode shows good performance, but in this mode, you cannot use the WHERE condition to filter data or add new fields on the field mapping page.
- The HDFS mode allows you to use the WHERE condition to filter data or add new fields on the field mapping page.
- The HDFS mode shows good performance, but in this mode, you cannot use the WHERE condition to filter data or add new fields on the field mapping page.
- The HDFS mode allows you to use the WHERE condition to filter data or add new fields on the field mapping page.NOTE:+
If the migration source is Hive and JDBC is used to read data, CDM does not support concurrency. That is, Concurrent Extractors can only be set to 1.
+
HDFS
SQL Statement
When Use SQL Statement is set to Yes, enter an SQL statement here. CDM exports data based on the SQL statement.
-NOTE:- SQL statements can only be used to query data. Join and nesting are supported, but multiple query statements are not allowed, for example, select * from table a; select * from table b.
- With statements are not supported.
- Comments, such as -- and /*, are not supported.
- Addition, deletion, and modification operations are not supported, including but not limited to the following:
- load data
- delete from
- alter table
- create table
- drop table
- into outfile
NOTE:- SQL statements can only be used to query data. Join and nesting are supported, but multiple query statements are not allowed, for example, select * from table a; select * from table b.
- With statements are not supported.
- Comments, such as -- and /*, are not supported.
- Addition, deletion, and modification operations are not supported, including but not limited to the following:
- load data
- delete from
- alter table
- create table
- drop table
- into outfile
- If the SQL statement is too long, the request fails to be delivered. If you continue to create a job, the system displays an error message indicating that the request is incorrect. In this case, you need to simplify or clear the SQL statement and try again.
select id,name from sqoop.user;
@@ -64,7 +66,7 @@This parameter is displayed when you select the HDFS read mode and click Show Advanced Attributes.
This parameter indicates extracting the partition of a specified value. The attribute name is the partition name. You can configure multiple values (separated by spaces) or a field value range. The time macro function is supported.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
- Attribute value in the single-value or multi-value filtering scenario:
${dateformat(yyyyMMdd, -1, DAY)} ${dateformat(yyyyMMdd)}
@@ -77,7 +79,7 @@This parameter is displayed when you select the JDBC read mode and click Show Advanced Attributes.
This parameter indicates the WHERE clause to be extracted. If this parameter is not set, the entire table is extracted. If the table to be migrated does not contain the fields specified by the WHERE clause, the migration will fail.
You can set a date macro variable to extract data generated on a specific date.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
age > 18 and age <= 60
@@ -91,7 +93,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0052.html b/docs/dataartsstudio/umn/dataartsstudio_01_0052.html index fa80328bb..13c406489 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0052.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0052.html @@ -21,7 +21,7 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job ParametersDirectory or file path from which data will be extracted. You can enter a maximum of 50 file paths. By default, the file paths are separated by vertical bars (|). You can also customize a file separator.
Directory from which data is to be migrated. All files (including all nested subdirectories and their subfiles) in the directory will be migrated.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
/ftp/a.csv|/ftp/b.txt
@@ -39,7 +39,7 @@
JSON Type
This parameter is displayed only when File Format is set to JSON. Type of a JSON object stored in a JSON file. The options are JSON object and JSON array.
+This parameter is displayed only when File Format is set to JSON. Type of a JSON object stored in a JSON file. The options are JSON object and JSON array.
JSON object
Use First Row as Header
This parameter is displayed only when File Format is set to CSV. When you migrate a CSV file to a table, CDM writes all data to the table by default. If you set this parameter to Yes, CDM uses the first N rows of the CSV file as the heading row and does not write the row to the destination table.
+This parameter is displayed only when File Format is set to CSV. When you migrate a CSV file to a table, CDM writes all data to the table by default. If you set this parameter to Yes, CDM uses the first N rows of the CSV file as the heading row and does not write the row to the destination table.
Yes
Directory Filter
If you set Filter Type to Wildcard or Regex, enter a wildcard character to filter paths. The paths that meet the filtering condition are migrated. You can configure multiple paths separated by commas (,).
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
*input,*out
@@ -182,7 +182,7 @@File Filter
If you set Filter Type to Wildcard or Regex, enter a wildcard character to filter paths. The files that meet the filtering condition are migrated. You can configure multiple files separated by commas (,).
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
*.csv
@@ -199,7 +199,7 @@If you set Time Filter to Yes, you can specify a point in time for Minimum Timestamp, and then only the files modified at or after the specified time are transferred. The time format must be yyyy-MM-dd HH:mm:ss.
This parameter can be set to a macro variable of date and time. For example, ${timestamp(dateformat(yyyy-MM-dd HH:mm:ss,-90,DAY))} indicates that only files generated within the latest 90 days are migrated.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
2019-07-01 00:00:00
@@ -209,7 +209,7 @@If you set Time Filter to Yes, you can specify a point in time for Maximum Timestamp, and then only the files modified before the specified time are transferred. The time format must be yyyy-MM-dd HH:mm:ss.
This parameter can be set to a macro variable of date and time. For example, ${timestamp(dateformat(yyyy-MM-dd HH:mm:ss))} indicates that only the files whose modification time is earlier than the current time are migrated.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
2019-07-30 00:00:00
@@ -260,7 +260,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0053.html b/docs/dataartsstudio/umn/dataartsstudio_01_0053.html index 52a255369..ebf84728b 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0053.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0053.html @@ -49,9 +49,9 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job ParametersFile Format
Format used for transmitting data. The CSV and JSON formats are supported for migration to tables, and the binary format is supported for file migration.
+Format used for transmitting data. The CSV and JSON formats are supported for migration to tables, and the binary format is supported for file migration.
Binary
+Binary
Compression Format
@@ -98,13 +98,20 @@.md5
Query Parameter
+If this parameter is set to Yes, the name of the object to be uploaded is a string with the query parameter removed.
+No
+diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0054.html b/docs/dataartsstudio/umn/dataartsstudio_01_0054.html index ea19b54de..6b25b5799 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0054.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0054.html @@ -25,8 +25,8 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job ParametersSQL Statement
When Use SQL Statement is set to Yes, enter an SQL statement here. CDM exports data based on the SQL statement.
-NOTE:- SQL statements can only be used to query data. Join and nesting are supported, but multiple query statements are not allowed, for example, select * from table a; select * from table b.
- With statements are not supported.
- Comments, such as -- and /*, are not supported.
- Addition, deletion, and modification operations are not supported, including but not limited to the following:
- load data
- delete from
- alter table
- create table
- drop table
- into outfile
NOTE:- SQL statements can only be used to query data. Join and nesting are supported, but multiple query statements are not allowed, for example, select * from table a; select * from table b.
- With statements are not supported.
- Comments, such as -- and /*, are not supported.
- Addition, deletion, and modification operations are not supported, including but not limited to the following:
- load data
- delete from
- alter table
- create table
- drop table
- into outfile
- If the SQL statement is too long, the request fails to be delivered. If you continue to create a job, the system displays an error message indicating that the request is incorrect. In this case, you need to simplify or clear the SQL statement and try again.
select id,name from sqoop.user;
@@ -48,7 +48,7 @@Name of the table from which data will be extracted. This parameter is displayed when Use SQL Statement is set to No. Click the icon next to the text box to go to the page for selecting the table or directly enter a table name.
If the desired table is not displayed, check whether the table exists or whether the login account has the permission to query metadata.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
NOTE:The table name can contain wildcard characters (*), which is used to export all tables whose names start with a certain prefix or end with a certain suffix. The number and types of fields in the tables must be the same. The examples are as follows:@@ -75,7 +75,7 @@- table* indicates that all tables whose names starting with table are exported.
- *table indicates that all tables whose names ending with table are exported.
- *table* indicates that all tables whose names containing table are exported.
WHERE clause used to specify the data extraction range. This parameter is displayed when Use SQL Statement is set to No. If this parameter is not set, the entire table is extracted.
You can set a date macro variable to extract data generated on a specific date.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
DS='${dateformat(yyyy-MM-dd,-1,DAY)}'
@@ -139,7 +139,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0055.html b/docs/dataartsstudio/umn/dataartsstudio_01_0055.html index 26eaa6d01..001499282 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0055.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0055.html @@ -48,7 +48,7 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job Parametersdiff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0056.html b/docs/dataartsstudio/umn/dataartsstudio_01_0056.html index 130fd54dd..b9154239c 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0056.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0056.html @@ -61,7 +61,7 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job Parametersdiff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0058.html b/docs/dataartsstudio/umn/dataartsstudio_01_0058.html index e89f2090e..b76adab32 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0058.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0058.html @@ -145,7 +145,7 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job Parametersdiff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0059.html b/docs/dataartsstudio/umn/dataartsstudio_01_0059.html index ba52e8a8c..1a76f5bbe 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0059.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0059.html @@ -1,7 +1,7 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job ParametersFrom Elasticsearch or CSS
-If the source link of a job is a link described in Link to Elasticsearch or Link to CSS, configure the source job parameters based on Table 1.
+If the source link of a job is a link described in Elasticsearch Link Parameters or CSS Link Parameters, configure the source job parameters based on Table 1.
-Table 1 Job parameters when Elasticsearch or CSS is the source Category
Basic parameters
-Index
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0060.html b/docs/dataartsstudio/umn/dataartsstudio_01_0060.html index a3943489f..2079df817 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0060.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0060.html @@ -52,7 +52,7 @@-Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job Parametersdiff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0061.html b/docs/dataartsstudio/umn/dataartsstudio_01_0061.html index 1a80f867f..e1d11fab2 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0061.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0061.html @@ -1,6 +1,6 @@ --Parent topic: Source Job Parameters+Parent topic: Configuring CDM Source Job ParametersDestination Job Parameters
+Configuring CDM Destination Job Parameters
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0062.html b/docs/dataartsstudio/umn/dataartsstudio_01_0062.html index 7b1b44dc5..5a7538eea 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0062.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0062.html @@ -27,7 +27,7 @@-
@@ -34,12 +34,10 @@
- To MongoDB
- - To Doris
-
-Parent topic: Managing Jobs+Parent topic: Creating a Job in a CDM ClusterOBS directory to which data will be written. Do not add / in front of the directory name.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
directory/
@@ -65,7 +65,7 @@KMS ID
Data encryption key. This parameter is displayed when Encryption is set to KMS. Click
+
next to the text box to select the KMS key that was created in DEW.Data encryption key. This parameter is displayed when Encryption is set to KMS. Click
next to the text box to select the KMS key that was created in DEW.- If the KMS key of the same project as that of the CDM cluster is used, you do not need to modify Project ID.
- If the KMS key of another project is used, you need to modify Project ID.
53440ccb-3e73-4700-98b5-71ff5476e621
@@ -221,7 +221,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0063.html b/docs/dataartsstudio/umn/dataartsstudio_01_0063.html index 3fa01f59c..6fcec5a19 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0063.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0063.html @@ -15,7 +15,7 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job ParametersHDFS directory to which data will be written.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
/user/output
@@ -139,7 +139,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0064.html b/docs/dataartsstudio/umn/dataartsstudio_01_0064.html index 9302a2c7d..d8d97e856 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0064.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0064.html @@ -15,7 +15,7 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job ParametersName of the HBase table to which data will be written. If you want to create an HBase table, you can copy the field names from the migration source. Click the icon next to the text box. The dialog box for selecting the table is displayed.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
TBL_2
@@ -83,7 +83,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0066.html b/docs/dataartsstudio/umn/dataartsstudio_01_0066.html index d9a8f0dda..04ca989c3 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0066.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0066.html @@ -22,7 +22,7 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job ParametersDestination table name. Click the icon next to the text box. The dialog box for selecting the table is displayed.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
TBL_X
@@ -38,6 +38,14 @@Non-auto creation
Source side null value conversion value
+Value to which the source null value is converted
+- TO_NULL
- TO_EMPTY_STRRING
- TO_NULL_STRING
TO_NULL
+Clear Data Before Import
Whether the data in the destination table is cleared before data import. The options are as follows:- Yes: The data is cleared.
- No: The data is not cleared. Instead, it will be added to the existing table.
Yes
Partition to Clear
+Processing mode of newline characters
This parameter is available when Clear Data Before Import is set to Yes.
-When you enter the information about the partitions to be cleared, the data in the partitions will be cleared.
+Policy for processing the newline characters in the data written to Hive textfile tables
+- Delete
- Replace with another string
- Ignore
Delete
+Hive Table Partition Field
+This parameter is unavailable when Auto Table Creation is set to Non-auto Creation.
+Partition fields for creating a Hive table. Use commas (,) to separate multiple fields.
+A,B
+Table Path
+This parameter is unavailable when Auto Table Creation is set to Non-auto Creation.
+It specifies the table path.
+-
+Storage Format
+This parameter is unavailable when Auto Table Creation is set to Non-auto Creation.
+It specifies the storage format.
+- Row-based storage format: TEXTFILE
- Column-based storage formats: ORC, RCFILE, and PARQUET
TEXTFILE data is stored in plaintext. If data contains special characters, data may be written incorrectly. Exercise caution when using this format. The ORC format is recommended.
+ORC
+ClearDataMode
+This parameter is available when Clear Data Before Import is set to Yes.
+It specifies the mode for clearing data in the Hive table.
+- LOAD_OVERWRITE: A temporary data file directory is generated and loaded to the Hive table using the load overwrite syntax of Hive.
- TRUNCATE: Data files in partitions are deleted, but partitions are not deleted.NOTE:+
If the destination is a partitioned table, you are advised to select LOAD_OVERWRITE. Otherwise, the cluster memory or disks may be overloaded.
+
TRUCATE
+Partitions info
+This parameter is available when Clear Data Before Import is set to Yes. If the destination is a partitioned table, you must specify partitions.
+- If you select the TRUCATE mode, only the data files in the partitions are deleted.
- If you select the LOAD_OVERWRITE mode, data is written to a specified partition and overwrites the existing data. +
Single partition: year=2020,location=sun
Multiple partitions: ['year=2020,location=sun', 'year=2021,location=earth']
+Partitions of the previous day:
+day='${dateformat(yyyy-MM-dd HH:mm:ss, -1, DAY)}',
Executing Analyze Statements
After all data is written, the ANALYZE TABLE statement is asynchronously executed to accelerate the Hive table query. The SQL statement is as follows:
+After all data is written, the ANALYZE TABLE statement is asynchronously executed to accelerate query of data from Hive tables.
+Run the following SQL statements:
- Non-partitioned table: ANALYZE TABLE tablename COMPUTE STATISTICS
- Partitioned table: ANALYZE TABLE tablename PARTITION(partcol1[=val1], partcol2[=val2], ...) COMPUTE STATISTICS
NOTE:Parameter Executing Analyze Statements applies only to the migration of a single table.
+Running the ANALYZE statements may exert pressure on Hive.
Yes
Maximum memory size of the internal write queue
+If the memory is insufficient, change the value of this parameter as needed. If the value is too small, the migration speed will be affected.
+The value ranges from 1 to 128 MB. The default value is empty, indicating that there is no limit. If you set a value beyond the range, there is no limit.
+16
+Maximum memory size of the internal conversion queue
+If the memory is insufficient, change the value of this parameter as needed. If the value is too small, the migration speed will be affected.
+The value ranges from 1 to 128 MB. The default value is empty, indicating that there is no limit. If you set a value beyond the range, there is no limit.
+16
+ - When Hive serves as the destination end, a table whose storage format is ORC is automatically created.
- Due to file format restrictions, complex data can be written only in ORC or Parquet format.
- If the source Hive contains both the array and map types of data, the destination table format can only be the ORC or parquet complex type. If the destination table format is RC or TEXT, the source data will be processed and can be successfully written.
- As the map type is an unordered data structure, the data type may change after a migration.
- If Hive serves as the migration destination and the storage format is Textfile, delimiters must be explicitly specified in the statement for creating Hive tables. The following is an example:
CREATE TABLE csv_tbl( +
- If the source Hive contains both the array and map types of data, the destination table format can only be the ORC or parquet complex type. If the destination table format is RC or TEXT, the source data will be processed and can be successfully written.
- As the map type is an unordered data structure, the data type may change after a migration.
- If Hive serves as the migration destination and the storage format is Textfile, delimiters must be explicitly specified in the statement for creating Hive tables. The following is an example:
CREATE TABLE csv_tbl( smallint_value smallint, tinyint_value tinyint, int_value int, @@ -100,7 +174,7 @@ STORED AS TEXTFILE;
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0068.html b/docs/dataartsstudio/umn/dataartsstudio_01_0068.html index c28de4e26..53b92d919 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0068.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0068.html @@ -35,7 +35,7 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job Parameters
Name of the table to which data will be written. Click the icon next to the text box. The dialog box for selecting the table is displayed.
This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
-NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
+NOTE:If you have configured a macro variable of date and time and schedule a CDM job through DataArts Studio DataArts Factory, the system replaces the macro variable of date and time with (Planned start time of the data development job – Offset) rather than (Actual start time of the CDM job – Offset).
table
@@ -121,7 +121,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0069.html b/docs/dataartsstudio/umn/dataartsstudio_01_0069.html index 57d7ee7b6..f75b8af79 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0069.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0069.html @@ -1,7 +1,7 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job ParametersTo DDS
-If the destination link of a job is a DDS link, configure the destination job parameters based on Table 1. +If the destination link of a job is a DDS link, configure the destination job parameters based on Table 1.Table 1 Parameter description Parameter
Description
@@ -12,7 +12,7 @@Database Name
Database to which data is to be imported.
+Database to which data is to be imported
ddsdb
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0070.html b/docs/dataartsstudio/umn/dataartsstudio_01_0070.html index c88bf0fdc..2ee0c0c13 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0070.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0070.html @@ -27,7 +27,7 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job ParametersUse Column Value as Field
This parameter is displayed when Value Storage Type is set to Hash. Only Hash is supported. If this function is enabled, values are alternately used as fields and values in sequence except the primary key column.
+This parameter is displayed when Value Storage Type is set to HASH. Only Hash is supported. If this function is enabled, values are alternately used as fields and values in sequence except the primary key column.
Yes
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0071.html b/docs/dataartsstudio/umn/dataartsstudio_01_0071.html index 3770ebbba..0cca9c83c 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0071.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0071.html @@ -1,62 +1,75 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job ParametersTo Elasticsearch/CSS
-If the destination link of a job is a link described in Link to Elasticsearch or Link to CSS, configure the destination job parameters based on Table 1.
+If the destination link of a job is a link described in Elasticsearch Link Parameters or CSS Link Parameters, configure the destination job parameters based on Table 1.
+-
The parameters required for table/file migration are different from those for entire DB migration. The following table lists the parameters for table/file migration. The actual parameters are subject to those displayed on the console.
+Table 1 Job parameters when Elasticsearch/CSS is the destination Parameter
+-Table 1 Job parameters when Elasticsearch/CSS is the destination Parameter
Description
+Description
Example Value
+Example Value
Index
+Index
Elasticsearch index, which is similar to the name of a relational database. CDM supports automatic creation of indexes and field types. The index and field type names can contain only lowercase letters.
+Elasticsearch index, which is similar to the name of a relational database. CDM supports automatic creation of indexes and field types. The index and field type names can contain only lowercase letters.
index
+index
Type
+Type
Elasticsearch type, which is similar to the table name of a relational database. The type name can contain only lowercase letters.
+Elasticsearch type, which is similar to the table name of a relational database. The type name can contain only lowercase letters.
NOTE:Elasticsearch 7.x and later versions do not support custom types. Instead, only the _doc type can be used. In this case, this parameter does not take effect even if it is set.
type
+type
Pipeline ID
+Operation
Pipeline used to convert the data format after data is transferred to Elasticsearch. Pipeline IDs are ready for use after being created in Kibana.
+Operation type
+- INDEX: No primary key is required. Elasticsearch generates IDs so that data is written to a new file with a unique ID for each write operation.
- CREATE: A primary key needs to be specified. If the primary key already exists, the write operation fails.
- UPDATE: A primary key needs to be specified. If the primary key already exists, the original data is overwritten.
- UPSERT: A primary key needs to be specified. If a primary key already exists, the existing data is overwritten. If there is no primary key, a new document is created for writing data.
pipeline_id
+INDEX
Write ES with Routing
+Pipeline ID
If you enable this function, a column can be written to Elasticsearch as a route.
+ID of the pipeline used to convert the format of the data transferred to Elasticsearch.
+If the destination is Elasticsearch, you need to create a pipeline ID in Kibana first.
+If the destination is CSS, you do not need to create a pipeline ID. Instead, enter the name of the configuration file, which is name by default.
+If the destination is Elasticsearch: pipeline_id
+If the destination is CSS: name (name of the configuration file)
+Write ES with Routing
+If you enable this function, a column can be written to Elasticsearch as a route.
NOTE:Before enabling this function, create indexes at the destination to improve the query efficiency.
No
+No
Route Column
+Route Column
This parameter is available when Write ES with Routing is set to Yes. It specifies the destination routing column. If the destination index exists but the column information cannot be obtained, you can manually enter the column. The route column can be empty. If it is empty, no routing value is specified for the data written to Elasticsearch.
+This parameter is available when Write ES with Routing is set to Yes. It specifies the destination routing column. If the destination index exists but the column information cannot be obtained, you can manually enter the column. The route column can be empty. If it is empty, no routing value is specified for the data written to Elasticsearch.
value1
+value1
Periodically Create Index
+Periodically Create Index
For streaming jobs that continuously write data to Elasticsearch, CDM periodically creates indexes and writes data to the indexes, which helps you delete expired data. The indexes can be created based on the following periods:- Every hour: CDM creates indexes on the hour. The new indexes are named in the format of Index name+Year+Month+Day+Hour, for example, index2018121709.
- Every day: CDM creates indexes at 00:00 every day. The new indexes are named in the format of Index name+Year+Month+Day, for example, index20181217.
- Every week: CDM creates indexes at 00:00 every Monday. The new indexes are named in the format of Index name+Year+Week, for example, index201842.
- Every month: CDM creates indexes at 00:00 on the first day of each month. The new indexes are named in the format of Index name+Year+Month, for example, index201812.
- Do not create: Do not create indexes periodically.
- Every hour: CDM creates indexes on the hour. The new indexes are named in the format of Index name+Year+Month+Day+Hour, for example, index2018121709.
- Every day: CDM creates indexes at 00:00 every day. The new indexes are named in the format of Index name+Year+Month+Day, for example, index20181217.
- Every week: CDM creates indexes at 00:00 every Monday. The new indexes are named in the format of Index name+Year+Week, for example, index201842.
- Every month: CDM creates indexes at 00:00 on the first day of each month. The new indexes are named in the format of Index name+Year+Month, for example, index201812.
- Do not create: Do not create indexes periodically.
When extracting data from a file, you must configure a single extractor, which means setting Concurrent Extractors to 1. Otherwise, this parameter is invalid.
Every hour
+Every hour
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0072.html b/docs/dataartsstudio/umn/dataartsstudio_01_0072.html index 1ac9f9595..cf47828bc 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0072.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0072.html @@ -2,7 +2,7 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job ParametersTo DLI
If the destination link of a job is a DLI link, configure the destination job parameters based on Table 1.
- When you use CDM to migrate data to DLI, DLI generates data files in the dli-trans* temporary OBS bucket. Therefore, you need to grant the account corresponding to the AK/SK the permissions to read and write the dli-trans* bucket and create directories. For details about how to add OBS permission policies, see Adding an OBS Bucket Policy.
+-
When data is migrated to DLI using CDM, DLI generates data files in the dli-trans* temporary OBS bucket. Therefore, you need to grant the user who uses the AK/SK the permissions to read and write the dli-trans* bucket and create directories. Otherwise, the migration will fail. For details about how to add permission policies for temporary bucket dli-trans*, see Adding an Authorization Policy for the dli-trans* Temporary Bucket.
Adding an OBS Bucket Policy
- Log in to the IAM console.
- In the navigation pane, choose Permissions > Policies/Roles and click Create Custom Policy in the upper right corner.Figure 1 Creating a custom policy-
- Enter a policy name and set Policy Content.Figure 2 Configuring the policy-
-
- Enter the policy description and click OK.
-Adding an Authorization Policy for the dli-trans* Temporary Bucket
- Log in to the IAM console.
- In the navigation pane, choose Permissions > Policies/Roles and click Create Custom Policy in the upper right corner.Figure 1 Creating a custom policy+
- On the Create Custom Policy page, select JSON for Policy View and create custom policy obs_dli-trans.
{ + "Version": "1.1", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "obs:object:GetObject", + "obs:object:DeleteObjectVersion", + "obs:bucket:GetBucketLocation", + "obs:object:GetAccessLabel", + "obs:bucket:PutEncryptionConfiguration", + "obs:bucket:PutBucketStoragePolicy", + "obs:object:DeleteAccessLabel", + "obs:bucket:PutBucketCustomDomainConfiguration", + "obs:bucket:GetLifecycleConfiguration", + "obs:bucket:PutBucketInventoryConfiguration", + "obs:bucket:DeleteDirectColdAccessConfiguration", + "obs:object:AbortMultipartUpload", + "obs:bucket:PutBucketLogging", + "obs:bucket:DeleteBucketWebsite", + "obs:object:DeleteObject", + "obs:bucket:PutBucketVersioning", + "obs:bucket:GetBucketWebsite", + "obs:bucket:GetBucketLogging", + "obs:bucket:DeleteBucketCustomDomainConfiguration", + "obs:object:PutObject", + "obs:object:RestoreObject", + "obs:bucket:PutReplicationConfiguration", + "obs:bucket:GetBucketQuota", + "obs:object:GetObjectVersionAcl", + "obs:bucket:DeleteBucket", + "obs:bucket:CreateBucket", + "obs:bucket:GetDirectColdAccessConfiguration", + "obs:bucket:PutDirectColdAccessConfiguration", + "obs:bucket:GetBucketAcl", + "obs:bucket:GetBucketVersioning", + "obs:bucket:GetBucketInventoryConfiguration", + "obs:bucket:GetBucketStoragePolicy", + "obs:bucket:GetEncryptionConfiguration", + "obs:bucket:PutBucketCORS", + "obs:bucket:PutBucketTagging", + "obs:bucket:GetBucketTagging", + "obs:bucket:PutLifecycleConfiguration", + "obs:bucket:GetBucketCustomDomainConfiguration", + "obs:object:ListMultipartUploadParts", + "obs:object:ModifyObjectMetaData", + "obs:bucket:ListBucketVersions", + "obs:bucket:PutBucketQuota", + "obs:object:PutAccessLabel", + "obs:bucket:ListBucket", + "obs:bucket:GetBucketCORS", + "obs:bucket:DeleteBucketInventoryConfiguration", + "obs:object:GetObjectVersion", + "obs:bucket:PutBucketWebsite", + "obs:bucket:DeleteReplicationConfiguration", + "obs:object:GetObjectAcl", + "obs:bucket:GetBucketNotification", + "obs:bucket:PutBucketNotification", + "obs:bucket:GetReplicationConfiguration", + "obs:bucket:GetBucketPolicy", + "obs:bucket:DeleteBucketTagging", + "obs:bucket:GetBucketStorage" + ], + "Resource": [ + "OBS:*:*:object:*", + "OBS:*:*:bucket:dli-trans*" + ] + } + ] +}+Figure 2 Creating custom policy obs_dli-trans+ + - Click OK.
- In the navigation pane, choose User Groups, locate the user group to which the DLI link user using the AK/SK belongs, and click Authorize to assign the custom obs_dli-trans policy to the user.Figure 3 Assigning the custom obs_dli-trans policy to a user group+ +
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0074.html b/docs/dataartsstudio/umn/dataartsstudio_01_0074.html index c50ba84dd..e998cfd21 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0074.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0074.html @@ -38,7 +38,7 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job Parametersdiff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0075.html b/docs/dataartsstudio/umn/dataartsstudio_01_0075.html index 928dd9397..c676ceb80 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0075.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0075.html @@ -8,7 +8,7 @@-Parent topic: Destination Job Parameters+Parent topic: Configuring CDM Destination Job Parameters-Constraints
Field names of the source and destination parameters cannot contain ampersands (&) or number signs (%).
Prerequisites
- A link has been created. For details, see Creating Links.
Prerequisites
- A link has been created. For details, see Creating a Link Between CDM and a Data Source.
- The CDM cluster can communicate with the data source.
Procedure
- Log in to the management console and choose Service List > Cloud Data Migration. In the left navigation pane, choose Cluster Management. Locate the target cluster and click Job Management.
- Choose . The page for configuring the job is displayed.
- Configure the related parameters of the source database according to Table 1.
@@ -62,25 +62,28 @@
Start Time
Start time (included). The format is yyyy-MM-dd hh:mm:ss. The dateformat time macro variable function is supported. Examples: 2017-12-31 20:00:00, ${dateformat(yyyy-MM-dd, -1, DAY)} 02:00:00, and ${dateformat(yyyy-MM-dd HH:mm:ss, -1, DAY)}
+Start time (included). The format is yyyy-MM-dd hh:mm:ss. The dateformat time macro variable function is supported.
+Examples: 2017-12-31 20:00:00, ${dateformat(yyyy-MM-dd, -1, DAY)} 02:00:00, and ${dateformat(yyyy-MM-dd HH:mm:ss, -1, DAY)}
-
+"2017-12-31 20:00:00"
End Time
End time (excluded) The format is yyyy-MM-dd hh:mm:ss. The dateformat time macro variable function is supported. Examples: 2018-01-01 20:00:00, ${dateformat(yyyy-MM-dd, -1, DAY)} 02:00:00, and ${dateformat(yyyy-MM-dd HH:mm:ss, -1, DAY)}
+End time (excluded) The format is yyyy-MM-dd hh:mm:ss. The dateformat time macro variable function is supported.
+Examples: 2018-01-01 20:00:00, ${dateformat(yyyy-MM-dd, -1, DAY)} 02:00:00, and ${dateformat(yyyy-MM-dd HH:mm:ss, -1, DAY)}
-
+"2018-01-01 20:00:00"
Redis
Key Filter Character
Filter character used to determine the keys to be migrated For example, if the value of this parameter is a*, all asterisks (*) will be migrated.
+Filter character used to determine the keys to be migrated
+For example, if the value of this parameter is a*, all asterisks (*) will be migrated.
-
+a*
DDS
@@ -94,7 +97,8 @@Query Filter
Filter used to match documents. Example: {HTTPStatusCode:{$gt:"400",$lt:"500"},HTTPMethod:"GET"}
+Filter used to match documents.
+Example: {HTTPStatusCode:{$gt:"400",$lt:"500"},HTTPMethod:"GET"}
-
- If a relational database is migrated, after job parameters are configured, click Next to access the page for selecting tables. You can select the tables to be migrated to the migration destination based on your requirements.
- Click Next and set job parameters.
Figure 1 Task parameters+
- If you are migrating an entire relational database, click Next after configuring job parameters to select source and destination tables. Ensure that the destination table names are the same as the source table names. For example, if the source table name is test, the destination table name must also be test.
Figure 1 Field mapping+
- Click Next and set job parameters.
Figure 2 Task parameters
Table 3 describes related parameters.@@ -60,3 +60,10 @@ + + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0081.html b/docs/dataartsstudio/umn/dataartsstudio_01_0081.html index 374cf4af5..4b5c4adbb 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0081.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0081.html @@ -1,6 +1,6 @@ --Table 3 Task configuration parameters Parameter
Concurrent Extractors
Number of extractors to be concurrently executed. Generally, retain the default value.
+Maximum number of threads of the job for reading data from the source
+NOTE:The number of concurrent threads may be less than or equal to the value of this parameter for some data sources that do not support concurrent extraction, for example, CSS and ClickHouse.
+1
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0076.html b/docs/dataartsstudio/umn/dataartsstudio_01_0076.html index 72c26d6ad..8ae346973 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0076.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0076.html @@ -48,9 +48,9 @@-Parent topic: Managing Jobs+Parent topic: Creating a Job in a CDM ClusterFigure 1 Setting Maximum Concurrent Extractors for a CDM cluster+
Figure 1 Setting Maximum Concurrent Extractors for a CDM cluster
- Configure the number of concurrent extractors based on the following rules:
- When data is to be migrated to files, CDM does not support multiple concurrent tasks. In this case, set a single process to extract data.
- If each row of the table contains less than or equal to 1 MB data, data can be extracted concurrently. If each row contains more than 1 MB data, it is recommended that data be extracted in a single thread.
- Set Concurrent Extractors for a job based on Maximum Concurrent Extractors for the cluster. It is recommended that Concurrent Extractors is less than Maximum Concurrent Extractors.
- If the destination is DLI, you are advised to set the number of concurrent extractors to 1. Otherwise, data may fail to be written.
Figure 2 Setting Concurrent Extractors for a job+
Figure 2 Setting Concurrent Extractors for a job
Managing Jobs
+Creating a Job in a CDM Cluster
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0082.html b/docs/dataartsstudio/umn/dataartsstudio_01_0082.html index 69abedd9f..13e3efdaa 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0082.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0082.html @@ -1,6 +1,6 @@ --
@@ -8,24 +8,24 @@
- Creating an Entire Database Migration Job
- - Source Job Parameters
+ - Configuring CDM Source Job Parameters
- - Destination Job Parameters
+ - Configuring CDM Destination Job Parameters
- - Configuring Field Mapping
+ - Configuring CDM Job Field Mapping
- - Scheduling Job Execution
+ - Configuring a Scheduled CDM Job
- - Job Configuration Management
+ - Managing CDM Job Configuration
- - Managing a Single Job
+ - Managing a CDM Job
- - Managing Jobs in Batches
+ - Managing CDM Jobs
-Parent topic: DataArts Migration+Parent topic: DataArts Migration (CDM Jobs)Scheduling Job Execution
+Configuring a Scheduled CDM Job
CDM supports scheduled execution of table/file migration jobs by minute, hour, day, week, and month. This section describes how to configure scheduled job parameters.
- When configuring scheduled jobs, do not set the same scheduled time for different jobs. Instead, set different times to avoid exceptions.
If you use DataArts Studio DataArts Factory to schedule the CDM migration job and configure this parameter, both configurations take effect. To ensure unified service logic and avoid scheduling conflicts, enable job scheduling in DataArts Factory and do not configure a scheduled task for the job in DataArts Migration.
- The scheduled execution function uses the Java Quartz timer, which is similar to the Cron expression configuration. It parses the minute, hour, day, and month of the start time, and constructs a cronb expression.
For example, in the daily scheduling mode where the interval is set to 1 day: if the current time is 2022-10-14 12:00 and the start time is set to 2022-10-14 00:00, the job is executed at 2022-10-15 00:00; if the current time is 2022-10-14 12:00 and the start time is set to 2022-10-14 00:00, the job is executed at 2022-10-15 00:00.
@@ -9,45 +9,51 @@
Scheduling Job Execution by Minute
CDM allows jobs to be executed every several minutes. It is recommended that the cycle be at least 5 minutes.- +- Start Time: indicates the time when the scheduled configuration takes effect, or the first time when the job is automatically executed.
- Cycle (minutes): indicates the interval when a job is executed starting from the start time.
- End Time: This parameter is optional. If it is not set, the scheduled job keeps being automatically executed. If it is set, the scheduled job will be automatically stopped at the end time.
Figure 1 Scheduling job execution by minute-
For example, the settings shown in Figure 1 mean that the job will be automatically executed at 00:00 on January 1, 2023 for the first time at a cycle of 30 minutes until 23:59 on December 31, 2023.
+For example, the settings shown in the above figure mean that the job will be automatically executed at 00:00 on January 1, 2023 for the first time at a cycle of 30 minutes until 23:59 on December 31, 2023.
-Scheduling Job Execution by Hour
CDM allows jobs to be executed every several hours.- +- Cycle (hours): indicates the interval when a job is automatically executed.
- Trigger Time (minute): indicates the exact time in each hour when a scheduled task is triggered. The value ranges from 0 to 59. You can set a maximum of 60 values and use commas (,) to separate these values. However, the values must be unique.
If the trigger time is not within the validity period, the system selects a trigger time closest to the validity period for the scheduled job to be automatically executed at the first time. The following gives an example:
- Start Time: 1:20
- Cycle (hours): 3
- Trigger Time (minute): 10
- Validity Period: includes Start Time and End Time.
- Start Time: indicates the time when the scheduled configuration takes effect.
- End Time: This parameter is optional, which indicates the time when the scheduled job is automatically stopped. If this parameter is not set, the scheduled job keeps being automatically executed.
Figure 2 Scheduling job execution by hour-
For example, the settings shown in Figure 2 mean that the job will be automatically executed at 00:10 on January 1, 2023 for the first time, at 00:30 for the second time, and at 00:50 for the third time. It will be executed three times every two hours until 23:59 on December 31, 2023.
+For example, the settings shown in the above figure mean that the job will be automatically executed at 00:10 on January 1, 2023 for the first time, at 00:30 for the second time, and at 00:50 for the third time. It will be executed three times every two hours until 23:59 on December 31, 2023.
Scheduling Job Execution by Day
CDM allows jobs to be executed every several days.- Cycle (days): indicates the interval when a job is executed starting from the start time.
- Validity Period: includes Start Time and End Time.
- Start Time: indicates the time when the scheduled configuration takes effect, or the first time when the job is automatically executed.
- End Time: This parameter is optional, which indicates the time when the scheduled job is automatically stopped. If this parameter is not set, the scheduled job keeps being automatically executed.
Scheduling Job Execution by Day
CDM allows jobs to be executed every several days.- +- Cycle (days): indicates the interval when a job is executed starting from the start time.
- Validity Period: includes Start Time and End Time.
- Start Time: indicates the time when the scheduled configuration takes effect, or the first time when the job is automatically executed.
- End Time: This parameter is optional, which indicates the time when the scheduled job is automatically stopped. If this parameter is not set, the scheduled job keeps being automatically executed.
Figure 3 Scheduling job execution by day-
For example, the settings shown in Figure 3 mean that the job will be automatically executed at 00:00 on January 1, 2023 for the first time, and will be executed once every three days. The configuration is valid permanently.
- +For example, the settings shown in the above figure mean that the job will be automatically executed at 00:00 on January 1, 2023 for the first time, and will be executed once every three days. The configuration is valid permanently.
Scheduling Job Execution by Week
CDM allows jobs to be executed every several weeks.- +- Cycle (weeks): indicates the interval when a scheduled job is executed starting from the start time.
- Trigger Time (day): You can specify the day of each week when the job is automatically executed. One or more days can be selected at a time.
- Validity Period: includes Start Time and End Time.
- Start Time: indicates the time when the scheduled configuration takes effect.
- End Time: This parameter is optional, which indicates the time when the scheduled job is automatically stopped. If this parameter is not set, the scheduled job keeps being automatically executed.
Figure 4 Scheduling job execution by week-
For example, the settings shown in Figure 4 mean that the job will be automatically executed at 00:00 every Tuesday, Saturday, and Sunday every two weeks starting from 00:00 on January 1, 2023 until 23:59 on December 31, 2023.
+For example, the settings shown in the above figure mean that the job will be automatically executed at 00:00 every Tuesday, Saturday, and Sunday every two weeks starting from 00:00 on January 1, 2023 until 23:59 on December 31, 2023.
Scheduling Job Execution by Month
CDM allows jobs to be executed every several months.- +- Cycle (months): indicates the interval when a scheduled job is executed starting from the start time.
- Trigger Time (day): indicates the day of each month when the job is executed. The value ranges from 1 to 31. You can set multiple values and use commas (,) to separate these values. However, the values must be unique.
- Validity Period: includes Start Time and End Time.
- Start Time: indicates the time when the scheduled configuration takes effect. The automatic execution time is accurate to hour, minute, and second.
- End Time: This parameter is optional, which indicates the time when the scheduled job is automatically stopped. If this parameter is not set, the scheduled job keeps being automatically executed.
Figure 5 Scheduling job execution by month-
For example, the settings shown in Figure 5 mean that the job will be automatically executed at 00:00 on the 5th and 25th days of each month starting from 00:00 on January 1, 2023 until 23:59 on December 31, 2023.
+For example, the settings shown in the above figure mean that the job will be automatically executed at 00:00 on the 5th and 25th days of each month starting from 00:00 on January 1, 2023 until 23:59 on December 31, 2023.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0083.html b/docs/dataartsstudio/umn/dataartsstudio_01_0083.html index 4a1be4618..664f25faf 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0083.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0083.html @@ -1,52 +1,54 @@ --Parent topic: Managing Jobs+Parent topic: Creating a Job in a CDM ClusterJob Configuration Management
+Managing CDM Job Configuration
On the Settings tab page, you can perform the following operations:
Maximum Concurrent Extractors
Maximum number of concurrent extraction tasks in a cluster
+ This parameter is also available on the Cluster Configuration page. You can change its value either on this page or the Cluster Configuration page.
+CDM migrates data through data migration jobs. It works in the following way:-- When data migration jobs are submitted, CDM splits each job into multiple tasks based on the Concurrent Extractors parameter in the job configuration.
Jobs for different data sources may be split based on different dimensions. Some jobs may not be split based on the Concurrent Extractors parameter.
- CDM submits the tasks to the running pool in sequence. Tasks (defined by Maximum Concurrent Extractors) run concurrently. Excess tasks are queued.
By setting an appropriate number of concurrent extractors for a job and the maximum number of concurrent extractors for the cluster, you can accelerate migration. You can configure the number of concurrent extractors as follows:
+By setting appropriate values for the Concurrent Extractors and Maximum Concurrent Extractors parameters, you can accelerate migration.
- You are advised to set Maximum Concurrent Extractors to twice the number of vCPUs. For details, see Table 1.
-
Table 1 Recommended maximum number of concurrent extractors for a CDM cluster Flavor
+-Table 1 Recommended maximum number of concurrent extractors for a CDM cluster Flavor
vCPUs/Memory
+vCPUs/Memory
Recommended Maximum Concurrent Extractors
+Recommended Maximum Concurrent Extractors
cdm.large
+cdm.large
8 vCPUs, 16 GB
+8 vCPUs, 16 GB
16
+16
cdm.xlarge
+cdm.xlarge
16 vCPUs, 32 GB
+16 vCPUs, 32 GB
32
+32
cdm.4xlarge
+cdm.4xlarge
64 vCPUs, 128 GB
+64 vCPUs, 128 GB
128
+128
- Configure the number of concurrent extractors based on the following rules:
- When data is to be migrated to files, CDM does not support multiple concurrent tasks. In this case, set a single process to extract data.
- If each row of the table contains less than or equal to 1 MB data, data can be extracted concurrently. If each row contains more than 1 MB data, it is recommended that data be extracted in a single thread.
- Set Concurrent Extractors for a job based on Maximum Concurrent Extractors for the cluster. It is recommended that Concurrent Extractors is less than Maximum Concurrent Extractors.
- If the destination is DLI, you are advised to set the number of concurrent extractors to 1. Otherwise, data may fail to be written.
- Configure the number of concurrent extractors based on the following rules:
-- When data is to be migrated to files, CDM does not support multiple concurrent tasks. In this case, set a single process to extract data.
- If each row of the table contains less than or equal to 1 MB data, data can be extracted concurrently. If each row contains more than 1 MB data, it is recommended that data be extracted in a single thread.
- Set Concurrent Extractors for a job based on Maximum Concurrent Extractors for the cluster. It is recommended that the value of Concurrent Extractors is less than that of Maximum Concurrent Extractors.
- If the migration source is Hive and JDBC is used to read data, CDM does not support multi-concurrency. In this case, set the number of concurrent extractors to 1.
- If the destination is DLI, you are advised to set the number of concurrent extractors to 1. Otherwise, data may fail to be written.
Scheduled Backup/Restoration
This function depends on the OBS service.
-- Prerequisites
An OBS link has been created. For details, see Link to OBS.
+Scheduled Backup/Restoration
This function depends on the OBS service. Backup files cannot be automatically aged. You need to manually delete backup files on a regular basis.
+- Prerequisites
An OBS link has been created. For details, see OBS Link Parameters.
- Scheduled backup
On the Job Management page, click Settings and configure Scheduled Backup and its related parameters.
Table 2 Scheduled backup parameters Parameter
@@ -112,7 +114,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0084.html b/docs/dataartsstudio/umn/dataartsstudio_01_0084.html index 2ebf6874a..bf0b4143d 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0084.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0084.html @@ -1,30 +1,28 @@ --Parent topic: Managing Jobs+Parent topic: Creating a Job in a CDM ClusterManaging a Single Job
+Managing a CDM Job
Existing CDM jobs can be viewed, modified, deleted, started, and stopped. This section describes how to view and modify a job.
-Viewing a Job
- Viewing job status
The job status can be New, Pending, Booting, Running, Failed, Succeeded, or Stopped.
Pending indicates that the job is waiting to be scheduled by the system, and Booting indicates that the data to be migrated is being analyzed.
- - Viewing the historical records
On the Historical Record page, you can view job execution records, read/write statistics, and job execution logs.
+ - Viewing the historical records
You can view job execution results and historical information in the last 30 days, including job execution records, read/write statistics, and job execution logs.
- Viewing job logs
On the Historical Record page, you can view all logs of a job.
Alternatively, in the Operation column, choose to view the latest logs of the job.
- Viewing the JSON file of a job
You can directly edit the JSON file of a job, which is equivalent to modifying the parameter settings of the job.
- Querying the job statistics
You can open the preview window of a configured database job and view up to 1,000 pieces of data. By comparing the number of data records of the migration source and destination, you can check whether the migration was successful and whether data was lost.
Modifying a Job
- Modifying the job parameters
You can reconfigure job parameters, but you cannot reselect source and destination links.
+-Modifying a Job
- Modifying the job parameters
You can reconfigure job parameters and reselect source and destination links.
- Editing the JSON file of a job
You can directly edit the JSON file of a job, which is equivalent to modifying the parameter settings of the job.
Procedure
- Log in to the management console and choose Service List > Cloud Data Migration. In the left navigation pane, choose Cluster Management. Locate the target cluster and click Job Management.
- Click Historical Jobs to view all historical jobs executed in the latest month.
CDM stores the jobs executed in the last month, including one-time jobs (jobs that are automatically deleted after execution) and jobs that are executed periodically. You can view and re-execute the jobs on the Historical Jobs tab page.
-For a job that is executed periodically, a historical job is generated on the Historical Jobs tab page each time when the job is executed, regardless of whether the job is executed successfully. The names of historical jobs will be the same as the original job but with a random character string appended.
- - Click Table/File Migration. The job list is displayed. You can perform the following operations on a single job:
- Modify the job parameters: Click Edit in the Operation column to modify the job parameters.
- Run the job: Click Run in the Operation column to manually start the job.
- View the historical records: Click Historical Record in the Operation column. On the Historical Record page that is displayed, view the job's historical execution records and read/write statistics. Click Log to view the job logs.
- Delete the job: Choose in the Operation column to delete the job.
- Stop the job: Choose in the Operation column to stop the job.
- View the job JSON: Choose in the Operation column to view the job JSON.
- Edit the job JSON: Choose in the Operation column to edit the job JSON files, which is similar to modify the job parameters.
- Configure a scheduled job: Locate a job and choose . You can set the cycle for periodically executing the job. For details, see Scheduling Job Execution.
- View logs: Locate a job, click More in the Operation column, and select Log to view the latest log of the job.
You can also view all logs of the job on the Historical Record page.
+Procedure
- Log in to the management console and choose Service List > Cloud Data Migration. In the left navigation pane, choose Cluster Management. Locate the target cluster and click Job Management.
- Click Table/File Migration. The job list is displayed. You can perform the following operations on a single job:
- Modify the job parameters: Click Edit in the Operation column to modify the job parameters.
- Run the job: Click Run in the Operation column to manually start the job.
- View the historical records: Click Historical Record in the Operation column. On the Historical Record page that is displayed, view the job's historical execution records and read/write statistics. Click Log to view the job logs.
- Delete the job: Choose in the Operation column to delete the job.
- Stop the job: Choose in the Operation column to stop the job.
- View the job JSON: Choose in the Operation column to view the job JSON.
- Edit the job JSON: Choose in the Operation column to edit the job JSON files, which is similar to modify the job parameters.
- Configure a scheduled job: Locate a job and choose . You can set the cycle for periodically executing the job. For details, see Configuring a Scheduled CDM Job.
- View logs: Locate a job, click More in the Operation column, and select Log to view the latest log of the job.
You can also view all logs of the job on the Historical Record page.
- Retry the job: Locate a failed job, click More in the Operation column, and select Retry. The job will be automatically retried three times.
- After the modification, click Save or Save and Run.
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0085.html b/docs/dataartsstudio/umn/dataartsstudio_01_0085.html index eb6636f29..6502841cf 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0085.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0085.html @@ -1,17 +1,19 @@ --Parent topic: Managing Jobs+Parent topic: Creating a Job in a CDM ClusterManaging Jobs in Batches
-Scenario
This section describes how to manage CDM table/file migration jobs in batches. The following operations are involved:- Manage jobs by group.
- Run jobs in batches.
- Delete jobs in batches.
- Export jobs in batches.
- Import jobs in batches.
Managing CDM Jobs
+Scenario
This section describes how to manage CDM table/file migration jobs in batches. The following operations are supported:- Managing jobs by group
- Running jobs in batches
- Deleting jobs in batches
- Exporting jobs in batches
- Importing jobs in batches
You can export and import jobs in batches in the following scenarios:- Job migration between CDM clusters: You can migrate jobs from a cluster of an earlier version to a new version.
- Job backup: You can stop or delete CDM clusters to reduce costs. In this case, you can export the job scripts in batches and save them, and create a cluster and import the job scripts if necessary.
- Batch job creation: You can manually create a job and export the job configuration file in JSON format. Copy the content in the JSON file to the same file or new files, and then import the file/files to CDM to create jobs in batches.
Procedure
- Log in to the management console and choose Service List > Cloud Data Migration. In the left navigation pane, choose Cluster Management. Locate the target cluster and click Job Management.
- Click Table/File Migration. The job list is displayed. You can perform the following batch operations:
- Manage jobs by group.
CDM allows users to add, modify, search for, and delete job groups. When a group is deleted, all jobs in the group are deleted.
-In the third step of creating a job, if jobs have been assigned to different groups, you can display, start, or export jobs by group.
+When creating a job, if jobs have been assigned to different groups, you can display, start, or export jobs by group.
+ Starting jobs by group will run all jobs in the group. If user isolation is enabled, starting jobs by group will still run all jobs in the group even if otherIAM users in the a cloud platform account cannot view the jobs in the group. Therefore, you are not advised to start jobs by group in user isolation scenarios.
+ - Run jobs in batches.
After selecting one or more jobs, click Run to start these jobs in batches.
- Delete jobs in batches.
After selecting one or more jobs, click Delete to delete these jobs in batches.
- Export jobs in batches.
Click Export.
-Figure 1 Export+
Figure 1 Export
- All jobs and links: Export all jobs and links at a time.
- All jobs: Export all jobs at a time.
- All links: Export all links at a time.
- Jobs by name: Select the jobs to export and click OK.
- All jobs by groups: Select the group to export and click OK.
Exported jobs are stored in JSON files, which can be used as backups or imported to other clusters.
diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0088.html b/docs/dataartsstudio/umn/dataartsstudio_01_0088.html index 930ff9831..7daaac05b 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0088.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0088.html @@ -13,8 +13,8 @@ For security purposes, no link password is exported when jobs are exported. All passwords are replaced by Add password here.
@@ -26,7 +28,7 @@diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0086.html b/docs/dataartsstudio/umn/dataartsstudio_01_0086.html index 518d99c59..1fa88c00c 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0086.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0086.html @@ -31,7 +31,7 @@-Parent topic: Managing Jobs+Parent topic: Creating a Job in a CDM Cluster-Parent topic: DataArts Migration+Parent topic: DataArts Migration (CDM Jobs)-Creating a Cloud Search Service Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- Select Cloud Search Service and click Next. On the page that is displayed, configure the CSS link parameters.
- Name: Enter a custom link name, for example, csslink.
- Elasticsearch Server List: Enter the IP address and port number of the Cloud Search Service cluster (cluster later than 5.x). The format is ip:port. Use semicolons to separate multiple addresses. For example, 192.168.0.1:9200;192.168.0.2:9200.
- Username and Password: Enter the username and password used for logging in to the Cloud Search Service cluster. The user must have the read and write permissions on the database.
- Click Save. The Link Management page is displayed.
Creating an OBS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 1 Selecting a connector type-
- Select Object Storage Service (OBS) and click Next to configure parameters for the OBS link.
- Name: Enter a custom link name, for example, obslink.
- OBS Server and Port: Enter the actual OBS address information.
- AK and SK: Enter the AK and SK used for logging in to OBS.To obtain an access key, perform the following steps:
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 2.
+
Creating an OBS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 1 Selecting a connector type+
- Select Object Storage Service (OBS) and click Next to configure parameters for the OBS link.
- Name: Enter a custom link name, for example, obslink.
- OBS Server and Port: Enter the actual OBS address information.
- AK and SK: Enter the AK and SK used for logging in to OBS.To obtain an access key, perform the following steps:-
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 2.
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
- Click Save. The Link Management page is displayed.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0089.html b/docs/dataartsstudio/umn/dataartsstudio_01_0089.html index 74cf30452..3be21244e 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0089.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0089.html @@ -10,11 +10,11 @@Creating a Migration Job
- Choose to create a job for exporting data from OBS to Cloud Search Service.Figure 3 Creating a job for migrating data from OBS to Cloud Search Service+
Creating a Migration Job
- Choose to create a job for exporting data from OBS to Cloud Search Service.Figure 3 Creating a job for migrating data from OBS to Cloud Search Service
- Job Name: Enter a unique name.
- Source Job Configuration
- Source Link Name: Select the obslink link created in Creating an OBS Link.
- Bucket Name: Select the bucket from which the data will be migrated.
- Source Directory/File: Set this parameter to the path of the data to be migrated. You can migrate all directories and files in the bucket.
- File Format: Select CSV for migrating files to a data table.
- Retain the default values of the optional parameters in Show Advanced Attributes.
- Destination Job Configuration
- Destination Link Name: Select the csslink link created in Creating a Cloud Search Service Link.
- Index: Select the Elasticsearch index of the data to be written. You can also enter a new index. CDM automatically creates the index on Cloud Search Service.
- Type: Select the Elasticsearch type of the data to be written. You can enter a new type. CDM automatically creates a type at the migration destination.
- Retain the default values of the optional parameters in Show Advanced Attributes.
- Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields. See Figure 4.
- If the field mapping is incorrect, you can drag the fields to adjust the mapping.
- If the type is automatically created at the migration destination, you need to configure the type and name of each field.
- CDM supports field conversion during the migration.
- Click Next and set task parameters. Generally, retain the default values of all parameters.In this step, you can configure the following optional functions:
- Retry If Failed: Determine whether to automatically retry the job if it fails. Retain the default value Never.
- Group: Select the group to which the job belongs. The default group is DEFAULT. On the Job Management page, jobs can be displayed, started, or exported by group.
- Schedule Execution: Determine whether to automatically execute the job at a scheduled time. Retain the default value No in this example.
- Concurrent Extractors: Enter the number of concurrent extractors. An appropriate value improves migration efficiency. Retain the default value 1.
- Write Dirty Data: Specify this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS for future viewing. Before writing dirty data, create an OBS link on the CDM console. Retain the default value No so that dirty data is not recorded.
Figure 5 Configuring the task+
Figure 5 Configuring the task
- Click Save and Run. The Job Management page is displayed, on which you can view the job execution progress and result.
- After the job is successfully executed, in the Operation column of the job, click Historical Record to view the job's historical execution records and read/write statistics.
On the Historical Record page, click Log to view the job logs.
- Choose to create a job for exporting data from OBS to Cloud Search Service.
In this scenario, if the CDM cluster is used only to migrate data from OBS to DLI and does not need to migrate data of other data sources, there is no special requirements on the VPC, subnet, and security group of the CDM cluster. You can specify them based on your needs. CDM accesses DLI and OBS through the intranet. The flavor of the CDM cluster is selected based on the amount of data to be migrated. Generally, cdm.medium meets the requirements for most migration scenarios.
-Creating a DLI Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- Select Data Lake Insight, click Next, and configure the DLI link parameters. See Figure 1.
- Name: Enter a custom link name, for example, dlilink.
- AK and SK: Enter the AK and SK used for accessing the DLI database.
- Project ID: Enter the project ID of the region to which DLI belongs.
- Click Save. The Link Management page is displayed.
Creating an OBS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 2 Selecting a connector type-
- Select Object Storage Service (OBS) and click Next to configure parameters for the OBS link.
- Name: Enter a custom link name, for example, obslink.
- OBS Server and Port: Enter the actual OBS address information.
- AK and SK: Enter the AK and SK used for logging in to OBS.To obtain an access key, perform the following steps:
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 3.
+
Creating an OBS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 2 Selecting a connector type+
- Select Object Storage Service (OBS) and click Next to configure parameters for the OBS link.
- Name: Enter a custom link name, for example, obslink.
- OBS Server and Port: Enter the actual OBS address information.
- AK and SK: Enter the AK and SK used for logging in to OBS.To obtain an access key, perform the following steps:-
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 3.
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
- Click Save. The Link Management page is displayed.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0091.html b/docs/dataartsstudio/umn/dataartsstudio_01_0091.html index 5051350e4..13566eea4 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0091.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0091.html @@ -4,7 +4,7 @@Creating a Migration Job
- Choose to create a job for migrating data from OBS to DLI. See Figure 4.
+
Creating a Migration Job
- Choose to create a job for migrating data from OBS to DLI. See Figure 4.
- Job Name: Enter a custom job name.
- Source Link Name: Select the obslink link created in Creating an OBS Link.
- Bucket Name: Select the bucket from which the data is to be migrated.
- Source Directory/File: Set this parameter to the path of the data to be migrated.
- File Format: Select CSV or JSON for transferring files to a data table.
- Retain the default values of the optional parameters in Show Advanced Attributes.
- Destination Link Name: Select the dlilink link created in Creating a DLI Link.
- Resource Queue: Enter the resource queue to which the destination table belongs.
- Database Name: Enter the name of the database to which data is to be written.
- Table Name: Enter the name of the table to which data is to be written. CDM cannot automatically create tables on DLI. The table must be created on DLI in advance, and the field types and formats of the table must be consistent with those of the data to be migrated.
- Clear Before Importing Data: Choose whether to clear data in the destination table before data import. In this example, retain the default value.
- Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields.
- If the field mapping is incorrect, you can drag the fields to adjust the mapping.
- CDM supports field conversion during the migration.
- Click Next and set task parameters. Generally, retain the default values of all parameters.In this step, you can configure the following optional functions:
- Retry If Failed: Determine whether to automatically retry the job if it fails. Retain the default value Never.
- Group: Select the group to which the job belongs. The default group is DEFAULT. On the Job Management page, jobs can be displayed, started, or exported by group.
- Schedule Execution: Determine whether to automatically execute the job at a scheduled time. Retain the default value No in this example.
- Concurrent Extractors: Enter the number of concurrent extractors. An appropriate value improves migration efficiency. Retain the default value 1.
- Write Dirty Data: Specify this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS for future viewing. Before writing dirty data, create an OBS link on the CDM console. Retain the default value No so that dirty data is not recorded.
Figure 5 Configuring the task+Figure 5 Configuring the task - Click Save and Run. The Job Management page is displayed, on which you can view the job execution progress and result.
- After the job is successfully executed, in the Operation column of the job, click Historical Record to view the job's historical execution records and read/write statistics.
On the Historical Record page, click Log to view the job logs.
- Choose to create a job for migrating data from OBS to DLI. See Figure 4.
--Scenario
Cloud Search Service provides users with structured and unstructured data search, statistics, and report capabilities. This section describes how to use CDM to migrate data from the Oracle database to Cloud Search Service. The procedure is as follows:
Prerequisites
- You have subscribed to Cloud Search Service and obtained the IP address and port number of the Cloud Search Service cluster.
- You have obtained the IP address, name, username, and password of the Oracle database.
- If the Oracle database is deployed on an on-premises data center or a third-party cloud, ensure that an IP address that can be accessed from the public network has been configured for the Oracle database, or the VPN or Direct Connect between the on-premises data center and cloud has been established.
- You have uploaded the Oracle database driver on the page.
Prerequisites
- You have subscribed to Cloud Search Service and obtained the IP address and port number of the Cloud Search Service cluster.
- You have obtained the IP address, name, username, and password of the Oracle database.
- If the Oracle database is deployed on an on-premises data center or a third-party cloud, ensure that an IP address that can be accessed from the public network has been configured for the Oracle database, or the VPN or Direct Connect between the on-premises data center and has been established.
- You have uploaded the Oracle database driver on the page.
Creating a CDM Cluster and Binding an EIP to the Cluster
- Create a CDM cluster.The key configurations are as follows:@@ -18,14 +18,14 @@
- The flavor of the CDM cluster is selected based on the amount of data to be migrated. Generally, cdm.medium meets the requirements for most migration scenarios.
- The CDM and Cloud Search Service clusters must be in the same VPC. In addition, it is recommended that the CDM cluster be in the same subnet and security group as the Cloud Search Service cluster.
- If the same subnet and security group cannot be used for security purposes, ensure that a security group rule has been configured to allow the CDM cluster to access the Cloud Search Service cluster.
-Creating an Oracle Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- Select Oracle and click Next to configure parameters for the Oracle link.
- Name: Enter a custom link name, for example, oracle_link.
- Database Server and Port: Enter the address and port number of the Oracle server.
- Database Name: Enter the name of the Oracle database whose data is to be exported.
- Username and Password: Enter the username and password used for logging in to the Oracle database. The user must have the permission to read the Oracle metadata.
- Click Save. The Link Management page is displayed.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0092.html b/docs/dataartsstudio/umn/dataartsstudio_01_0092.html index 0bc834a12..5b37edce1 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0092.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0092.html @@ -6,7 +6,7 @@Creating a Migration Job
- Choose to create a job for exporting data from the Oracle database to Cloud Search Service.Figure 1 Creating a job for migrating data from Oracle to Cloud Search Service+
Creating a Migration Job
- Choose to create a job for exporting data from the Oracle database to Cloud Search Service.Figure 1 Creating a job for migrating data from Oracle to Cloud Search Service
- Job Name: Enter a unique name.
- Source Job Configuration
- Source Link Name: Select the oracle_link link created in Creating an Oracle Link.
- Schema/Tablespace: Enter the name of the database whose data is to be migrated.
- Table Name: Enter the name of the table to be migrated.
- Retain the default values of the optional parameters in Show Advanced Attributes.
- Destination Job Configuration
- Destination Link Name: Select the csslink link created in Creating a Cloud Search Service Link.
- Index: Select the Elasticsearch index of the data to be written. You can also enter a new index. CDM automatically creates the index on Cloud Search Service.
- Type: Select the Elasticsearch type of the data to be written. You can enter a new type. CDM automatically creates a type at the migration destination.
- Retain the default values of the optional parameters in Show Advanced Attributes.
- Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields, as shown in Figure 2.
- If the field mapping is incorrect, you can drag the fields to adjust the mapping.
- If the type is automatically created at the migration destination, you need to configure the type and name of each field.
- CDM supports field conversion during the migration.
- Click Next and set task parameters. Generally, retain the default values of all parameters.In this step, you can configure the following optional functions:
- Retry If Failed: Determine whether to automatically retry the job if it fails. Retain the default value Never.
- Group: Select the group to which the job belongs. The default group is DEFAULT. On the Job Management page, jobs can be displayed, started, or exported by group.
- Schedule Execution: Determine whether to automatically execute the job at a scheduled time. Retain the default value No in this example.
- Concurrent Extractors: Enter the number of concurrent extractors. An appropriate value improves migration efficiency. Retain the default value 1.
- Write Dirty Data: Specify this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS for future viewing. Before writing dirty data, create an OBS link on the CDM console. Retain the default value No so that dirty data is not recorded.
Figure 3 Configuring the task+Figure 3 Configuring the task - Click Save and Run. The Job Management page is displayed, on which you can view the job execution progress and result.
- After the job is successfully executed, in the Operation column of the job, click Historical Record to view the job's historical execution records and read/write statistics.
On the Historical Record page, click Log to view the job logs.
- Choose to create a job for exporting data from the Oracle database to Cloud Search Service.
Hive partitions are implemented by using the HDFS subdirectory function. Each subdirectory contains the column names and values of each partition. If there are multiple partitions, many HDFS subdirectories exist. It is not easy to load external data to each partition of the Hive table without relying on tools. With CDM, you can easily load data of the external data sources (relational databases, object storage services, and file system services) to Hive partition tables.
This section describes how to migrate data from the MySQL database to the MRS Hive partition table.
@@ -24,8 +24,8 @@Scenario
Suppose that there is a trip_data table in the MySQL database. The table stores cycling records such as the start time, end time, start sites, end sites, and rider IDs. For details about the fields in the trip_data table, see Figure 1.
- +The following describes how to use CDM to import the trip_data table in the MySQL database to the MRS Hive partition table. The procedure is as follows:
Creating a MySQL Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Select RDS for MySQL and click Next to set the link parameters.Figure 2 Creating a MySQL Link-
Click Show Advanced Attributes to view more optional parameters. For details, see Link to an RDS for MySQL/MySQL Database. Retain the default values for the optional parameters and configure the mandatory parameters described in Table 1.
+Creating a MySQL Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Select RDS for MySQL and click Next to set the link parameters.Figure 2 Creating a MySQL Link+
Click Show Advanced Attributes to view more optional parameters. For details, see RDS for MySQL/MySQL Database Link Parameters. Retain the default values for the optional parameters and configure the mandatory parameters described in Table 1.
Table 1 MySQL link parameters Parameter
Creating a Hive Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 3 Selecting a connector type+
Creating a Hive Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 3 Selecting a connector type
- Select MRS Hive and click Next to configure parameters for the MRS Hive link.
Table 2 lists the parameters. Configure these parameters based on your actual situation.
Table 2 MRS Hive link parameters Parameter
@@ -132,7 +132,9 @@Manager IP
Floating IP address of MRS Manager. Click Select next to the Manager IP text box to select an MRS cluster. CDM automatically fills in the authentication information.
+Floating IP address of MRS Manager. Click Select next to the Manager IP text box to select an MRS cluster. CDM automatically fills in the authentication information.NOTE:+DataArts Studio does not support MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2, and only supports MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1.
+127.0.0.1
ldapUsername
This parameter is mandatory when Enable ldap is enabled.
+This parameter is mandatory when Enable ldap is enabled.
Enter the username configured when LDAP authentication was enabled for MRS Hive.
-
@@ -187,7 +189,7 @@ldapPassword
This parameter is mandatory when Enable ldap is enabled.
+This parameter is mandatory when Enable ldap is enabled.
Enter the password configured when LDAP authentication was enabled for MRS Hive.
-
@@ -204,7 +206,7 @@This parameter is mandatory when OBS storage support is enabled. The account corresponding to the AK/SK pair must have the OBS Buckets Viewer permission. Otherwise, OBS cannot be accessed and the "403 AccessDenied" error is reported.
You need to create an access key for the current account and obtain an AK/SK pair.
-- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 4.
+
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 4.
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.NOTE:
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0098.html b/docs/dataartsstudio/umn/dataartsstudio_01_0098.html index b799ca4a4..89d0e72a2 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0098.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0098.html @@ -13,85 +13,84 @@Creating a Migration Job
- Click the Table/File Migration tab and then Create Job.
Set Clear Data Before Import to Yes, so that the data in the Hive table will be cleared before data import.
- After configuring the parameters, click Next to go to the Map Field page shown in Figure 5.
Map the fields of the MySQL table and Hive table. The Hive table has three more fields y, ym, and ymd than the MySQL table, which are the Hive partition fields. Because the fields of the source table cannot be directly mapped to the destination table, you need to configure an expression to extract data from the StartDate field in the source table.
- - - Click
to display the Converter List dialog box, and then choose . See Figure 6.The expressions for the y, ym, and ymd fields are as follows:
+ + - Click
to display the Converter List dialog box, and then choose . See Figure 6.The expressions for the y, ym, and ymd fields are as follows:
DateUtils.format(DateUtils.parseDate(row[2],"yyyy-MM-dd HH:mm:ss.SSS"),"yyyy")
DateUtils.format(DateUtils.parseDate(row[2],"yyyy-MM-dd HH:mm:ss.SSS"),"yyyyMM")
DateUtils.format(DateUtils.parseDate(row[2],"yyyy-MM-dd HH:mm:ss.SSS"),"yyyyMMdd")
- + The expressions in CDM support field conversion of common character strings, dates, and values.
- Click Next and set task parameters. Generally, retain the default values of all parameters.In this step, you can configure the following optional functions:
- Retry If Failed: Determine whether to automatically retry the job if it fails. Retain the default value Never.
- Group: Select the group to which the job belongs. The default group is DEFAULT. On the Job Management page, jobs can be displayed, started, or exported by group.
- Schedule Execution: Determine whether to automatically execute the job at a scheduled time. Retain the default value No in this example.
- Concurrent Extractors: Enter the number of concurrent extractors. An appropriate value improves migration efficiency. Retain the default value 1.
- Write Dirty Data: Specify this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS for future viewing. Before writing dirty data, create an OBS link on the CDM console. Retain the default value No so that dirty data is not recorded.
Figure 7 Configuring the task+Figure 7 Configuring the task - Click Save and Run. The Job Management page is displayed, on which you can view the job execution progress and result.
- After the job is successfully executed, in the Operation column of the job, click Historical Record to view the job's historical execution records and read/write statistics.
On the Historical Record page, click Log to view the job logs.
Creating a MySQL Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Select RDS for MySQL and click Next to set the link parameters.Figure 1 Creating a MySQL link-
Click Show Advanced Attributes to view more optional parameters. For details, see Link to an RDS for MySQL/MySQL Database. Retain the default values of the optional parameters and configure the mandatory parameters according to Table 1.
+Creating a MySQL Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Select MySQL and click Next. On the displayed page, configure MySQL link parameters.
Click Show Advanced Attributes to view more optional parameters. For details, see RDS for MySQL/MySQL Database Link Parameters. Retain the default values of the optional parameters and configure the mandatory parameters according to Table 1.
-Table 1 MySQL link parameters Parameter
+Table 1 MySQL link parameters Parameter
Description
+Description
Example Value
+Example Value
Name
+Name
Unique link name
+Unique link name
mysqllink
+mysqllink
Database Server
+Database Server
IP address or domain name of the MySQL database server
+IP address or domain name of the MySQL database server
-
+-
Port
+Port
MySQL database port
+MySQL database port
3306
+3306
Database Name
+Database Name
Name of the MySQL database
+Name of the MySQL database
sqoop
+sqoop
Username
+Username
User who has the read, write, and delete permissions on the MySQL database
+User who has the read, write, and delete permissions on the MySQL database
admin
+admin
Password
+Password
Password of the user
+Password of the user
-
+-
Use Local API
+Use Local API
Whether to use the local API of the database for acceleration. (The system attempts to enable the local_infile system variable of the MySQL database.)
+Whether to use the local API of the database for acceleration. (The system attempts to enable the local_infile system variable of the MySQL database.)
Yes
+Yes
Use Agent
+Use Agent
Whether to extract data from the data source through an agent
+The agent function will be unavailable soon and does not need to be configured.
No
+-
local_infile Character Set
+local_infile Character Set
When using local_infile to import data to MySQL, you can configure the encoding format.
+When using local_infile to import data to MySQL, you can configure the encoding format.
utf8
+utf8
Driver Version
+Driver Version
Before connecting CDM to a relational database, you need to upload the JDK 8 .jar driver of the relational database. Download the MySQL driver 5.1.48 from https://downloads.mysql.com/archives/c-j/, obtain mysql-connector-java-5.1.48.jar, and upload it.
+Before connecting CDM to a relational database, you need to upload the JDK 8 .jar driver of the relational database. Download the MySQL driver 5.1.48 from https://downloads.mysql.com/archives/c-j/, obtain mysql-connector-java-5.1.48.jar, and upload it.
-
+-
-Creating an RDS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 2 Selecting a connector type+-
Creating an RDS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 1 Selecting a connector type
- Select RDS for MySQL and click Next to configure parameters for the RDS for MySQL link.
- Name: Enter a custom link name, for example, rds_link.
- Database Server and Port: Enter the address information about the RDS for MySQL database.
- Database Name: Enter the name of the RDS for MySQL database.
- Username and Password: Enter the username and password used for logging in to the database.
- During RDS link creation, if Use Local API in Show Advanced Attributes is set to Yes, you can use the LOAD DATA function provided by MySQL to speed up data import.
- The LOAD DATA function is disabled by default on RDS for MySQL, so you need to modify the parameter group of the MySQL instance and set local_infile to ON to enable this function.
- If the local_infile parameter group cannot be edited, it is the default parameter group. You need to create a parameter group and modify its value, and apply it to the MySQL instance of RDS.
- Click Save. The Link Management page is displayed.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0099.html b/docs/dataartsstudio/umn/dataartsstudio_01_0099.html index b8ee06d29..a0e7d116b 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0099.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0099.html @@ -4,7 +4,7 @@Creating an Entire DB Migration Job
- After the two links are created, choose to create a migration job. See Figure 3.
+
Creating an Entire DB Migration Job
- After the two links are created, choose to create a migration job. See Figure 2.
- Job Name: Enter a name for the entire DB migration job.
- Source Job Configuration
- Source Link Name: Select the mysqllink created in Creating a MySQL Link.
- Schema/Tablespace: Select the on-premises MySQL database from which data is to be exported.
- Destination Job Configuration
- Destination Link Name: Select the rds_link link created in Creating an RDS Link.
- Schema/Tablespace: Select the name of the RDS database to which data is to be imported.
- Auto Table Creation: Select Auto creation, which indicates that CDM automatically creates tables in the RDS database when tables of the on-premises MySQL database do not exist in the RDS database.
- Clear Data Before Import: Select Yes, which indicates that when a table with the same name as the table in the on-premises MySQL database exists in the RDS database, CDM clears data in the table on RDS.
- Constraint Conflict Handling: Select insert into.
- Retain the default values of the optional parameters in Show Advanced Attributes.
- After the two links are created, choose to create a migration job. See Figure 2.
-Scenario
CSS provides users with structured and unstructured data search, statistics, and report capabilities. This section describes how to use CDM to migrate the entire Elasticsearch database to Cloud Search Service. The procedure is as follows:
Prerequisites
- You have sufficient EIP quota.
- You have subscribed to CSS and obtained the IP address and port number of the CSS cluster.
- You have obtained the IP address, port number, username, and password of the on-premises Elasticsearch database server.
If the Elasticsearch server is deployed on an on-premises data center or a third-party cloud, ensure that an IP address that can be accessed from the public network has been configured for the Elasticsearch server, or the VPN or Direct Connect between the on-premises data center and cloud has been established.
+Prerequisites
- You have sufficient EIP quota.
- You have subscribed to CSS and obtained the IP address and port number of the CSS cluster.
- You have obtained the IP address, port number, username, and password of the on-premises Elasticsearch database server.
If the Elasticsearch server is deployed on an on-premises data center or a third-party cloud, ensure that an IP address that can be accessed from the public network has been configured for the Elasticsearch server, or the VPN or Direct Connect between the on-premises data center and has been established.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0100.html b/docs/dataartsstudio/umn/dataartsstudio_01_0100.html index b756cc6e4..ab3f6ad83 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0100.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0100.html @@ -12,85 +12,84 @@Creating a CDM Cluster and Binding an EIP to the Cluster
- Create a CDM cluster.The key configurations are as follows:
- The flavor of the CDM cluster is selected based on the amount of data to be migrated. Generally, cdm.medium meets the requirements for most migration scenarios.
- The CDM and Cloud Search Service clusters must be in the same VPC. In addition, it is recommended that the CDM cluster be in the same subnet and security group as the Cloud Search Service cluster.
- If the same subnet and security group cannot be used for security purposes, ensure that a security group rule has been configured to allow the CDM cluster to access the Cloud Search Service cluster.
-Creating an Elasticsearch Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Select Elasticsearch and click Next to configure parameters for the Elasticsearch link. The parameters are the same as those for the CSS link.
- Name: Enter a custom link name, for example, es_link.
- Elasticsearch Server List: Enter the IP address and port number of the on-premises Elasticsearch database. Use semicolons to separate multiple addresses.
- Click Save. The Link Management page is displayed.
@@ -34,3 +34,10 @@Creating an Entire DB Migration Job
- Choose to create an entire DB migration job.Figure 1 Creating an entire DB migration job+
Creating an Entire DB Migration Job
- Choose to create an entire DB migration job.Figure 1 Creating an entire DB migration job
- Job Name: Enter a unique name.
- Source Job Configuration
- Source Link Name: Select the es_link link created in Creating an Elasticsearch Link.
- Index: Click the icon next to the text box to select an index in the on-premises Elasticsearch database or manually enter an index name. The name can contain only lowercase letters. If multiple indexes need to be migrated at a time, set this parameter to a wildcard character. CDM migrates all indexes that meet the wildcard condition. For example, if this parameter is set to cdm*, CDM migrates all indexes starting with cdm, such as cdm01, cdmB3, cdm_45 and so on.
- Destination Job Configuration
- Destination Link Name: Select the csslink link created in Creating a Cloud Search Service Link.
- Index: Enter the index of the data to be written. You can select an existing index in Cloud Search Service or manually enter an index name that does not exist. The name can contain only lowercase letters. CDM automatically creates the index in Cloud Search Service. If multiple indexes are migrated at a time, this parameter cannot be configured. CDM automatically creates indexes at the migration destination.
- Clear Data Before Import: If the selected index already exists in Cloud Search Service, you can choose whether to clear the data in the index before importing data. If you select No, the data is added to the index.
- Click Save and Run. The Job Management page is displayed, on which you can view the job execution progress and result.
A sub-job will be generated for each type in the on-premises Elasticsearch index for concurrent execution. You can click the job name to view the sub-job progress.
-
- After the job is successfully executed, in the Operation column of the job, click Historical Record to view the job's historical execution records, read/write statistics, and job logs (only the sub-jobs have job logs).Figure 2 Historical Record+
- After the job is successfully executed, in the Operation column of the job, click Historical Record to view the job's historical execution records, read/write statistics, and job logs (only the sub-jobs have job logs).Figure 2 Historical Record
- Choose to create an entire DB migration job.
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
Creating a MySQL Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Select RDS for MySQL and click Next to set the link parameters.Figure 1 Creating a MySQL link-
Click Show Advanced Attributes to view more optional parameters. For details, see Link to an RDS for MySQL/MySQL Database. Retain the default values of the optional parameters and configure the mandatory parameters according to Table 1.
+Creating a MySQL Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Select MySQL and click Next. On the displayed page, configure MySQL link parameters.
Click Show Advanced Attributes to view more optional parameters. For details, see RDS for MySQL/MySQL Database Link Parameters. Retain the default values of the optional parameters and configure the mandatory parameters according to Table 1.
-Table 1 MySQL link parameters Parameter
+Table 1 MySQL link parameters Parameter
Description
+Description
Example Value
+Example Value
Name
+Name
Unique link name
+Unique link name
mysqllink
+mysqllink
Database Server
+Database Server
IP address or domain name of the MySQL database server
+IP address or domain name of the MySQL database server
-
+-
Port
+Port
MySQL database port
+MySQL database port
3306
+3306
Database Name
+Database Name
Name of the MySQL database
+Name of the MySQL database
sqoop
+sqoop
Username
+Username
User who has the read, write, and delete permissions on the MySQL database
+User who has the read, write, and delete permissions on the MySQL database
admin
+admin
Password
+Password
Password of the user
+Password of the user
-
+-
Use Local API
+Use Local API
Whether to use the local API of the database for acceleration. (The system attempts to enable the local_infile system variable of the MySQL database.)
+Whether to use the local API of the database for acceleration. (The system attempts to enable the local_infile system variable of the MySQL database.)
Yes
+Yes
Use Agent
+Use Agent
Whether to extract data from the data source through an agent
+The agent function will be unavailable soon and does not need to be configured.
No
+-
local_infile Character Set
+local_infile Character Set
When using local_infile to import data to MySQL, you can configure the encoding format.
+When using local_infile to import data to MySQL, you can configure the encoding format.
utf8
+utf8
Driver Version
+Driver Version
Before connecting CDM to a relational database, you need to upload the JDK 8 .jar driver of the relational database. Download the MySQL driver 5.1.48 from https://downloads.mysql.com/archives/c-j/, obtain mysql-connector-java-5.1.48.jar, and upload it.
+Before connecting CDM to a relational database, you need to upload the JDK 8 .jar driver of the relational database. Download the MySQL driver 5.1.48 from https://downloads.mysql.com/archives/c-j/, obtain mysql-connector-java-5.1.48.jar, and upload it.
-
+-
-Creating an OBS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 2 Selecting a connector type-
- Select Object Storage Service (OBS) and click Next to configure parameters for the OBS link.
- Name: Enter a custom link name, for example, obslink.
- OBS Server and Port: Enter the actual OBS address information.
- AK and SK: Enter the AK and SK used for logging in to OBS.To obtain an access key, perform the following steps:
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 3.
+
Creating an OBS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 1 Selecting a connector type+
- Select Object Storage Service (OBS) and click Next to configure parameters for the OBS link.
- Name: Enter a custom link name, for example, obslink.
- OBS Server and Port: Enter the actual OBS address information.
- AK and SK: Enter the AK and SK used for logging in to OBS.To obtain an access key, perform the following steps:-
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 2.
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
- Click Save. The Link Management page is displayed.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0101.html b/docs/dataartsstudio/umn/dataartsstudio_01_0101.html index 6a7eafad9..e84bfd5e0 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0101.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0101.html @@ -12,85 +12,84 @@Creating a Migration Job
- Choose to create a job for exporting data from the MySQL database to OBS.Figure 4 Creating a job for migrating data from MySQL to OBS+
Creating a Migration Job
- Choose to create a job for exporting data from the MySQL database to OBS.Figure 3 Creating a job for migrating data from MySQL to OBS
- Job Name: Enter a unique name.
- Source Job Configuration
- Source Link Name: Select the mysqllink created in Creating a MySQL Link.
- Use SQL Statement: Select No.
- Schema/Tablespace: name of the schema or tablespace from which data is to be extracted
- Table Name: name of the table from which data is to be extracted
- Retain the default values of other optional parameters.
- Destination Job Configuration
- Destination Link Name: Select the obslink created in Creating an OBS Link.
- Bucket Name: Select the bucket from which the data will be migrated.
- Write Directory: Enter the directory to which data is to be written on the OBS server.
- File Format: Select CSV.
- Retain the default values of the optional parameters in Show Advanced Attributes.
- Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields, as shown in Figure 5.
- If the field mapping is incorrect, you can drag the fields to adjust the mapping.
- The expressions in CDM support field conversion of common character strings, dates, and values.
- Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields, as shown in Figure 4.
- If the field mapping is incorrect, you can drag the fields to adjust the mapping.
- The expressions in CDM support field conversion of common character strings, dates, and values.
- Click Next and set task parameters. Generally, retain the default values of all parameters.In this step, you can configure the following optional functions:
- Retry Upon Failure: If the job fails to be executed, you can determine whether to automatically retry. Retain the default value Never.
- Group: Select the group to which the job belongs. The default group is DEFAULT. On the Job Management page, jobs can be displayed, started, or exported by group.
- Schedule Execution: Enable it if you need to configure scheduled jobs. Retain the default value No.
- Concurrent Extractors: Enter the number of extractors to be concurrently executed. CDM supports concurrent extraction of MySQL data. If indexes are configured for the source table, you can increase the number of concurrent extractors to accelerate the migration.
- Write Dirty Data: Specify this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS for future viewing. Before writing dirty data, create an OBS link. For file-to-table data migration, you are advised to write dirty data.
- Delete Job After Completion: Retain the default value Do not delete. You can also set this parameter to Delete to prevent an accumulation of too many migration jobs.
- Click Save and Run. The Job Management page is displayed, on which you can view the job execution progress and result.
- After the job is successfully executed, in the Operation column of the job, click Historical Record to view the job's historical execution records and read/write statistics.
On the Historical Record page, click Log to view the job logs.
@@ -127,3 +126,10 @@
- Choose to create a job for exporting data from the MySQL database to OBS.
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
Creating a MySQL Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Select RDS for MySQL and click Next to set the link parameters.Figure 1 Creating a MySQL link-
Click Show Advanced Attributes to view more optional parameters. For details, see Link to an RDS for MySQL/MySQL Database. Retain the default values of the optional parameters and configure the mandatory parameters according to Table 1.
+Creating a MySQL Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Select MySQL and click Next. On the displayed page, configure MySQL link parameters.
Click Show Advanced Attributes to view more optional parameters. For details, see RDS for MySQL/MySQL Database Link Parameters. Retain the default values of the optional parameters and configure the mandatory parameters according to Table 1.
-Table 1 MySQL link parameters Parameter
+Table 1 MySQL link parameters Parameter
Description
+Description
Example Value
+Example Value
Name
+Name
Unique link name
+Unique link name
mysqllink
+mysqllink
Database Server
+Database Server
IP address or domain name of the MySQL database server
+IP address or domain name of the MySQL database server
-
+-
Port
+Port
MySQL database port
+MySQL database port
3306
+3306
Database Name
+Database Name
Name of the MySQL database
+Name of the MySQL database
sqoop
+sqoop
Username
+Username
User who has the read, write, and delete permissions on the MySQL database
+User who has the read, write, and delete permissions on the MySQL database
admin
+admin
Password
+Password
Password of the user
+Password of the user
-
+-
Use Local API
+Use Local API
Whether to use the local API of the database for acceleration. (The system attempts to enable the local_infile system variable of the MySQL database.)
+Whether to use the local API of the database for acceleration. (The system attempts to enable the local_infile system variable of the MySQL database.)
Yes
+Yes
Use Agent
+Use Agent
Whether to extract data from the data source through an agent
+The agent function will be unavailable soon and does not need to be configured.
No
+-
local_infile Character Set
+local_infile Character Set
When using local_infile to import data to MySQL, you can configure the encoding format.
+When using local_infile to import data to MySQL, you can configure the encoding format.
utf8
+utf8
Driver Version
+Driver Version
Before connecting CDM to a relational database, you need to upload the JDK 8 .jar driver of the relational database. Download the MySQL driver 5.1.48 from https://downloads.mysql.com/archives/c-j/, obtain mysql-connector-java-5.1.48.jar, and upload it.
+Before connecting CDM to a relational database, you need to upload the JDK 8 .jar driver of the relational database. Download the MySQL driver 5.1.48 from https://downloads.mysql.com/archives/c-j/, obtain mysql-connector-java-5.1.48.jar, and upload it.
-
+-
Use Agent
Whether to extract data from the data source through an agent
+The agent function will be unavailable soon and does not need to be configured.
Yes
+-
Agent
Click Select and select the created agent.
+The agent function will be unavailable soon and does not need to be configured.
-
- Click Save.
-+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0103.html b/docs/dataartsstudio/umn/dataartsstudio_01_0103.html index 729d58a47..07de4a0e0 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0103.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0103.html @@ -13,13 +13,13 @@ -Creating a Migration Job
- Choose to create a job for exporting data from the MySQL database to DWS.Figure 2 Creating a job for migrating data from MySQL to DWS+
Creating a Migration Job
- Choose to create a job for exporting data from the MySQL database to DWS.Figure 1 Creating a job for migrating data from MySQL to DWS
- Job Name: Enter a unique name.
- Source Job Configuration
- Source Link Name: Select the mysqllink created in Creating a MySQL Link.
- Use SQL Statement: Select No.
- Schema/Tablespace: name of the schema or tablespace from which data is to be extracted
- Table Name: name of the table from which data is to be extracted
- Retain the default values of other optional parameters.
- Destination Job Configuration
- Destination Link Name: Select the dwslink created in Creating a DWS Link.
- Schema/Tablespace: Select the DWS database to which data is to be written.
- Auto Table Creation: This parameter is displayed only when both the migration source and destination are relational databases.
- Table Name: Name of the table to which data is to be written. You can enter a table name that does not exist. CDM automatically creates the table in DWS.
- isCompress: whether to compress data. If you select Yes, high-level compression will be performed. CDM applies to compression scenarios where the I/O read/write volume is large and the CPU is sufficient (the computing load is relatively low).
- Orientation: You can create row- or column-store tables as needed. Generally, if a table contains many columns (called a wide table) and its query involves only a few columns, column storage is recommended. If a table contains only a few columns and a query includes most of the fields, row storage is recommended.
- Extend char length: If the data encoding formats of the migration source and destination are different, the character length of the automatic table creation may be insufficient. If you select Yes for this parameter, the character length will be increased by three times during automatic table creation.
- Clear Data Before Import: whether to clear data in the destination table before the migration task starts.
- Destination Job Configuration
- Destination Link Name: Select the dwslink created in Creating a DWS Link.
- Schema/Tablespace: Select the DWS database to which data is to be written.
- Auto Table Creation: This parameter is displayed only when both the migration source and destination are relational databases.
- Table Name: Name of the table to which data is to be written. You can enter a table name that does not exist. CDM automatically creates the table in DWS.
- isCompress: whether to compress data. If you select Yes, high-level compression will be performed. CDM applies to compression scenarios where the I/O read/write volume is large and the CPU is sufficient (the computing load is relatively low).
- Orientation: You can create row- or column-store tables as needed. Generally, if a table contains many columns (called a wide table) and its query involves only a few columns, column storage is recommended. If a table contains only a few columns and a query includes most of the fields, row storage is recommended.
- Extend char length: If the data encoding formats of the migration source and destination are different, the character length of the automatic table creation may be insufficient. If you select Yes for this parameter, the character length will be increased by three times during automatic table creation.
- Clear Data Before Import: whether to clear data in the destination table before the migration task starts.
- Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields, as shown in Figure 3.
- If the field mapping is incorrect, you can drag the fields to adjust the mapping.
- The expressions in CDM support field conversion of common character strings, dates, and values.
- Click Next. The Map Field page is displayed. CDM automatically matches the source and destination fields, as shown in Figure 2.
- If the field mapping is incorrect, you can drag the fields to adjust the mapping.
- The expressions in CDM support field conversion of common character strings, dates, and values.
- Click Next and set task parameters. Generally, retain the default values of all parameters.In this step, you can configure the following optional functions:
- Retry Upon Failure: If the job fails to be executed, you can determine whether to automatically retry. Retain the default value Never.
- Group: Select the group to which the job belongs. The default group is DEFAULT. On the Job Management page, jobs can be displayed, started, or exported by group.
- Schedule Execution: Enable it if you need to configure scheduled jobs. Retain the default value No.
- Concurrent Extractors: Enter the number of extractors to be concurrently executed. You can increase the value of this parameter to improve migration efficiency.
- Write Dirty Data: Dirty data may be generated during data migration between tables. You are advised to select Yes.
- Delete Job After Completion: Retain the default value Do not delete.
- Click Save and Run. The Job Management page is displayed, on which you can view the job execution progress and result.
- After the job is successfully executed, in the Operation column of the job, click Historical Record to view the job's historical execution records and read/write statistics.
On the Historical Record page, click Log to view the job logs.
@@ -195,3 +194,10 @@
- Choose to create a job for exporting data from the MySQL database to DWS.
Creating an MRS HDFS Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.Figure 1 Selecting a connector type+-
Creating an MRS HDFS Link
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.Figure 1 Selecting a connector type
- Select MRS HDFS and click Next to configure parameters for the MRS HDFS link.
- Name: Enter a custom link name, for example, mrs_hdfs_link.
- Manager IP: IP address of MRS Manager. Click Select next to the Manager IP text box to select a created MRS cluster. CDM automatically fills in the authentication information.
- Username: If Authentication Method is set to KERBEROS, set the username and password for logging in to MRS Manager.
If you need to create a snapshot when exporting a directory from HDFS, the user configured here must have the administrator permission on HDFS.
- Password: password for logging in to MRS Manager
- Authentication Method: authentication method for accessing MRS
- Run Mode: Select the running mode of the HDFS link.
Creating an OBS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 2 Selecting a connector type-
- Select Object Storage Service (OBS) and click Next to configure parameters for the OBS link.
- Name: Enter a custom link name, for example, obslink.
- OBS Server and Port: Enter the actual OBS address information.
- AK and SK: Enter the AK and SK used for logging in to OBS.To obtain an access key, perform the following steps:
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 3.
+
Creating an OBS Link
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.Figure 2 Selecting a connector type+
- Select Object Storage Service (OBS) and click Next to configure parameters for the OBS link.
- Name: Enter a custom link name, for example, obslink.
- OBS Server and Port: Enter the actual OBS address information.
- AK and SK: Enter the AK and SK used for logging in to OBS.To obtain an access key, perform the following steps:-
- Log in to the management console, move the cursor to the username in the upper right corner, and select My Credentials from the drop-down list.
- On the My Credentials page, choose Access Keys, and click Create Access Key. See Figure 3.
- Click OK and save the access key file as prompted. The access key file will be saved to your browser's configured download location. Open the credentials.csv file to view Access Key Id and Secret Access Key.
- Only two access keys can be added for each user.
- To ensure access key security, the access key is automatically downloaded only when it is generated for the first time and cannot be obtained from the management console later. Keep them properly.
- Click Save. The Link Management page is displayed.
+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0104.html b/docs/dataartsstudio/umn/dataartsstudio_01_0104.html index 7d9a3cd6b..67aa75b48 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0104.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0104.html @@ -1,15 +1,16 @@Creating a Migration Job
- Choose to create a job for exporting data from the MRS HDFS database to OBS.Figure 4 Creating a job for migrating data from MRS HDFS to OBS+
Creating a Migration Job
- Choose to create a job for exporting data from the MRS HDFS database to OBS.Figure 4 Creating a job for migrating data from MRS HDFS to OBS
- Job Name: Enter a unique name.
- Source Job Configuration
- Source Link Name: Select the hdfs_llink created in Creating an MRS HDFS Link.
- Source Directory/File: Enter the directory or file path of the data to be migrated.
- File Format: Select the file format used for data transmission. Select Binary. If files are transferred without being parsed, the file format does not have to be Binary. This applies to file copy.
- Retain the default values of other optional parameters.
- Destination Job Configuration
- Destination Link Name: Select the obs_link created in Creating an OBS Link.
- Bucket Name: Select the bucket from which the data will be migrated.
- Write Directory: Enter the directory to which data is to be written on the OBS server.
- File Format: Select Binary.
- Retain the default values of the optional parameters in Show Advanced Attributes.
- Choose to create a job for exporting data from the MRS HDFS database to OBS.
Configuring Field Converters
-Scenario
- After the job parameters are configured, field mapping needs to be configured. You can click
in the Operation column to create a field converter. - If files are migrated between FTP, SFTP, OBS, and HDFS and the migration source's File Format is set to Binary, files will be directly transferred, free from field mapping.
-Scenario
- After the job parameters are configured, field mapping needs to be configured. You can click
in the Operation column to create a field converter. - If files are migrated between FTP, SFTP, OBS, and HDFS and the migration source's File Format is set to Binary, files will be directly transferred, free from field mapping.
You can create a field converter on the Map Field page when creating a table/file migration job.Figure 1 Creating a field converter+
You can create a field converter on the Map Field page when creating a table/file migration job.Figure 1 Creating a field converter
CDM can convert fields during migration. Currently, the following field converters are supported:
-+ + \ No newline at end of file diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0107.html b/docs/dataartsstudio/umn/dataartsstudio_01_0107.html new file mode 100644 index 000000000..67ff117ba --- /dev/null +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0107.html @@ -0,0 +1,23 @@ + + +Constraints
- If Use SQL Statement is set to Yes in the source job configuration, converters cannot be created.
- On the Map Field tab page, if CDM fails to obtain all columns by obtaining sample values (for example, when data is exported from HBase, CloudTable, or MongoDB, there is a high probability that CDM failed to obtain all columns), you can click
and select Add a new field to add new fields to ensure that the data imported to the migration destination is complete. - When a relational database, Hive, DLI, or MRS Hudi is used as the migration source, sample values cannot be obtained.
- When SQLServer is the destination, fields of the timestamp type cannot be written. You must change their type (for example, to datetime) so that they can be written.
- Column names are displayed when the source of the migration job is OBS, CSV files are to be migrated, and parameter Extract first row as columns is set to Yes.
- Field converters configuration is not involved when the binary format is used to migrate files to files.
- In the automatic table creation scenario, you need to manually add fields to the destination table in advance and then add fields to the field mapping.
- After a field is added, its sample value is not displayed on the console. This does not affect the field value transmission. CDM directly writes the field value to the destination end.
- If the field mapping is incorrect, you can adjust the field mapping by dragging fields or clicking
to map fields in batches. - An expression processes the data of a field. When creating an expression converter, you are not advised to use a time macro. If you need to use a time macro, use either of the following methods (if the source is of the file type, only Method 1 is supported):
- Method 1: When creating an expression converter, use two single quotation marks ('') to enclose the expression.
For example, if expression ${dateformat(yyyy-MM-dd)} is not enclosed in quotation marks, the hyphen (-) in the value 2017-10-16 parsed from the expression will be recognized as a minus sign, and further calculation will be performed to generate result 1991, which is incorrect. If you enclose the expression in quotation marks, that is, '${dateformat(yyyy-MM-dd)}', you will obtain '2017-10-16', which is correct.
-
-
- Method 2: Add a custom source field, enter a macro variable of date and time for Example Value, and map the field to a destination field again.
+
Constraints
- If Use SQL Statement is set to Yes in the source job configuration, converters cannot be created.
- On the Map Field tab page, if CDM fails to obtain all columns by obtaining sample values (for example, when data is exported from HBase, CloudTable, or MongoDB, there is a high probability that CDM failed to obtain all columns), you can click
and select Add a new field to add new fields to ensure that the data imported to the migration destination is complete. - When a relational database, Hive, DLI, or MRS Hudi is used as the migration source, sample values cannot be obtained.
- When SQLServer is the destination, fields of the timestamp type cannot be written. You must change their type (for example, to datetime) so that they can be written.
- Column names are displayed when the source of the migration job is OBS, CSV files are to be migrated, and parameter Extract first row as columns is set to Yes.
- Field converters configuration is not involved when the binary format is used to migrate files to files.
- In the automatic table creation scenario, you need to manually add fields to the destination table in advance and then add fields to the field mapping.
- After a field is added, its sample value is not displayed on the console. This does not affect the field value transmission. CDM directly writes the field value to the destination end.
- If the field mapping is incorrect, you can adjust the field mapping by dragging fields or clicking
to map fields in batches. - An expression processes the data of a field. When you create an expression converter, do not use a time macro. If you need to use a time macro, use either of the following methods (if the source is of the file type, only Method 1 is supported):
- Method 1: When creating an expression converter, use two single quotation marks ('') to enclose the expression.
For example, if expression ${dateformat(yyyy-MM-dd)} is not enclosed in quotation marks, the hyphen (-) in the value 2017-10-16 parsed from the expression will be recognized as a minus sign, and further calculation will be performed to generate result 1991, which is incorrect. If you enclose the expression in quotation marks, that is, '${dateformat(yyyy-MM-dd)}', you will obtain '2017-10-16', which is correct.
+Figure 2 Using two single quotation marks ('') to enclose an expression+ +
- Method 2: Add a custom source field, enter a macro variable of date and time for Example Value, and map the field to a destination field again.Figure 3 Adding a custom source field
- Method 1: When creating an expression converter, use two single quotation marks ('') to enclose the expression.
- If the data is imported to GaussDB(DWS), you need to select the distribution columns in the destination fields. You are advised to select the distribution columns according to the following rules:
- Use the primary key as the distribution column.
- If multiple data segments are combined as primary keys, specify all primary keys as the distribution column.
- In the scenario where no primary key is available, if no distribution column is selected, DWS uses the first column as the distribution column by default. As a result, data skew risks exist.
-Remove line break
This converter is used to delete the newline characters, such as \n, \r, and \r\n from the field.
Expression Conversion
This converter uses the JSP expression language (EL) to convert the current field or a row of data. The JSP EL is used to create arithmetic and logical expressions. Within a JSP EL expression, you can use integers, floating point numbers, strings, the built-in constants true and false for boolean values, and null.
+@@ -111,3 +114,10 @@Expression Conversion
This converter uses the JSP expression language (EL) to convert the current field or a row of data. The JSP EL is used to create arithmetic and logical expressions. In an expression, you can use integers, floating point numbers, strings, constants true and false, and null.
+During data conversion, if the content to be replaced contains a special character, use a backslash (\) to escape the special character to a common one.
- The expression supports the following environment variables:
- value: indicates the current field value.
- row: indicates the current row, which is an array type.
- The expression supports the following Utils:
- If the field is of the string type, convert all character strings into lowercase letters, for example, convert aBC to abc.
Expression: StringUtils.lowerCase(value)
- Convert all character strings of the current field to uppercase letters.
- Convert the format of the first date field from 2018-01-05 15:15:05 to 20180105.
Expression: DateUtils.format(DateUtils.parseDate(row[0],"yyyy-MM-dd HH:mm:ss"),"yyyyMMdd")
- - Convert a timestamp to a date string in yyyy-MM-dd hh:mm:ss format, for example, convert 1701312046588 to 2023-11-30 10:40:46.
Expression: DateUtils.format(NumberUtils.toLong(value),"yyyy-MM-dd HH:mm:ss")
+ - Convert a timestamp to a date string in yyyy-MM-dd hh:mm:ss format, for example, convert 1701312046588 to 2023-11-30 10:40:46.
Expression: DateUtils.format(NumberUtils.toLong(value),"yyyy-MM-dd HH:mm:ss")
- Convert a date string in the yyyy-MM-dd hh:mm:ss format to a timestamp.
Expression: DateUtils.getTime(DateUtils.parseDate(value,"yyyy-MM-dd hh:mm:ss"))
- If the field value is a date string in yyyy-MM-dd format, extract the year from the field value, for example, extract 2017 from 2017-12-01.
Expression: StringUtils.substringBefore(value,"-")
- If the field value is of the numeric type, convert the value to a new value which is two times greater than the original value: @@ -69,12 +71,13 @@
- If the field is of the string type, remove the substring at the end of the field. If the specified substring is not at the end of the field, no conversion is performed. For example, remove .com at the end of www.domain.com.
Expression: StringUtils.removeEnd(value,".com")
- If the field is of the string type, delete the substring at the beginning of the field. If the specified substring is not at the beginning of the field, no conversion is performed. For example, delete www. at the beginning of www.domain.com.
Expression: StringUtils.removeStart(value,"www.")
- If the field is of the string type, replace all the specified character strings in the field. For example, replace a in aba with z to obtain zbz.
Expression: StringUtils.replace(value,"a","z")
+If the content to be replaced contains a special character, the special character must be escaped to a common character. For example, if you want to delete \t from a string, use the following expression: StringUtils.replace(value,"\\t",""), which means escaping the backslash (\) again.
- If the field is of the string type, replace multiple characters in the character string at a time. For example, replace h in hello with j and o with y to obtain jelly.
Expression: StringUtils.replaceChars(value,"ho","jy")
- If the string starts with the specified prefix (case sensitive), true is returned; otherwise, false is returned. For example, abcdef starts with abc, so that true is returned.
Expression: StringUtils.startsWith(value,"abc")
- If the field is of the string type, delete all the specified characters at the beginning and end of the field. the field. For example, delete all x, y, z, and b from abcyx to obtain abc.
Expression: StringUtils.strip(value,"xyzb")
- - If the field is of the string type, delete all the specified characters at the end of the field, for example, delete the abc string at the end of the field.
Expression: StringUtils.stripEnd(value,"abc")
+ - If the field is of the string type, delete all the specified characters at the end of the field, for example, delete the "abc" string at the end of the field.
- If the field is of the string type, delete all the specified characters at the beginning of the field, for example, delete all spaces at the beginning of the field. -
- If the field is of the string type, obtain the substring after the specified position (the index starts from 0, including the character at the specified position) of the character string. If the specified position is a negative number, calculate the position in the descending order. The first digit at the end is -1. For example, obtain the second character (c) of abcde and the string after it, that is, cde.
Expression: StringUtils.substring(value,2)
+ - If the field is of the string type, obtain the substring after the specified position (the index starts from 0, including the character at the specified position) of the character string. If the specified position is a negative number, calculate the position in the descending order. The first digit at the end is -1. For example, obtain the character whose index is 2 from abcde (that is, c) and the string after it, that is, cde.
Expression: StringUtils.substring(value,2)
- If the field is of the string type, obtain the substring in a specified range (the index starts from 0, including the character at the start and excluding the character at the end). If the range is a negative number, calculate the position in the descending order. The first digit at the end is -1. For example, obtain the string between the second character (c) and fourth character (e) of abcde, that is, cd.
Expression: StringUtils.substring(value,2,4)
- If the field is of the string type, obtain the substring after the first specified character. For example, obtain the substring after the first b in abcba, that is, cba.
Expression: StringUtils.substringAfter(value,"b")
- If the field is of the string type, obtain the substring after the last specified character. For example, obtain the substring after the last b in abcba, that is, a.
Expression: StringUtils.substringAfterLast(value,"b")
@@ -94,13 +97,13 @@ - Convert the character string to a value of the long type. If the conversion fails, the specified value, for example, 1L, is returned.
Expression: NumberUtils.toLong(value,1L)
- Convert the character string to a value of the short type. If the conversion fails, 0 is returned.
Expression: NumberUtils.toShort(value)
- Convert the character string to a value of the short type. If the conversion fails, the specified value, for example, 1, is returned.
Expression: NumberUtils.toShort(value,1)
- - Convert the IP string to a value of the long type, for example, convert 10.78.124.0 to 172915712.
Expression: CommonUtils.ipToLong(value)
+ - Convert the IP string to a value of the long type, for example, convert 10.78.124.0 to 172915712.
Expression: CommonUtils.ipToLong(value)
- Read an IP address and physical address mapping file from the network, and download the mapping file to the map collection. url indicates the address for storing the IP mapping file, for example, http://10.114.205.45:21203/sqoop/IpList.csv.
Expression: HttpsUtils.downloadMap("url")
- Cache the IP address and physical address mappings and specify a key for retrieval, for example, ipList.
Expression: CommonUtils.setCache("ipList",HttpsUtils.downloadMap("url"))
- Obtain the cached IP address and physical address mappings.
- Check whether the IP address and physical address mappings are cached.
- Based on the specified offset type (month/day/hour/minute/second) and offset (positive number indicates increase and negative number indicates decrease), convert the time in the specified format to a new time, for example, add 8 hours to 2019-05-21 12:00:00.
Expression: DateUtils.getCurrentTimeByZone("yyyy-MM-dd HH:mm:ss",value, "hour", 8)
- - If the value is empty or null, "aaa" is returned. Otherwise, value is returned.
Expression: StringUtils.defaultIfEmpty(value,"aaa")
+ - If the value is empty or null, "aaa" is returned. Otherwise, value is returned.
Expression: StringUtils.defaultIfEmpty(value,"aaa")
- If the field is of the string type, convert all character strings into lowercase letters, for example, convert aBC to abc.
- Method 1: When creating an expression converter, use two single quotation marks ('') to enclose the expression.
(Optional) Defining a Workspace Role
++For IAM users with DARTS User account permissions, DataArts Studio workspace roles determine their permissions in workspaces. Preset roles include admin, developer, deployer, operator, and viewer. For details about the permissions of each role, see "Permissions" in Service Overview.
+If the preset roles meet your needs, skip this section. Otherwise, create custom roles by following the instructions in this section.
++Context
- DARTS Administrator or Tenant Administrator can create custom roles in a workspace.
+Notes and Constraints
- Due to the constraints of the authentication cache mechanism, when the permissions of a custom role change, permissions of the members of the workspace that this role has been associated with will not be updated immediately. Instead, the updated permissions take effect six minutes after the workspace members with this role stop accessing the DataArts Studio console.
- A custom role with the same permissions as the workspace admin cannot perform some operations that can only be performed by the admin.
+Procedure
- Log in to the DataArts Studio console as user DARTS Administrator or Tenant Administrator. For details, see Accessing the DataArts Studio Instance Console.
- On the Roles page, click Create. In the displayed dialog box, set the following parameters:
- Role Name: unique identifier of a custom role. You are advised to name the role to fit its application scenario and avoid any meaningless description.
- Role Description: description of the role, for example, the key permission differences between the role and preset roles
- Role Type: Select a proper type based on where this role will be used. Role permissions differ in the simple and enterprise mode. For details about the enterprise mode, see DataArts Studio Enterprise Mode Overview.
- If you want to use this role in a workspace in simple mode, select DEV&PROD_CUSTOM.
- If you want to use this role in the development environment of a workspace in enterprise mode, select DEV_CUSTOM.
- If you want to use this role in the production environment of a workspace in enterprise mode, select PROD_CUSTOM.
- Reuse: Use this function if you only need to slightly adjust the permissions of the preset roles to meet your requirements. Otherwise, you can directly select permissions for the role.+
When you create or edit a role, an error message indicating insufficient permissions may be displayed even if you have the DARTS Administrator or Tenant Administrator permissions. This may be caused by network constraints. You can change another network and try again.
+
- After configuring the role name, type, and permissions, click OK.
- Assign the created custom role to the IAM user by following the instructions in Adding Workspace Members and Assigning Roles.
++ diff --git a/docs/dataartsstudio/umn/dataartsstudio_01_0108.html b/docs/dataartsstudio/umn/dataartsstudio_01_0108.html index 6bd10fdfb..6fc87c3b2 100644 --- a/docs/dataartsstudio/umn/dataartsstudio_01_0108.html +++ b/docs/dataartsstudio/umn/dataartsstudio_01_0108.html @@ -58,15 +58,15 @@++Parent topic: Authorizing Users to Use DataArts Studio+- Field Delimiter
Character used to separate columns in a CSV file. The value can be a single character, multiple characters, or special characters. For details, see Table 1.
- Encoding Type
Encoding type of a CSV file. The default value is UTF-8.
If this parameter is specified at the migration source, the specified encoding type is used to parse the file. If this parameter is specified at the migration destination, the specified encoding type is used to write data to the file.
-- Use Quote Character
- Exporting data from a database or NoSQL to CSV files (configuring Use Quote Character at the migration destination): If a field delimiter appears in the character string of a column of data at the migration source, set Use Quote Character to Yes at the migration destination to quote the character string as a whole and write it into the CSV file. Currently, CDM uses double quotation marks ("") as the quote character only. Figure 1 shows that the value of the name field in the database contains a comma (,). +
- Use Quote Character
- Exporting data from a database or NoSQL to CSV files (configuring Use Quote Character at the migration destination): If a field delimiter appears in the character string of a column of data at the migration source, set Use Quote Character to Yes at the migration destination to quote the character string as a whole and write it into the CSV file. Currently, CDM uses double quotation marks ("") as the quote character only. Figure 1 shows that the value of the name field in the database contains a comma (,).
If you do not use the quote character, the exported CSV file is displayed as follows:
-3.hello,world,abc
+3,hello,world,abc
If you use the quote character, the exported CSV file is displayed as follows:
3,"hello,world",abc
If the data in the database contains double quotation marks ("") and you set Use Quote Character to Yes, the quote character in the exported CSV file is displayed as three double quotation marks ("""). For example, if the value of a field is a"hello,world"c, the exported data is as follows:
"""a"hello,world"c"""
- - Exporting CSV files to a database or NoSQL (configuring Use Quote Character at the migration source): If you want to import the CSV files with quoted values to a database correctly, set Use Quote Character to Yes at the migration source to write the quoted values as a whole.
- Exporting data from a database or NoSQL to CSV files (configuring Use Quote Character at the migration destination): If a field delimiter appears in the character string of a column of data at the migration source, set Use Quote Character to Yes at the migration destination to quote the character string as a whole and write it into the CSV file. Currently, CDM uses double quotation marks ("") as the quote character only. Figure 1 shows that the value of the name field in the database contains a comma (,).
- Use RE to Separate Fields
This function is used to parse complex semi-structured text, such as log files. For details, see Using Regular Expressions to Separate Semi-structured Text.
+ - Exporting CSV files to a database or NoSQL (configuring Use Quote Character at the migration source): If you want to import the CSV files with quoted values to a database correctly, set Use Quote Character to Yes at the migration source to write the quoted values as a whole.
- Use RE to Separate Fields
This function is used to parse complex semi-structured text, such as log files. For details, see Using Regular Expressions to Separate Semi-structured Text.
- Use First Row as Header
This parameter is used when CSV files are exported to other locations. If this parameter is specified at the migration source, CDM uses the first row as the header when extracting data. When the CSV files are transferred, the headers are skipped. The number of rows extracted from the migration source is more than the number of rows written to the migration destination. The log files will output the information that the header is skipped during the migration.
- File Size
@@ -107,7 +107,7 @@ }]This parameter is used when data is exported from the database to a CSV file. If a table contains a large amount of data, a large CSV file is generated after migration, which is inconvenient to download or view. In this case, you can specify this parameter at the migration destination so that multiple CSV files with the specified size can be generated. The value of this parameter is an integer. The unit is MB.
- JSON Reference Node
Root node that records data. The data corresponding to the node is a JSON array. CDM extracts data from the array in the same mode. Use periods (.) to separate multi-layer nested JSON nodes.
-- Copying Data from a JSON File
- Example 1: Extract data from multiple objects that are separated or merged. A JSON file contains multiple JSON objects. The following gives an example:
{ + - Copying Data from a JSON File
- Example 1Extract data from multiple objects that are separated or merged. A JSON file contains multiple JSON objects. The following gives an example:
{ "took": 190, "timed_out": false, "total": 1000001, @@ -125,49 +125,50 @@ "total": 1000003, "max_score": 1.0 }+To extract data from the JSON object and write data to the database in the following formats, set File Format to JSON and JSON Type to JSON object, and then map fields. -took
+Table 2 Example
- Example 1
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- On the Cluster Management page, locate a cluster and click Job Management in the Operation column. On the displayed page, click the Links tab and then Create Link.
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- Click Job Management in the Operation column of the CDM cluster. On the displayed page, click the Links tab and then Create Link. The Select Connector page is displayed.
- Manage jobs by group.
- Modifying the job parameters
- Viewing job status
- Prerequisites
- Configure the number of concurrent extractors based on the following rules:
- Log in to the cloud console. Choose DataArts Studio from the service list in the upper left corner.
- If there are multiple DataArts Studio instances in the current region, the instance list is displayed by default. Locate an instance and click Access to access the DataArts Studio console.
+
