yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
f9e2808b7ccbc9deda30932a9f5a6a56623025f5
doc-exports/docs/dataartsstudio/umn/dataartsstudio_01_0014.html
chenxiaoxiong f9e2808b7c DataArts UMN 20250810 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
2025-09-02 10:44:13 +00:00

69 KiB
Raw Blame History

Supported Data Sources (2.9.3.300)

CDM provides the following migration modes which support different data sources:

This section describes the data sources supported by CDM clusters of version 2.9.3.300. The supported data sources vary depending on the CDM cluster version.

Data Sources Supported by Table/File Migration

Table/File migration can migrate data in tables or files.

Table 1 describes the supported data sources.
Table 1 Supported data sources during table/file migration

Category

Source

Destination

Description

Data warehouse

Data Warehouse Service
  • Data warehouse: GaussDB(DWS), Data Lake Insight (DLI), and MRS ClickHouse
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

The DWS physical machine management mode is not supported.

Data Lake Insight (DLI)
  • Data warehouse: GaussDB(DWS), Data Lake Insight (DLI), and MRS ClickHouse
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
  • NoSQL: CloudTable and MongoDB
  • Search: Elasticsearch and Cloud Search Service (CSS)

Recommended MongoDB version: 4.2

You must have the SELECT permission (for querying tables) on all fields of the DLI data source.

MRS ClickHouse

Data warehouse: MRS ClickHouse and Data Lake Insight (DLI)
  • Recommended MRS ClickHouse version: 21.3.4.X

  • MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.

Hadoop

MRS HDFS
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Supported by local storage. Only MRS Hive and MRS Hudi are supported in storage-compute decoupling scenarios.
  • Only MRS Hive is supported in Ranger scenarios.
  • Not supported if SSL is enabled for ZooKeeper
  • Recommended MRS HDFS versions:
    • 2.8.X
    • 3.1.X
  • Recommended MRS HBase versions:
    • 2.1.X
    • 1.3.X
  • MRS Hive and MRS Hudi 2.x versions are not supported. The following versions are recommended:
    • 1.2.X
    • 3.1.X

  • MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.

MRS HBase

MRS Hive
  • Data warehouse: GaussDB(DWS), Data Lake Insight (DLI), and MRS ClickHouse
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

MRS Hudi
  • Data warehouse: GaussDB(DWS)
  • Hadoop: MRS HBase

FusionInsight HDFS
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • FusionInsight cannot serve as the destination.
  • Supported only by local storage and not in storage-compute decoupling scenarios
  • Not supported by Ranger
  • Not supported if SSL is enabled for ZooKeeper
  • Recommended FusionInsight HDFS versions:
    • 2.8.X
    • 3.1.X
  • Recommended FusionInsight HBase versions:
    • 2.1.X
    • 1.3.X
  • Recommended FusionInsight Hive versions:
    • 1.2.X
    • 3.1.X

FusionInsight HBase

FusionInsight Hive

Apache HBase
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Apache cannot serve as the destination.
  • Supported only by local storage and not in storage-compute decoupling scenarios
  • Not supported by Ranger
  • Not supported if SSL is enabled for ZooKeeper
  • Recommended Apache HBase versions:
    • 2.1.X
    • 1.3.X
  • Apache Hive 2.x versions are not supported. The following versions are recommended:
    • 1.2.X
    • 3.1.X
  • Recommended Apache HDFS versions:
    • 2.8.X
    • 3.1.X

Apache Hive

Apache HDFS

Object storage

Object Storage Service (OBS)
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Object Storage Migration Service (OMS) is recommended for migration between object storage services.
  • Binary files cannot be imported to a database or NoSQL.

File system

FTP
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Object-based storage: Object Storage Service (OBS)
  • The file system cannot serve as the destination.
  • Only text files such as CSV files can be migrated from FTP or SFTP servers to search services. Binary files cannot.
  • Only binary files can be migrated from FTP or SFTP servers to OBS.
  • obsutil is recommended for migrating data from HTTP servers to OBS.

SFTP

HTTP

Hadoop: MRS HDFS

Relational database

RDS for MySQL
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, MRS Hive, and MRS Hudi
  • Object-based storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Recommended Microsoft SQL Server version: 2005 or later

RDS for SQL Server
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
  • Search: Elasticsearch and Cloud Search Service (CSS)

RDS for PostgreSQL

MySQL
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, MRS Hive, and MRS Hudi
  • Object-based storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

PostgreSQL

Oracle

Microsoft SQL Server
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

SAP HANA
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS Hive
SAP HANA data sources have the following restrictions:
  • SAP HANA cannot serve as the destination.
  • Only the 2.00.050.00.1592305219 version is supported.
  • Only the Generic Edition is supported.
  • BW/4 FOR HANA is not supported.
  • Only database names, table names, and column names consisting of English letters are supported. Special characters such as spaces and symbols are not allowed.
  • The following data types are supported: date, digit, Boolean, and character (except SHORTTEXT). Other data types such as binary are not supported.
  • During migration, tables cannot be automatically created at the destination.

Database Sharding
  • Data warehouse: Data Lake Insight (DLI)
  • Hadoop: MRS HBase and MRS Hive
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Object-based storage: Object Storage Service (OBS)

Database shards cannot serve as the destination.

ShenTong
  • Hadoop: MRS Hive and MRS Hudi

-

NoSQL

Distributed Cache Service (DCS)

Hadoop: MRS HDFS, MRS HBase, and MRS Hive

NoSQL except CloudTable cannot serve as the destination.

Redis

Document Database Service

MongoDB

CloudTable HBase
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

Cassandra
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

Message system

Apache Kafka

Search: Cloud Search Service (CSS)

The message system cannot serve as the destination.

DMS Kafka

MRS Kafka
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • MRS Kafka cannot serve as the destination.
  • Supported only by local storage and not in storage-compute decoupling scenarios
  • Not supported by Ranger
  • Not supported if SSL is enabled for ZooKeeper

  • MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.

Search

Elasticsearch
  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

Only the non-security mode is supported.

Cloud Search Service (CSS)

N/A

In the preceding table, the non-cloud data sources, such as MySQL, include on-premises MySQL, MySQL built on ECSs, or MySQL on the third-party cloud.

Supported Data Sources in Entire DB Migration

Entire DB migration is used when an on-premises data center or a database created on an ECS needs to be synchronized to a database service or big data service on the cloud. It is suitable for offline database migration but not online real-time migration.

Table 2 lists the data sources supporting entire DB migration using CDM.

Table 2 Supported data sources in entire DB migration

Category

Data Source

Read

Write

Description

Data warehouse

Data Warehouse Service

Supported

Supported

-

Hadoop

(available only for local storage, and not for storage-compute decoupling, Ranger, or ZooKeeper for which SSL is enabled)

MRS HBase

Supported

Supported

Entire DB migration only to MRS HBase

Recommended versions:
  • 2.1.X
  • 1.3.X

MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.

MRS Hive

Supported

Supported

Entire DB migration only to a relational database

2.x versions are not supported. The following versions are recommended:
  • 1.2.X
  • 3.1.X

MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.

FusionInsight HBase

Supported

Not supported
Recommended versions:
  • 2.1.X
  • 1.3.X

FusionInsight Hive

Supported

Not supported

Entire DB migration only to a relational database

2.x versions are not supported. The following versions are recommended:
  • 1.2.X
  • 3.1.X

Apache HBase

Supported

Not supported
Recommended versions:
  • 2.1.X
  • 1.3.X

Apache Hive

Supported

Not supported

Entire DB migration only to a relational database

2.x versions are not supported. The following versions are recommended:
  • 1.2.X
  • 3.1.X

MRS Hudi

Supported

Supported

Supported only by local storage and in storage-compute decoupling scenarios

2.x versions are not supported. The following versions are recommended:

  • 1.2.X
  • 3.1.X

MRS clusters whose Kerberos encryption type is aes256-sha2,aes128-sha2 are not supported, and only MRS clusters whose Kerberos encryption type is aes256-sha1,aes128-sha1 are supported.

Relational database

RDS for MySQL

Supported

Supported

Migration from OLTP to OLTP is not supported. In this scenario, you are advised to use the Data Replication Service (DRS).

RDS for PostgreSQL

Supported

Supported

RDS for SQL Server

Supported

Supported

MySQL

Supported

Not supported

PostgreSQL

Supported

Not supported

Microsoft SQL Server

Supported

Not supported

Oracle

Supported

Not supported

SAP HANA

Supported

Not supported
  • Only the 2.00.050.00.1592305219 version is supported.
  • Only the Generic Edition is supported.
  • BW/4 FOR HANA is not supported.
  • Only database names, table names, and column names consisting of English letters are supported. Special characters such as spaces and symbols are not allowed.
  • The following data types are supported: date, digit, Boolean, and character (except SHORTTEXT). Other data types such as binary are not supported.
  • During migration, tables cannot be automatically created at the destination.

Dameng database

Supported

Not supported

Only to DWS and Hive

NoSQL

Distributed Cache Service (DCS)

Not supported

Supported

Only migration from MRS to DCS is supported.

Document Database Service (DDS)

Supported

Supported

Only migration between DDS and MRS is supported.

CloudTable

Supported

Supported

-
Parent topic: Supported Data Sources