yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
main
doc-exports/docs/dataartsstudio/umn/dataartsstudio_03_0124.html
chenxiaoxiong f9e2808b7c DataArts UMN 20250810 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
2025-09-02 10:44:13 +00:00

8.9 KiB
Raw Permalink Blame History

What Is the Number of Concurrent Jobs for Different CDM Cluster Versions?

CDM migrates data through data migration jobs. It works in the following way:
  1. When data migration jobs are submitted, CDM splits each job into multiple tasks based on the Concurrent Extractors parameter in the job configuration.

    Jobs for different data sources may be split based on different dimensions. Some jobs may not be split based on the Concurrent Extractors parameter.

  2. CDM submits the tasks to the running pool in sequence. Tasks (defined by Maximum Concurrent Extractors) run concurrently. Excess tasks are queued.

Changing Concurrent Extractors

  1. The maximum number of concurrent extractors for a cluster varies depending on the CDM cluster flavor. You are advised to set the maximum number of concurrent extractors to twice the number of vCPUs of the CDM cluster.
    Table 1 Maximum number of concurrent extractors for a CDM cluster

    Flavor

    vCPUs/Memory

    Maximum Concurrent Extractors

    cdm.large

    8 vCPUs, 16 GB

    16

    cdm.xlarge

    16 vCPUs, 32 GB

    32

    cdm.4xlarge

    64 vCPUs, 128 GB

    128
    Figure 1 Setting Maximum Concurrent Extractors for a CDM cluster
  2. Configure the number of concurrent extractors based on the following rules:
    1. When data is to be migrated to files, CDM does not support multiple concurrent tasks. In this case, set a single process to extract data.
    2. If each row of the table contains less than or equal to 1 MB data, data can be extracted concurrently. If each row contains more than 1 MB data, it is recommended that data be extracted in a single thread.
    3. Set Concurrent Extractors for a job based on Maximum Concurrent Extractors for the cluster. It is recommended that Concurrent Extractors is less than Maximum Concurrent Extractors.
    4. If the destination is DLI, you are advised to set the number of concurrent extractors to 1. Otherwise, data may fail to be written.
    Figure 2 Setting Concurrent Extractors for a job
<script language="JavaScript"> </script>