# Adding CloudTable Dump Tasks

#### Function

This API is used to add CloudTable dump tasks.

#### URI

POST /v2/{project_id}/streams/{stream_name}/transfer-tasks

Table 1 Path parameters
Parameter | Mandatory | Type | Description
--- | --- | --- | ---
project_id | Yes | String | Project ID.
stream_name | Yes | String | Name of the stream. Maximum: 60

#### Request Parameters

Table 2 Request header parameters
Parameter | Mandatory | Type | Description
--- | --- | --- | ---
X-Auth-Token | Yes | String | User token. The token can be obtained by calling the IAM API (value of X-Subject-Token in the response header).

Table 3 Request body parameters
Parameter | Mandatory | Type | Description
--- | --- | --- | ---

destination_type

Yes

String

Dump destination. Possible values:

- OBS: Data is dumped to OBS.
- MRS: Data is dumped to MRS.
- DLI: Data is dumped to DLI.
- CLOUDTABLE: Data is dumped to CloudTable.
- DWS: Data is dumped to DWS.

Default: NOWHERE

Enumeration values:

- CLOUDTABLE

cloudtable_destination_descriptor | No | CloudtableDestinationDescriptorRequest object | Parameter list of the CloudTable to which data in the DIS stream will be dumped.

Table 4 CloudtableDestinationDescriptorRequest
Parameter | Mandatory | Type | Description
--- | --- | --- | ---
task_name | Yes | String | Name of the dump task. The task name consists of letters, digits, hyphens (-), and underscores (_). It must be a string of 1 to 64 characters.

agency_name

Yes

String

Name of the agency created on IAM. DIS uses an agency to access your specified resources. The parameters for creating an agency are as follows:

- Agency Type: Cloud service
- Cloud Service: DIS
- Validity Period: unlimited
- Scope: Global service, Project: OBS. Select the Tenant Administrator role for the global service project.

If agencies have been created, you can obtain available agencies from the agency list by using the "Listing Agencies " API.

This parameter cannot be left blank and the parameter value cannot exceed 64 characters.

If there are dump tasks on the console, the system displays a message indicating that an agency will be automatically created. The name of the automatically created agency is dis_admin_agency.

Maximum: 64

deliver_time_interval | Yes | Integer | User-defined interval at which data is imported from the current DIS stream into OBS. If no data is pushed to the DIS stream during the current interval, no dump file package will be generated. Value range: 30-900 Default value: 300 Unit: second Minimum: 30 Maximum: 900 Default: 300

consumer_strategy

No

String

Offset.

- LATEST: Maximum offset, indicating that the latest data will be extracted.
- TRIM_HORIZON: Minimum offset, indicating that the earliest data will be extracted.

Default value: LATEST

Default: LATEST

Enumeration values:

- LATEST
- TRIM_HORIZON

cloudtable_cluster_name | Yes | String | Name of the CloudTable cluster to which data will be dumped. If you choose to dump data to OpenTSDB, OpenTSDB must be enabled for the cluster.
cloudtable_cluster_id | Yes | String | ID of the CloudTable cluster to which data will be dumped. If you choose to dump data to OpenTSDB, OpenTSDB must be enabled for the cluster.
cloudtable_table_name | No | String | HBase table name of the CloudTable cluster to which data will be dumped. The parameter is mandatory when data is dumped to the CloudTable HBase.
cloudtable_schema | No | CloudtableSchema object | Schema configuration of the CloudTable HBase data. You can set either this parameter or opentsdb_schema, but this parameter is mandatory when data will be dumped to HBase. After this parameter is set, the JSON data in the stream can be converted to another format and then be imported to the CloudTable HBase.
opentsdb_schema | No | Array of OpenTSDBSchema objects | Schema configuration of the CloudTable OpenTSDB data. You can set either this parameter or opentsdb_schema, but this parameter is mandatory when data will be dumped to OpenTSDB. After this parameter is set, the JSON data in the stream can be converted to another format and then be imported to the CloudTable OpenTSDB.
cloudtable_row_key_delimiter | No | String | Delimiter used to separate the user data that generates HBase row keys. Value range: , . | ; \ - _ and ~ Default value: .
obs_backup_bucket_path | No | String | Name of the OBS bucket used to back up data that failed to be dumped to CloudTable.
backup_file_prefix | No | String | Self-defined directory created in the OBS bucket and used to back up data that failed to be dumped to CloudTable. Directory levels are separated by slashes (/) and cannot start with slashes. Value range: a string of letters, digits, and underscores (_) The maximum length is 50 characters. This parameter is left empty by default.
retry_duration | No | String | Time duration for DIS to retry if data fails to be dumped to CloudTable. If this threshold is exceeded, the data that fails to be dumped will be backed up to the OBS bucket/backup_file_prefix /cloudtable_error or OBS bucket/backup_file_prefix/opentsdb_error directory. Value range: 0-7,200 Unit: second Default value: 1,800

Table 5 CloudtableSchema
Parameter | Mandatory | Type | Description
--- | --- | --- | ---
row_key | Yes | Array of RowKey objects | HBase rowkey schema used by the CloudTable cluster to convert JSON data into HBase rowkeys. Value range: 1-64
columns | Yes | Array of Column objects | HBase column schema used by the CloudTable cluster to convert JSON data into HBase columns. Value range: 1 to 4,096

Table 6 RowKey
Parameter | Mandatory | Type | Description
--- | --- | --- | ---
value | Yes | String | JSON attribute name, which is used to generate HBase rowkeys for JSON data in the DIS stream.

type

Yes

String

JSON attribute type of JSON data in the DIS stream. Value range:

- Bigint
- Double
- Boolean
- Timestamp
- String
- Decimal

Enumeration values:

- Bigint
- Double
- Boolean
- Timestamp
- String
- Decimal

Table 7 Column
Parameter | Mandatory | Type | Description
--- | --- | --- | ---
column_family_name | Yes | String | Name of the HBase column family to which data will be dumped.
column_name | Yes | String | Name of the HBase column to which data will be dumped. Value range: a string of 1 to 32 characters, consisting of only letters, digits, and underscores (_)
value | Yes | String | JSON attribute name, which is used to generate HBase column values for JSON data in the DIS stream.

type

Yes

String

JSON attribute type of JSON data in the DIS stream.

Value range:

- Bigint
- Double
- Boolean
- Timestamp
- String
- Decimal

Enumeration values:

- Bigint
- Double
- Boolean
- Timestamp
- String
- Decimal

Table 8 OpenTSDBSchema
Parameter | Mandatory | Type | Description
--- | --- | --- | ---
metric | Yes | Array of OpenTSDBMetric objects | Schema configuration of the OpenTSDB data metric in the CloudTable cluster. After this parameter is set, the JSON data in the stream can be converted to the metric of the OpenTSDB data.
timestamp | Yes | OpenTSDBTimestamp object | Schema configuration of the OpenTSDB data timestamp in the CloudTable cluster. After this parameter is set, the JSON data in the stream can be converted to the timestamp of the OpenTSDB data.
value | Yes | OpenTSDBValue object | Schema configuration of the OpenTSDB data value in the CloudTable cluster. After this parameter is set, the JSON data in the stream can be converted to the value of the OpenTSDB data.
tags | Yes | Array of OpenTSDBTags objects | Schema configuration of the OpenTSDB data tags in the CloudTable cluster. After this parameter is set, the JSON data in the stream can be converted to the tags of the OpenTSDB data.

Table 9 OpenTSDBMetric
Parameter | Mandatory | Type | Description
--- | --- | --- | ---

type

Yes

String

- When type is set to Constant, the value of metric is the value of Value.
- When value is set to String, the value of metric is the value of the JSON attribute of the user data in the stream.

Enumeration values:

- Constant
- String

value | Yes | String | Constant value or JSON attribute name of the user data in the stream. This value is 1 to 32 characters long. Only letters, digits, and periods (.) are allowed.

Table 10 OpenTSDBTimestamp
Parameter | Mandatory | Type | Description
--- | --- | --- | ---

type

Yes

String

- When type is set to Timestamp, the value type of the JSON attribute of the user data in the stream is Timestamp, and the timestamp of OpenTSDB can be generated without converting the data format.
- When type is set to String, the value type of the JSON attribute of the user data in the stream is Date, and the timestamp of OpenTSDB can be generated only after the data format is converted.

value | Yes | String | JSON attribute name of the user data in the stream. Value range: a string of 1 to 32 characters, consisting of only letters, digits, and underscores (_)

format

Yes

String

This parameter is mandatory when type is set to String. When the value type of the JSON attribute of the user data in the stream is Date, format is required to convert the data format to generate the timestamp of OpenTSDB.

Value range:

- yyyy/MM/dd HH:mm:ss
- MM/dd/yyyy HH:mm:ss
- dd/MM/yyyy HH:mm:ss
- yyyy-MM-dd HH:mm:ss
- MM-dd-yyyy HH:mm:ss
- dd-MM-yyyy HH:mm:ss

Enumeration values:

- yyyy/MM/dd HH:mm:ss
- MM/dd/yyyy HH:mm:ss
- dd/MM/yyyy HH:mm:ss
- yyyy-MM-dd HH:mm:ss
- MM-dd-yyyy HH:mm:ss
- dd-MM-yyyy HH:mm:ss

Table 11 OpenTSDBValue
Parameter | Mandatory | Type | Description
--- | --- | --- | ---

type

Yes

String

Dump destination. Possible values: Value range:

- Bigint
- Double
- Boolean
- Timestamp
- String
- Decimal

value | Yes | String | Constant value or JSON attribute name of the user data in the stream. Value range: a string of 1 to 32 characters, consisting of only letters, digits, and underscores (_)

Table 12 OpenTSDBTags
Parameter | Mandatory | Type | Description
--- | --- | --- | ---
name | Yes | String | Tag name of the OpenTSDB data that stores the data in the stream. Value range: a string of 1 to 32 characters, consisting of only letters, digits, and underscores (_)

type

Yes

String

Type name of the JSON attribute of the user data in the stream.

Value range:

- Bigint
- Double
- Boolean
- Timestamp
- String
- Decimal

value | Yes | String | Constant value or JSON attribute name of the user data in the stream. Value range: a string of 1 to 32 characters, consisting of only letters, digits, and underscores (_)

#### Response Parameters

None

#### Example Requests

- Adding CloudTable HBase Dump Tasks

      POST https://{Endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks

      {
        "destination_type" : "CLOUDTABLE",
        "cloudtable_destination_descriptor" : {
          "task_name" : "hbasetask",
          "consumer_strategy" : "TRIM_HORIZON",
          "agency_name" : "dis_admin_agency",
          "cloudtable_cluster_name" : "cloudtablecluster",
          "cloudtable_cluster_id" : "b8c095e2-db5f-4732-8a1d-eacd662e35dc",
          "cloudtable_table_name" : "cloudtabletable",
          "cloudtable_row_key_delimiter" : "|",
          "retry_duration" : 1800,
          "obs_backup_bucket_path" : "obsbackupbucket",
          "backup_file_prefix" : "",
          "cloudtable_schema" : {
            "row_key" : [ {
              "value" : "datavalue",
              "type" : "String"
            } ],
            "columns" : [ {
              "column_family_name" : "cfname1",
              "column_name" : "ID",
              "value" : "datavalue1",
              "type" : "String"
            }, {
              "column_family_name" : "cfname2",
              "column_name" : "VALUE",
              "value" : "datavalue2",
              "type" : "String"
            } ]
          }
        }
      }

- Adding CloudTable OpenTSDB Dump Tasks

      POST https://{Endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks

      {
        "destination_type" : "CLOUDTABLE",
        "cloudtable_destination_descriptor" : {
          "task_name" : "opentsdbtask",
          "consumer_strategy" : "LATEST",
          "agency_name" : "dis_admin_agency",
          "cloudtable_cluster_name" : "cloudtablecluster",
          "cloudtable_cluster_id" : "b8c095e2-db5f-4732-8a1d-eacd662e35dc",
          "retry_duration" : 1800,
          "obs_backup_bucket_path" : "obsbackupbucket",
          "backup_file_prefix" : "",
          "opentsdb_schema" : [ {
            "metric" : [ {
              "type" : "Constant",
              "value" : "age"
            } ],
            "timestamp" : {
              "value" : "date",
              "type" : "String",
              "format" : "yyyy/MM/dd HH:mm:ss"
            },
            "value" : {
              "value" : "value",
              "type" : "Bigint"
            },
            "tags" : [ {
              "name" : "name",
              "value" : "name",
              "type" : "Bigint"
            } ]
          } ]
        }
      }

#### Example Responses

None

#### Status Codes

Status Code | Description
--- | ---
201 | Normal response.

#### Error Codes

See Error Codes.

Parent topic: Dump Task Management