forked from docs/data-ingestion-service
Update content
This commit is contained in:
parent
0aad323403
commit
6c2b63509d
Binary file not shown.
|
Before Width: | Height: | Size: 75 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 6.2 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 339 B |
Binary file not shown.
|
Before Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 949 B |
Binary file not shown.
|
Before Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 43 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 84 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 949 B |
Binary file not shown.
|
Before Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 949 B |
Binary file not shown.
|
Before Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 101 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 36 KiB |
@ -1,52 +0,0 @@
|
||||
:original_name: dis_01_0223.html
|
||||
|
||||
.. _dis_01_0223:
|
||||
|
||||
Billing Model
|
||||
=============
|
||||
|
||||
Data Ingestion Service (DIS) is charged for resource usage, there are three main billing factors: the duration of Partition and PUT Payload Unit number.
|
||||
|
||||
- General partition contains duration, PUT Payload Unit number, Data storage size billing factor. Advanced partition contains duration billing factor.
|
||||
- The duration of Partition: Partition is the base capacity unit of DIS. There are two types of partition: General partition and advanced partition. The general partition provides a capacity of 1MB/sec data input , 2MB/sec data outputand support up to 1000 records put per second. The advanced partition provides a capacity of 5MB/sec data input ,10MB/sec data output and support up to 2000 records put per second .You specify the number of partitions when create the stream based on the throughput requirements. You are charged for each partition at an hourly rate.
|
||||
- PUT Payload Unit number: A record is the data that your data producer adds to DIS stream. A PUT Payload Unit is counted in 25KB payload "chunks" that comprise a record. For example, a 5KB record contains one PUT Payload Unit, a 45KB record contains two PUT Payload Units, and a 1000KB record contains 40 PUT Payload Units. PUT Payload Unit is charged with a per million PUT Payload Units rate.
|
||||
- Data storage size: the data storage size in DIS input from the partition. The Customer can specify stream data storage time: from One day to Seven days when create the stream. If the data storage time exceeds the specified time, the data will be automatically aged. In general, the data storage one day is free, it means that 84GB of storage space for each partition is free to use. Of course, the final free space size can be specified according to the actual situation.
|
||||
|
||||
DIS is charged for resource usage as follow:
|
||||
|
||||
- One Partition Costs = partition time \* partition rate + Payload Unit Num \* Payload Unit Rate + Data storage size \* storage size rate
|
||||
- Total Costs per stream = Partition 1 Cost + Partition 2 Cost + … + Partition n Cost
|
||||
|
||||
Both resources will be charged at the same time. Each charged will be listed separately. There will be 3 CDR records for each stream in the billing interval.
|
||||
|
||||
The data will be cached for some time in the service, and default cache time is 24 hours. If the user want dump the data to OBS, the operation in OBS will be charged separately.
|
||||
|
||||
Example:
|
||||
|
||||
Let us assume that our data producers put 50 records per second in aggregate, and each record is 35KB. In this case, the total data input rate is 1.7MB/sec (50 records/sec*35KB/record). For simplicity, we assume that the throughput and data size of each record are stable and constant throughout the day.
|
||||
|
||||
We first calculate the number of partitions needed for our stream to achieve the required throughput. As one partition provides a capacity of 1MB/sec data input and supports 1000 records/sec, two partitions provide a capacity of 2MB/sec data input and support 2000 records/sec. So a stream with two partitions satisfies our required throughput of 1.7MB/sec at 50 records/sec.
|
||||
|
||||
We then calculate our costs using DIS pricing:
|
||||
|
||||
- Partition Hour: One Partition costs $X per hour. Our stream has two partitions so that it costs $2X per hour.
|
||||
- PUT Payload Unit (25KB): The 1 million Payload Unit costs $Y. As our record is 35KB, each record contains two PUT Payload Units. Our data producers put 50 records or 100 PUT Payload Units per second in aggregate. That is 180000 records or 360000 PUT Payload Units per hour.so it costs 0.36*Y per hour.
|
||||
|
||||
If the data in the DIS stored 1 day, the Data storage size is always less than 84GB, so the total cost per hour is :$(2X + Y)
|
||||
|
||||
Optionally, we can choose to increase the data retention period of our stream from 24 hours to up to 7 days. In this example, if the throughput is stable, the data storage size is 514.08GB per partition(0.85MB/sec \* 3600sec \* 24hour \* 7day).We assume that the data storage rate is $Z per GB*Hour, the cost is $860.16*Z( the detail is (514.08-84)GB \* 2 partitions \* Z). The total cost should $(2*X + Y + 860.16*Z).
|
||||
|
||||
**Shard Hour**
|
||||
|
||||
Shard is the base throughput unit of an Amazon Kinesis stream. One shard provides a capacity of 1MB/sec data input and 2MB/sec data output. One shard can support up to 1000 records per second. You specify the number of shards needed within your stream based on your throughput requirements. You are charged for each shard at an hourly rate.
|
||||
|
||||
**PUT Payload Unit (25KB)**
|
||||
|
||||
A record is the data that your data producer adds to your Amazon Kinesis stream. A PUT Payload Unit is counted in 25KB payload "chunks" that comprise a record. For example, a 5KB record contains one PUT Payload Unit, a 45KB record contains two PUT Payload Units, and a 1MB record contains 40 PUT Payload Units. PUT Payload Unit is charged with a per million PUT Payload Units rate.
|
||||
|
||||
**Extended Data Retention (Up to 7 days)**
|
||||
|
||||
Amazon Kinesis stores your data for 24 hours by default. You can choose to increase the data retention period of your stream to up to 7 days. You are charged for an additional rate on each shard hour incurred by your stream once you enable extended data retention.
|
||||
|
||||
- Getting records from Amazon Kinesis stream is free.
|
||||
- Data transfer is free. AWS does not charge for data transfer from your data producers to Amazon Kinesis Streams, or from Amazon Kinesis Streams to your Amazon Kinesis Applications.
|
||||
@ -1,16 +0,0 @@
|
||||
:original_name: dis_01_0226.html
|
||||
|
||||
.. _dis_01_0226:
|
||||
|
||||
CDR Folder
|
||||
==========
|
||||
|
||||
+-------------------------------------------+------------------------------------------+
|
||||
| Export Directory | Backup Directory |
|
||||
+===========================================+==========================================+
|
||||
| - /opt/meterfiles/uploads/dispartition | - /opt/meterfiles/backup/dispartition |
|
||||
+-------------------------------------------+------------------------------------------+
|
||||
| - /opt/meterfiles/uploads/dispayloadunit | - /opt/meterfiles/backup/dispayloadunit |
|
||||
+-------------------------------------------+------------------------------------------+
|
||||
| - /opt/meterfiles/uploads/disdatasize | - /opt/meterfiles/backup/disdatasize |
|
||||
+-------------------------------------------+------------------------------------------+
|
||||
@ -1,18 +0,0 @@
|
||||
:original_name: dis_01_0224.html
|
||||
|
||||
.. _dis_01_0224:
|
||||
|
||||
CDR Folder and File Name
|
||||
========================
|
||||
|
||||
- :ref:`Overview <dis_01_0225>`
|
||||
- :ref:`CDR Folder <dis_01_0226>`
|
||||
- :ref:`Service CDR File Name <dis_01_0227>`
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
overview
|
||||
cdr_folder
|
||||
service_cdr_file_name
|
||||
@ -1,10 +0,0 @@
|
||||
:original_name: dis_01_0225.html
|
||||
|
||||
.. _dis_01_0225:
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
Similar to other cloud services , CDR files for DIS will be generated, containing DIS usage information, OTC billing system will fetch these files from certain directory on Common CDR SFTP via public key to complete the charging of DIS.
|
||||
|
||||
The file compression agonization, MD5 verification, error file handling are totally same as the CDR files of other cloud services.for details, please refer to document of"OTC CDR File Specifications".
|
||||
@ -1,22 +0,0 @@
|
||||
:original_name: dis_01_0227.html
|
||||
|
||||
.. _dis_01_0227:
|
||||
|
||||
Service CDR File Name
|
||||
=====================
|
||||
|
||||
OTC_{region_code}_dispart_YYYYMMDDHHMMSS_YYYYMMDDHHMMSS_{az_code}_{serial_number}.csv/zip
|
||||
|
||||
OTC_{region_code}_dispayunit_YYYYMMDDHHMMSS_YYYYMMDDHHMMSS_{az_code}_{serial_number}.csv/zip
|
||||
|
||||
OTC_{region_code}_dissize_YYYYMMDDHHMMSS_YYYYMMDDHHMMSS_{az_code}_{serial_number}.csv/zip
|
||||
|
||||
The {az_code} is not used by dis, the value is set the AZ01 number by default.
|
||||
|
||||
**For example:**
|
||||
|
||||
OTC_eu-de_dispart_20170427100000_20170427105959_eu-de-01_0001.csv/zip
|
||||
|
||||
OTC_eu-de_dispayunit_20170427100000_20170427105959_eu-de-01_0001.csv/zip
|
||||
|
||||
OTC_eu-de_dissize\ **\_20170427100000_20170427105959\_ eu-de-01 \_0001.csv/zip**
|
||||
@ -1,20 +0,0 @@
|
||||
:original_name: dis_01_0228.html
|
||||
|
||||
.. _dis_01_0228:
|
||||
|
||||
CDR Specification
|
||||
=================
|
||||
|
||||
- :ref:`The CDR Specification <dis_01_0229>`
|
||||
- :ref:`The Partition CDR Content <dis_01_0230>`
|
||||
- :ref:`The PUT Payload Unit CDR Content <dis_01_0231>`
|
||||
- :ref:`The Data Store Size CDR Content <dis_01_0232>`
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
the_cdr_specification
|
||||
the_partition_cdr_content
|
||||
the_put_payload_unit_cdr_content
|
||||
the_data_store_size_cdr_content
|
||||
@ -1,30 +0,0 @@
|
||||
:original_name: dis_01_0229.html
|
||||
|
||||
.. _dis_01_0229:
|
||||
|
||||
The CDR Specification
|
||||
=====================
|
||||
|
||||
Charging requirement: Charging starts from the time when the DIS instance is created and stops when the DIS instance is deleted.
|
||||
|
||||
CDR record cycle: The minimum granularity is minute, and the current cycle is 60 minutes. The interval for uploading offline CDR files is one hour.
|
||||
|
||||
There are three types of Product ID corresponding to three billing factors in DIS. The detail information of the product ID looks like below:
|
||||
|
||||
.. table:: **Table 1** DIS Product ID
|
||||
|
||||
+--------------------------------+-----------------------+-------------------------------------------------------------+-----------------+
|
||||
| ResourceTypeCode | ID Commercial Product | Describe of Commercial Product EN | Unit of Measure |
|
||||
+================================+=======================+=============================================================+=================+
|
||||
| *otc*.resource.type.dispart | OTC_DIS_GEN_TIME | The length of time the user is using the general partition | second |
|
||||
+--------------------------------+-----------------------+-------------------------------------------------------------+-----------------+
|
||||
| | OTC_DIS_ADV_TIME | The length of time the user is using the advanced partition | second |
|
||||
+--------------------------------+-----------------------+-------------------------------------------------------------+-----------------+
|
||||
| *otc*.resource.type.dispayunit | OTC_DIS_GEN_UNIT | The number of the PUT Payload Unit for general partition | number |
|
||||
+--------------------------------+-----------------------+-------------------------------------------------------------+-----------------+
|
||||
| | OTC_DIS_ADV_UNIT | The number of the PUT Payload Unit for advanced partition | number |
|
||||
+--------------------------------+-----------------------+-------------------------------------------------------------+-----------------+
|
||||
| *otc*.resource.type.dissize | OTC_DIS_GEN_STORE | The data storage size in general partition. | byte |
|
||||
+--------------------------------+-----------------------+-------------------------------------------------------------+-----------------+
|
||||
| | OTC_DIS_ADV_STORE | The data storage size in advanced partition. | byte |
|
||||
+--------------------------------+-----------------------+-------------------------------------------------------------+-----------------+
|
||||
File diff suppressed because it is too large
Load Diff
@ -1,74 +0,0 @@
|
||||
:original_name: dis_01_0230.html
|
||||
|
||||
.. _dis_01_0230:
|
||||
|
||||
The Partition CDR Content
|
||||
=========================
|
||||
|
||||
.. table:: **Table 1** DIS Partition CDR Record Content
|
||||
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| Field | Location | Type | Description | Remarks |
|
||||
+======================+=============+=============+==================================================+===============================================================================================================================================================================================================================================================================================+
|
||||
| RecordType | 1 | String | Type of a record in a file. | Number of normal offline CDR records: 20 |
|
||||
| | | | | |
|
||||
| | | | | Number of corrected error offline CDR records: 30 |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| TimeStamp | 2 | String | Time when an offline CDR record is generated. | UTC Format: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| UserID | 3 | String | Unique ID of a subscriber. | The value is the same as the value of **ProjectID**. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| RegionCode | 4 | String | ID of a region. | ID of a region. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| AZCode | 5 | String | Code of an available zone. | The DIS is Region Service, The AZCode is set AZ01 by default. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| CloudServiceTypeCode | 6 | String | Type of a cloud service. | Cloud service type corresponding to **ResourceID**. For example, cloudservicetype for VM in **EC2** is recorded as **otc.service.type.ec2**, and cloudservicetype for VM in **VDC** is recorded as **otc\ .\ service.type.vdc**. |
|
||||
| | | | | |
|
||||
| | | | | For DIS: the string is **otc.service.type.dis** |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ResourceTypeCode | 7 | String | Type of a cloud resource. | Cloud resource type corresponding to **ResourceID**, such as VM. |
|
||||
| | | | | |
|
||||
| | | | | **ResourceType** is a resource type specified by **CloudServiceType**, and **CloudServiceType** contains multiple sets of **ResourceType**. For example, EC2 is provided by VM, Volume, IP, and Image, and EC2 is a **CloudServiceType**, and VM, Volume, IP, and Image are **ResourceType**. |
|
||||
| | | | | |
|
||||
| | | | | For DIS: the string is **otc.resource.type.dis** |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ResourceSpecCode | 8 | String | Code of specified resource specifications. | There are two type resourcespeccode for DIS: |
|
||||
| | | | | |
|
||||
| | | | | dis.general.partition |
|
||||
| | | | | |
|
||||
| | | | | dis.advanced.partition |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ResourceID | 9 | String | ID of a resource instance. | ID of the partition in the DIS stream instance. |
|
||||
| | | | | |
|
||||
| | | | | The ID is different from partitions. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| BSSParams | 10 | String | Extended operation parameter. | The contract ID of one tenant. The contract ID is the xdomain_id of user token. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| BeginTime | 11 | String | Start time of the statistics period of a record. | UTC Format: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| EndTime | 12 | String | End time of the statistics period of a record. | UTC Format: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| Duration | 13 | String | Name of a specific accumulative factor. | Accumulative factor: Duration |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| Time | 14 | Double | Accumulative value of **AccumulateFactorName**. | Accumulative value of duration in the period ranging from BeginTime to EndTime multiply the number of partitions. The value is 0 or a positive number. |
|
||||
| | | | | |
|
||||
| | | | | The time unit is hour. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ExtendParams | 15 | String | Extended field. | This field is not used in this CDR record, the value is empty. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ProductID | 16 | String | Product ID of this service | Product ID : |
|
||||
| | | | | |
|
||||
| | | | | OTC_DIS_GEN_TIME |
|
||||
| | | | | |
|
||||
| | | | | OTC_DIS_ADV_TIME |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| CET/CEST BeginTime | 17 | String | Start time of the statistics period of a record. | Server Local Time: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| CET/CEST EndTime | 18 | String | End time of the statistics period of a record. | Server Local Time: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| Tag | 19 | String | Extended field. | This field is not used in DIS service, the value is empty. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
|
||||
Example of Partition Duration CDR content:
|
||||
|
||||
20|20161013150423 \| a21be352dfa7682dfa768dfa7682dfdf \| eu-west-0 *\|* \| *otc.service.type.dis* \| *otc.resource.type.dispartition* \| dis.general.partition \| 2cegee33-2173-4982-881c-7w8f27d23h54 \|\ *00000000001000003344* \| 20161013130000 \| 20161013135959 \| Duration \| 3600 \| \| OTC_DIS_GEN_TIME \| 20161013140000 \| 20161013145959\|
|
||||
@ -1,78 +0,0 @@
|
||||
:original_name: dis_01_0231.html
|
||||
|
||||
.. _dis_01_0231:
|
||||
|
||||
The PUT Payload Unit CDR Content
|
||||
================================
|
||||
|
||||
.. table:: **Table 1** DIS PUT Payload Unit CDR Record Content
|
||||
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| Field | Location | Type | Description | Remarks |
|
||||
+======================+=============+=============+==================================================+===============================================================================================================================================================================================================================================================================================+
|
||||
| RecordType | 1 | String | Type of a record in a file. | Number of normal offline CDR records: 20 |
|
||||
| | | | | |
|
||||
| | | | | Number of corrected error offline CDR records: 30 |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| TimeStamp | 2 | String | Time when an offline CDR record is generated. | UTC Format: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| UserID | 3 | String | Unique ID of a subscriber. | The value is the same as the value of **ProjectID**. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| RegionCode | 4 | String | ID of a region. | ID of a region. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| AZCode | 5 | String | Code of an available zone. | Code of an available zone. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| CloudServiceTypeCode | 6 | String | Type of a cloud service. | Cloud service type corresponding to **ResourceID**. For example, cloudservicetype for VM in **EC2** is recorded as **otc.service.type.ec2**, and cloudservicetype for VM in **VDC** is recorded as **otc.service.type.vdc**. |
|
||||
| | | | | |
|
||||
| | | | | For DIS: the string is **otc.service.type.dis** |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ResourceTypeCode | 7 | String | Type of a cloud resource. | Cloud resource type corresponding to **ResourceID**, such as VM. |
|
||||
| | | | | |
|
||||
| | | | | **ResourceType** is a resource type specified by **CloudServiceType**, and **CloudServiceType** contains multiple sets of **ResourceType**. For example, EC2 is provided by VM, Volume, IP, and Image, and EC2 is a **CloudServiceType**, and VM, Volume, IP, and Image are **ResourceType**. |
|
||||
| | | | | |
|
||||
| | | | | For DIS: the string is **otc.resource.type.dis** |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ResourceSpecCode | 8 | String | Code of specified resource specifications. | There are two type resourcespeccode for DIS: |
|
||||
| | | | | |
|
||||
| | | | | dis.general.partition |
|
||||
| | | | | |
|
||||
| | | | | dis.advanced.partition |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ResourceID | 9 | String | ID of a resource instance. | ID of the partition in the DIS stream instance. |
|
||||
| | | | | |
|
||||
| | | | | The ID is different from partitions. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| BSSParams | 10 | String | Extended operation parameter. | The contract ID of one tenant. The contract ID is the xdomain_id of user token. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| BeginTime | 11 | String | Start time of the statistics period of a record. | UTC Format: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| EndTime | 12 | String | End time of the statistics period of a record. | UTC Format: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| InputTraffic | 13 | String | Name of a specific accumulative factor. | Accumulative factor: InputUnitNum |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| InputTrafficValue | 14 | Double | Accumulative value of **AccumulateFactorName**. | Accumulative value of InputUnitNum. |
|
||||
| | | | | |
|
||||
| | | | | This is the total Num of input record(sum/million) in the partition during this time period. |
|
||||
| | | | | |
|
||||
| | | | | The value unit is sum/million |
|
||||
| | | | | |
|
||||
| | | | | . |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ExtendParams | 15 | String | Extended field. | This field indicates the amount of data the user is actually putting. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| ProductID | 16 | String | Product ID of this service | Product ID: |
|
||||
| | | | | |
|
||||
| | | | | OTC_DIS_GEN_UNIT |
|
||||
| | | | | |
|
||||
| | | | | OTC_DIS_ADV_UNIT |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| CET/CEST BeginTime | 17 | String | Start time of the statistics period of a record. | Server Local Time: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| CET/CEST EndTime | 18 | String | End time of the statistics period of a record. | Server Local Time: YYYYMMDDHH24MISS |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| Tag | 19 | String | Extended field. | This field is not used in DIS service, the value is empty. |
|
||||
+----------------------+-------------+-------------+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
||||
|
||||
Example of PUT Payload Unit CDR content:
|
||||
|
||||
20|20161013150423 \| a21be352dfa7682dfa768dfa7682dfdf \| eu-west-0 *\|* \| *otc.service.type.dis* \| *otc.resource.type.dispayloadunit* \| dis.general.partition \| 1dbcee33-7837-4316-852d-5d9f27f11e28 \|\ *00000000001000003344* \| 20161013130000 \| 20161013135959 \| InputUnitNum \| 12000 \| 3201284 \| OTC_DIS_GEN_UNIT \|20161013140000 \| 20161013145959\|
|
||||
@ -1,18 +0,0 @@
|
||||
:original_name: dis_01_0222.html
|
||||
|
||||
.. _dis_01_0222:
|
||||
|
||||
CDR Specification
|
||||
=================
|
||||
|
||||
- :ref:`Billing Model <dis_01_0223>`
|
||||
- :ref:`CDR Folder and File Name <dis_01_0224>`
|
||||
- :ref:`CDR Specification <dis_01_0228>`
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
billing_model
|
||||
cdr_folder_and_file_name/index
|
||||
cdr_specification/index
|
||||
@ -1,38 +0,0 @@
|
||||
:original_name: dis_01_0014.html
|
||||
|
||||
.. _dis_01_0014:
|
||||
|
||||
Change History
|
||||
==============
|
||||
|
||||
+-----------------------------------+----------------------------------------------------------------------------------+
|
||||
| Release Date | What's New |
|
||||
+===================================+==================================================================================+
|
||||
| 2022-06-15 | Modified the following section: |
|
||||
| | |
|
||||
| | :ref:`What Is DIS? <dis_01_0001>` |
|
||||
| | |
|
||||
| | Added the following sections: |
|
||||
| | |
|
||||
| | :ref:`CDR Specification <dis_01_0222>` |
|
||||
+-----------------------------------+----------------------------------------------------------------------------------+
|
||||
| 2020-01-03 | Modified the following section: |
|
||||
| | |
|
||||
| | :ref:`Viewing Stream Monitoring Metrics <dis_01_0012>` |
|
||||
+-----------------------------------+----------------------------------------------------------------------------------+
|
||||
| 2019-06-12 | Modified the following section: |
|
||||
| | |
|
||||
| | :ref:`Related Services <dis_01_0004>` |
|
||||
+-----------------------------------+----------------------------------------------------------------------------------+
|
||||
| 2019-05-15 | Modified the following section: |
|
||||
| | |
|
||||
| | :ref:`What Is DIS? <dis_01_0001>` |
|
||||
+-----------------------------------+----------------------------------------------------------------------------------+
|
||||
| 2019-01-29 | Modified the following section: |
|
||||
| | |
|
||||
| | :ref:`Step 2: Preparing a DIS Application Development Environment <dis_01_0602>` |
|
||||
+-----------------------------------+----------------------------------------------------------------------------------+
|
||||
| 2018-11-09 | Accepted in OTC 3.2. |
|
||||
+-----------------------------------+----------------------------------------------------------------------------------+
|
||||
| 2018-03-16 | This is the first official release. |
|
||||
+-----------------------------------+----------------------------------------------------------------------------------+
|
||||
@ -1,74 +0,0 @@
|
||||
:original_name: dis_06_0013.html
|
||||
|
||||
.. _dis_06_0013:
|
||||
|
||||
Configuring Sample Projects
|
||||
===========================
|
||||
|
||||
The **dis-sdk-**\ *1.2.3*\ **.zip** package of the latest version in :ref:`Downloading SDKs <dis_06_0005>` provides a sample project. You can use a development tool (such as Eclipse) on the local server to compile and run the sample project. You can also develop your applications based on the sample project. The sample project code is available in the **\\dis-sdk-demo\\src\\com\\bigdata\\dis\\sdk\\demo** directory.
|
||||
|
||||
================= ====================
|
||||
Sample Code Description
|
||||
================= ====================
|
||||
ConsumerDemo.java How to download data
|
||||
ProducerDemo.java How to upload data
|
||||
================= ====================
|
||||
|
||||
Procedure
|
||||
---------
|
||||
|
||||
#. Decompress the **dis-sdk-**\ *1.2.3*\ **.zip** package obtained in :ref:`Downloading SDKs <dis_06_0005>` to obtain the **dis-sdk-demo** package and sample project.
|
||||
#. Import the Eclipse project.
|
||||
|
||||
a. Start Eclipse and choose **File** > **Import**. The **Import** dialog box is displayed.
|
||||
|
||||
b. Choose **General** > **Existing Projects into Workspace** and click **Next**. The **Import** dialog box is displayed.
|
||||
|
||||
c. Click **Browse** and select a save location for the **dis-sdk-demo** sample project. In the **Projects** area, select a sample project, as shown in :ref:`Figure 1 <dis_06_0013__f23cd224614d04888bfa7c759107662f3>`.
|
||||
|
||||
.. _dis_06_0013__f23cd224614d04888bfa7c759107662f3:
|
||||
|
||||
.. figure:: /_static/images/en-us_image_0000001222097302.png
|
||||
:alt: **Figure 1** Import Projects
|
||||
|
||||
**Figure 1** Import Projects
|
||||
|
||||
d. Click **Finish** to import the project.
|
||||
|
||||
#. Configure the demo project.
|
||||
|
||||
a. Set the project code to **UTF-8**.
|
||||
|
||||
#. In the navigation tree, right-click the required project under **Project Explorer** and choose **Properties** from the shortcut menu. The **Properties for dis-sdk-demo** dialog box is displayed.
|
||||
#. In the navigation tree, choose **Resource**. The **Resource** page is displayed in the right pane.
|
||||
#. In the **Other** drop-down list, select **UTF-8**.
|
||||
#. Click **Apply and Close**.
|
||||
|
||||
|
||||
.. figure:: /_static/images/en-us_image_0000001222417262.png
|
||||
:alt: **Figure 2** Resource
|
||||
|
||||
**Figure 2** Resource
|
||||
|
||||
b. Import a JAR dependency package.
|
||||
|
||||
#. In the navigation tree, right-click the required project under **Project Explorer** and choose **Properties** from the shortcut menu. The **Properties for dis-sdk-demo** dialog box is displayed.
|
||||
#. In the navigation tree, choose **Java Build Path**. The **Java Build Path** page is displayed in the right pane.
|
||||
#. Click the **Libraries** tab, and then click **Add External JARs**. The **JAR Selection** dialog box is displayed.
|
||||
#. Select the location of the **dis-sdk** folder in the decompressed **dis-sdk-**\ 1.2.3 file, and then click **Open**.
|
||||
#. On the **Properties for dis-sdk-demo** page, click **Apply and Close** to import all the **.jar** files in the current path and the **third_lib** folder to the project.
|
||||
|
||||
|
||||
.. figure:: /_static/images/en-us_image_0000001266977433.png
|
||||
:alt: **Figure 3** Java Build Path
|
||||
|
||||
**Figure 3** Java Build Path
|
||||
|
||||
c. Add the JDK.
|
||||
|
||||
#. In the navigation tree, right-click the required project under **Project Explorer** and choose **Properties** from the shortcut menu. The **Properties for dis-sdk-demo** dialog box is displayed.
|
||||
#. In the navigation tree, choose **Java Build Path**. The **Java Build Path** page is displayed in the right pane.
|
||||
#. Click the **Libraries** tab, and then click **Add Library**. The **Add Library** dialog box is displayed.
|
||||
#. Select **JRE System Library** and click **Next**. Verify that the version of **Workspace default JRE** is **jdk1.8** or later.
|
||||
#. Click **Finish** to exit the **Add Library** dialog box.
|
||||
#. Click **Apply and Close**.
|
||||
@ -1,8 +0,0 @@
|
||||
:original_name: dis_06_0020.html
|
||||
|
||||
.. _dis_06_0020:
|
||||
|
||||
Connecting to kafka consumer
|
||||
============================
|
||||
|
||||
The DIS SDK provides compatibility with native kafka client APIs.
|
||||
@ -1,30 +0,0 @@
|
||||
:original_name: dis_06_0501.html
|
||||
|
||||
.. _dis_06_0501:
|
||||
|
||||
Creating a Stream
|
||||
=================
|
||||
|
||||
Initialize a DIS client instance named **dic**. For details, see :ref:`Initializing a DIS Client <dis_06_0015>`.
|
||||
|
||||
When you use the DIS SDK to create a DIS stream, specify the stream name, number of partitions in the stream, and stream type.
|
||||
|
||||
**STREAM_TYPE_COMMON** indicates a common stream, and **STREAM_TYPE_ADVANCED** indicates an advanced stream.
|
||||
|
||||
.. code-block::
|
||||
|
||||
//Configure the channel name.
|
||||
String streamName = "myStream";
|
||||
//Configure the number of partitions in the stream.
|
||||
Integer partitionCount = new Integer(3);
|
||||
CreateStreamRequest createStreamRequest = new CreateStreamRequest();
|
||||
createStreamRequest.setStreamName(streamName);
|
||||
createStreamRequest.setPartitionCount(partitionCount);
|
||||
//Configure the stream type.
|
||||
createStreamRequest.setStreamType(CreateStreamRequest.STREAM_TYPE_COMMON);
|
||||
|
||||
After configuring **CreateStreamRequest**, you can create a stream by calling **createStream**.
|
||||
|
||||
.. code-block::
|
||||
|
||||
dic.createStream(createStreamRequest);
|
||||
@ -1,23 +0,0 @@
|
||||
:original_name: dis_06_0502.html
|
||||
|
||||
.. _dis_06_0502:
|
||||
|
||||
Deleting a Stream
|
||||
=================
|
||||
|
||||
Initialize a DIS client instance named **dic**. For details, see :ref:`Initializing a DIS Client <dis_06_0015>`.
|
||||
|
||||
Use the DIS SDK to delete a specified DIS stream.
|
||||
|
||||
.. code-block::
|
||||
|
||||
//Specify the name of the stream to be deleted.
|
||||
String streamName = "myStream";
|
||||
DeleteStreamRequest deleteStreamRequest = new DeleteStreamRequest();
|
||||
deleteStreamRequest.setStreamName(streamName);
|
||||
|
||||
After configuring **DeleteStreamRequest**, you can delete a stream by calling deleteStream.
|
||||
|
||||
.. code-block::
|
||||
|
||||
dic.deleteStream(deleteStreamRequest);
|
||||
@ -1,10 +0,0 @@
|
||||
:original_name: dis_06_0005.html
|
||||
|
||||
.. _dis_06_0005:
|
||||
|
||||
Downloading SDKs
|
||||
================
|
||||
|
||||
Data Ingestion Service Software Development Kit (DIS SDK) is the encapsulation of RESTful APIs provided by DIS to simplify development. You can use DIS by calling API functions provided by DIS SDK.
|
||||
|
||||
Download the package from https://dis.obs.eu-de.otc.t-systems.com/dis/download/dis-sdk-1.2.3.zip.
|
||||
@ -1,81 +0,0 @@
|
||||
:original_name: dis_06_0017.html
|
||||
|
||||
.. _dis_06_0017:
|
||||
|
||||
Downloading Streaming Data
|
||||
==========================
|
||||
|
||||
Sample Code
|
||||
-----------
|
||||
|
||||
Use the initialized client instance to obtain data through the DIS stream. The sample code is available in the **ConsumerDemo.java** file under the **dis-sdk-demo\\src\\main\\java\\com\\bigdata\\dis\\sdk\\demo** directory.
|
||||
|
||||
The value of **streamName** must be the same as that of **Stream Name** configured in :ref:`Step 1: Creating a DIS Stream <dis_01_0601>`.
|
||||
|
||||
.. code-block::
|
||||
|
||||
//Configure the stream name.
|
||||
String streamName = "streamName";
|
||||
// Configure the ID of the partition for data download.
|
||||
String partitionId = "0";
|
||||
//Configure the sequence number for data download.
|
||||
String startingSequenceNumber = "0";
|
||||
//Configure the data download mode.
|
||||
String cursorType = PartitionCursorTypeEnum.AT_SEQUENCE_NUMBER.name();
|
||||
try
|
||||
{
|
||||
//Obtain data cursors.
|
||||
GetPartitionCursorRequest request = new GetPartitionCursorRequest();
|
||||
request.setStreamName(streamName);
|
||||
request.setPartitionId(partitionId);
|
||||
request.setStartingSequenceNumber(startingSequenceNumber);
|
||||
request.setCursorType(cursorType);
|
||||
GetPartitionCursorResult response = dic.getPartitionCursor(request);
|
||||
String cursor = response.getPartitionCursor();
|
||||
log.info("Get stream {}[partitionId={}] cursor success : {}", streamName, partitionId, cursor);
|
||||
GetRecordsRequest recordsRequest = newGetRecordsRequest();
|
||||
GetRecordsResult recordResponse = null;
|
||||
while (true)
|
||||
{
|
||||
recordsRequest.setPartitionCursor(cursor);
|
||||
recordsRequest.setLimit(2);
|
||||
recordResponse = dic.getRecords(recordsRequest);
|
||||
//Obtain the next-batch data cursors.
|
||||
cursor = recordResponse.getNextPartitionCursor();
|
||||
for (Record record : recordResponse.getRecords())
|
||||
{
|
||||
log.info("Get Record [{}], partitionKey [{}], sequenceNumber [{}].",
|
||||
new String(record.getData().array()),
|
||||
record.getPartitionKey(),
|
||||
record.getSequenceNumber());
|
||||
}
|
||||
Thread.sleep(1000);
|
||||
}
|
||||
}
|
||||
catch (DISClientException e)
|
||||
{
|
||||
log.error("Failed to get a normal response, please check params and retry. Error message [{}]", e.getMessage(), e);
|
||||
}
|
||||
catch (ResourceAccessException e)
|
||||
{
|
||||
log.error("Failed to access endpoint. Error message [{}]", e.getMessage(), e);
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
log.error(e.getMessage(), e);
|
||||
}
|
||||
|
||||
Running the Program
|
||||
-------------------
|
||||
|
||||
Right-click the program and choose **Run As > 1 Java Application** from the shortcut menu. If the program runs successfully, the information similar to the following is displayed on the console:
|
||||
|
||||
.. code-block::
|
||||
|
||||
14:55:42.954 [main] INFOcom.bigdata.dis.sdk.DISConfig - get from classLoader
|
||||
14:55:44.103 [main] INFOcom.bigdata.dis.sdk.util.config.ConfigurationUtils - get from classLoader
|
||||
14:55:44.105 [main] INFOcom.bigdata.dis.sdk.util.config.ConfigurationUtils - propertyMapFromFile size : 2
|
||||
14:55:45.235 [main] INFOcom.bigdata.dis.sdk.demo.ConsumerDemo - Get stream streamName[partitionId=0] cursor success : eyJnZXRJdGVyYXRvclBhcmFtIjp7InN0cmVhbS1uYW1lIjoiZGlzLTEzbW9uZXkiLCJwYXJ0aXRpb24taWQiOiIwIiwiY3Vyc29yLXR5cGUiOiJBVF9TRVFVRU5DRV9OVU1CRVIiLCJzdGFydGluZy1zZXF1ZW5jZS1udW1iZXIiOiIxMDY4OTcyIn0sImdlbmVyYXRlVGltZXN0YW1wIjoxNTEzNjY2NjMxMTYxfQ
|
||||
14:55:45.305 [main] INFOcom.bigdata.dis.sdk.demo.ConsumerDemo - Get Record [hello world.], partitionKey [964885], sequenceNumber [0].
|
||||
14:55:45.305 [main] INFOcom.bigdata.dis.sdk.demo.ConsumerDemo - Get Record [hello world.], partitionKey [910960], sequenceNumber [1].
|
||||
14:55:46.359 [main] INFOcom.bigdata.dis.sdk.demo.ConsumerDemo - Get Record [hello world.], partitionKey [528377], sequenceNumber [2].
|
||||
@ -1,9 +0,0 @@
|
||||
:original_name: dis_06_0014.html
|
||||
|
||||
.. _dis_06_0014:
|
||||
|
||||
Importing Third-party Dependency Packages
|
||||
=========================================
|
||||
|
||||
#. In the navigation tree, click **Project Explorer**, right-click a project and choose **Properties > Java Build Path > Add External JARs**.
|
||||
#. Locate the directory containing the decompressed **dis-sdk** folder, and import all the files with the extension **JAR** in the current directory and the **third_party** folder to the project.
|
||||
@ -1,40 +0,0 @@
|
||||
:original_name: dis_06_0011.html
|
||||
|
||||
.. _dis_06_0011:
|
||||
|
||||
DIS SDK Usage Guide
|
||||
===================
|
||||
|
||||
- :ref:`Downloading SDKs <dis_06_0005>`
|
||||
- :ref:`Preparing the Environment <dis_06_0012>`
|
||||
- :ref:`Connecting to kafka consumer <dis_06_0020>`
|
||||
- :ref:`Configuring Sample Projects <dis_06_0013>`
|
||||
- :ref:`Importing Third-party Dependency Packages <dis_06_0014>`
|
||||
- :ref:`Initializing a DIS Client <dis_06_0015>`
|
||||
- :ref:`Creating a Stream <dis_06_0501>`
|
||||
- :ref:`Deleting a Stream <dis_06_0502>`
|
||||
- :ref:`Querying a Stream List <dis_06_0503>`
|
||||
- :ref:`Querying Stream Details <dis_06_0504>`
|
||||
- :ref:`Uploading Streaming Data <dis_06_0016>`
|
||||
- :ref:`Downloading Streaming Data <dis_06_0017>`
|
||||
- :ref:`Uploading Real-Time Files <dis_06_0022>`
|
||||
- :ref:`Obtaining the Data Cursor <dis_06_0505>`
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
downloading_sdks
|
||||
preparing_the_environment
|
||||
connecting_to_kafka_consumer
|
||||
configuring_sample_projects
|
||||
importing_third-party_dependency_packages
|
||||
initializing_a_dis_client
|
||||
creating_a_stream
|
||||
deleting_a_stream
|
||||
querying_a_stream_list
|
||||
querying_stream_details
|
||||
uploading_streaming_data
|
||||
downloading_streaming_data
|
||||
uploading_real-time_files/index
|
||||
obtaining_the_data_cursor
|
||||
@ -1,35 +0,0 @@
|
||||
:original_name: dis_06_0015.html
|
||||
|
||||
.. _dis_06_0015:
|
||||
|
||||
Initializing a DIS Client
|
||||
=========================
|
||||
|
||||
You can use either of the following methods to initialize a DIS SDK client instance: For details about **endpoint**, **ak**, **sk**, **region**, and **projectId**, see :ref:`Obtaining Authentication Information <dis_01_0043>`.
|
||||
|
||||
- Use the code to initialize the DIS SDK client instance. For details about the code example, see the **ProducerDemo.java** file.
|
||||
|
||||
.. code-block::
|
||||
|
||||
//Create a DIS client instance.
|
||||
DIS dic = DISClientBuilder.standard()
|
||||
.withEndpoint("https://ip:port")
|
||||
.withAk("xxxx")
|
||||
.withSk("xxxx")
|
||||
.withProjectId("xxxxxxx")
|
||||
.withRegion("xxxx")
|
||||
.build();
|
||||
|
||||
- Use the configuration file to initialize a DIS SDK client instance.
|
||||
|
||||
Add the following configuration items to the **dis.propertites** file in the **dis-sdk-demo\\resources** directory:
|
||||
|
||||
- ak/sk: AK/SK created on the IAM
|
||||
- region: region of the stream
|
||||
- endpoint: access address of the DIS
|
||||
- projectId: project ID of the stream
|
||||
|
||||
.. code-block::
|
||||
|
||||
//Create a DIS client instance.
|
||||
DIS dic = DISClientBuilder.standard().build();
|
||||
@ -1,29 +0,0 @@
|
||||
:original_name: dis_06_0505.html
|
||||
|
||||
.. _dis_06_0505:
|
||||
|
||||
Obtaining the Data Cursor
|
||||
=========================
|
||||
|
||||
Initialize a DIS client instance named **dic**. For details, see :ref:`Initializing a DIS Client <dis_06_0015>`.
|
||||
|
||||
Use the DIS SDK to obtain the information about the data cursor.
|
||||
|
||||
.. code-block::
|
||||
|
||||
//Configure the stream name.
|
||||
String streamName = "myStream";
|
||||
// Configure the ID of the partition for data download.
|
||||
String partitionId = "0";
|
||||
//Configure the sequence number for data download.
|
||||
String startingSequenceNumber = "0";
|
||||
//Configure the data download mode.
|
||||
String cursorType = PartitionCursorTypeEnum.AT_SEQUENCE_NUMBER.name();
|
||||
|
||||
GetPartitionCursorRequest request = new GetPartitionCursorRequest();
|
||||
request.setStreamName(streamName);
|
||||
request.setPartitionId(partitionId);
|
||||
request.setStartingSequenceNumber(startingSequenceNumber);
|
||||
request.setCursorType(cursorType);
|
||||
GetPartitionCursorResult response = dic.getPartitionCursor(request);
|
||||
String cursor = response.getPartitionCursor();
|
||||
@ -1,10 +0,0 @@
|
||||
:original_name: dis_06_0012.html
|
||||
|
||||
.. _dis_06_0012:
|
||||
|
||||
Preparing the Environment
|
||||
=========================
|
||||
|
||||
- Download JDK1.8 or a later version from the `Oracle official website <http://www.oracle.com/technetwork/java/archive-139210.html>`__ and install it, and configure Java environment variables.
|
||||
- Download Eclipse IDE for Java Developers of the latest version from the `Eclipse's official website <http://www.eclipse.org/downloads/eclipse-packages/>`__, and install it.
|
||||
- Configure the JDK in Eclipse.
|
||||
@ -1,24 +0,0 @@
|
||||
:original_name: dis_06_0503.html
|
||||
|
||||
.. _dis_06_0503:
|
||||
|
||||
Querying a Stream List
|
||||
======================
|
||||
|
||||
Initialize a DIS client instance named **dic**. For details, see :ref:`Initializing a DIS Client <dis_06_0015>`.
|
||||
|
||||
Use the DIS SDK to list active streams.
|
||||
|
||||
Use the **setLimit** method to set the number of streams returned each time. If **setLimit** is not specified, a maximum of 10 streams are returned by default.
|
||||
|
||||
.. code-block::
|
||||
|
||||
ListStreamsRequest listStreamsRequest = new ListStreamsRequest();
|
||||
listStreamsRequest.setLimit(5);
|
||||
System.out.println("listStream: " + JsonUtils.objToJson(dic.listStreams(listStreamsRequest)));
|
||||
|
||||
The returned stream list is as follows:
|
||||
|
||||
.. code-block::
|
||||
|
||||
listStream: {"total_number":20,"stream_names":["Stream0","Stream1","Stream2","Stream3","Stream4"],"has_more_streams":true}
|
||||
@ -1,23 +0,0 @@
|
||||
:original_name: dis_06_0504.html
|
||||
|
||||
.. _dis_06_0504:
|
||||
|
||||
Querying Stream Details
|
||||
=======================
|
||||
|
||||
Initialize a DIS client instance named **dic**. For details, see :ref:`Initializing a DIS Client <dis_06_0015>`.
|
||||
|
||||
Use the DIS SDK to query the details about a specified stream.
|
||||
|
||||
.. code-block::
|
||||
|
||||
String streamName = "myStream";
|
||||
DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest();
|
||||
describeStreamRequest.setStreamName(streamName);
|
||||
System.out.println("descStream: " + JsonUtils.objToJson(dic.describeStream(describeStreamRequest)));
|
||||
|
||||
The returned stream details are as follows:
|
||||
|
||||
.. code-block::
|
||||
|
||||
descStream: {"stream_name":"myStream","create_time":1515140868451,"last_modified_time":1515140868451,"retention_period":24,"status":"RUNNING","stream_type":"ADVANCED","partitions":[{"status":"ACTIVE","partition_id":"shardId-0000000000","hash_range":"[0 : 4611686018427387902]","sequence_number_range":"[0 : 0]"},{"status":"ACTIVE","partition_id":"shardId-0000000001","hash_range":"[4611686018427387903 : 9223372036854775807]","sequence_number_range":"[0 : 0]"}],"has_more_partitions":false}
|
||||
@ -1,73 +0,0 @@
|
||||
:original_name: dis_06_0506.html
|
||||
|
||||
.. _dis_06_0506:
|
||||
|
||||
Creating a Demo Class
|
||||
=====================
|
||||
|
||||
Creating a FileProducerAsyncDemo Class
|
||||
--------------------------------------
|
||||
|
||||
In the Project Explorer pane, right-click **com.bigdata.dis.sdk.demo**, and choose **New > Class** from the shortcut menu. The **New Java Class** dialog box is displayed. Set the category name to **FileProducerAsyncDemo**. Add the following codes to the new category:
|
||||
|
||||
.. code-block::
|
||||
|
||||
package com.bigdata.dis.sdk.demo;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import com.bigdata.dis.data.iface.request.PutFilesRequest;
|
||||
import com.bigdata.dis.data.iface.response.PutFilesResult;
|
||||
import com.bigdata.dis.sdk.DISAsync;
|
||||
import com.bigdata.dis.sdk.DISClientAsyncBuilder;
|
||||
import com.bigdata.dis.sdk.core.builder.DefaultExecutorFactory;
|
||||
import com.bigdata.dis.sdk.core.handler.AsyncHandler;
|
||||
|
||||
public class FileProducerAsyncDemo
|
||||
{
|
||||
private static final Logger log = LoggerFactory.getLogger(FileProducerAsyncDemo.class);
|
||||
|
||||
public static void main(String args[])
|
||||
throws Exception
|
||||
{
|
||||
DISAsync dicAsync = DISClientAsyncBuilder.standard()
|
||||
.withEndpoint("https://ip:port")
|
||||
.withAk("xxxx")
|
||||
.withSk("xxxx")
|
||||
.withProjectId("xxxx")
|
||||
.withRegion("xxxx")
|
||||
.withExecutorFactory(new DefaultExecutorFactory())
|
||||
.build();
|
||||
|
||||
// Configure the stream name.
|
||||
String streamName = "fileUploadStream";
|
||||
|
||||
PutFilesRequest putFilesRequest = new PutFilesRequest();
|
||||
putFilesRequest.setStreamName(streamName);
|
||||
|
||||
//Configure the path of the file to be uploaded.
|
||||
putFilesRequest.setFilePath("xxxx");
|
||||
|
||||
//Configure the file name for storing the uploaded real-time data file.
|
||||
putFilesRequest.setFileName("xxxx");
|
||||
|
||||
dicAsync.putFilesAsync(putFilesRequest, new AsyncHandler<PutFilesResult>()
|
||||
{
|
||||
|
||||
@Override
|
||||
public void onSuccess(PutFilesResult result)
|
||||
{
|
||||
log.info("Upload file to DIS successful!");
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onError(Exception exception)
|
||||
{
|
||||
log.error("Fail to upload file to DIS.", exception);
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
}
|
||||
}
|
||||
@ -1,18 +0,0 @@
|
||||
:original_name: dis_06_0022.html
|
||||
|
||||
.. _dis_06_0022:
|
||||
|
||||
Uploading Real-Time Files
|
||||
=========================
|
||||
|
||||
- :ref:`Creating a Demo Class <dis_06_0506>`
|
||||
- :ref:`Initializing a DIS SDK Asynchronous Client Instance <dis_06_0023>`
|
||||
- :ref:`Uploading Files <dis_06_0507>`
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
creating_a_demo_class
|
||||
initializing_a_dis_sdk_asynchronous_client_instance
|
||||
uploading_files
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user