2023-01-30
diff --git a/docs/dli/umn/dli_01_0001.html b/docs/dli/umn/dli_01_0001.html index 5d9350688..4ea540017 100644 --- a/docs/dli/umn/dli_01_0001.html +++ b/docs/dli/umn/dli_01_0001.html @@ -1,19 +1,72 @@ -Job Management
- --
-
- Overview
-
- - SQL Job Management
-
- - Flink Job Management
-
- - Spark Job Management
-
- - Setting the Priority for a Job
-
-
DLI Job Development Process
+This chapter walks you through on how to develop a DLI job.
+Creating an IAM User and Granting Permissions
- To manage fine-grained permissions for your DLI resources using IAM, create an IAM user and grant them permissions to DLI if you are an enterprise user. For details, see Creating an IAM User and Granting Permissions.
- When using DLI for the first time, you need to update the DLI agency according to the console's guidance so that DLI can use other cloud services and perform resource O&M operations on your behalf. The agency includes permissions to obtain IAM user information, access and use VPCs, CIDR blocks, routes, and peering connections, and send notifications via SMN in case of job execution failure.
For more information on the specific permissions included in the agency, refer to Configuring DLI Agency Permissions.
+
Creating Compute Resources and Metadata Required for Running Jobs
- Before submitting a job using DLI, you need to create an elastic resource pool and create queues within it. This will provide the necessary compute resources for running the job. For how to create an elastic resource pool and create queues within it, see Overview of DLI Elastic Resource Pools and Queues.
Alternatively, you can enhance DLI's computing environment by creating custom images. Specifically, to enhance the functions and performance of Spark and Flink jobs, you can create custom images by downloading the base images provided by DLI and adding dependencies (files, JAR files, or software) and private capabilities required for job execution. This changes the container runtime environment for the jobs.
+For example, you can add a Python package or C library related to machine learning to a custom image to help you extend functions. For how to create a custom image, see Using a Custom Image to Enhance the Job Running Environment.
+ - DLI metadata is the basis for developing SQL and Spark jobs. Before executing a job, you need to define databases and tables based on your business scenario.+
Flink allows for dynamic data types, enabling the definition of data structures at runtime without the need for predefined metadata.
+- Define your data structures, including data catalogs, databases, and tables. For details, see Creating Databases and Tables.
- Create a bucket to store temporary data generated during job running, such as job logs and job results. For details, see Configuring a DLI Job Bucket.
- Configure the permission to access metadata. For details, see Configuring Database Permissions on the DLI Console and Configuring Table Permissions on the DLI Console.
Importing Data to DLI
- DLI allows you to analyze and query data stored in OBS without the need to migrate it. Simply upload your data to OBS and use DLI for data analysis.
- Cross-source access can reduce data duplication and latency when real-time access and processing of data from different sources is required for service needs.
The prerequisites for cross-source access are that DLI can communicate with the data source network and DLI can obtain the access credentials to the data source.
+- Configure network connection between DLI and the data source by referring to Configuring the Network Connection Between DLI and Data Sources (Enhanced Datasource Connection).
- Manage data source credentials.
- You can use DLI's datasource authentication to manage the authentication information for accessing a specified datasource.
This applies to SQL jobs and Flink 1.12 jobs. For details, see Using DLI Datasource Authentication to Manage Access Credentials for Data Sources.
+ - You can also use DEW to manage access credentials for data sources and use a custom agency to authorize DLI to access DEW.
This applies to Spark 3.3.1 or later and Flink 1.15 or later. For details, see Using DEW to Manage Access Credentials for Data Sources and Configuring an Agency to Allow DLI to Access Other Cloud Services.
+
- You can use DLI's datasource authentication to manage the authentication information for accessing a specified datasource.
Submitting a Job Using DLI
- DLI offers a serverless service that integrates stream processing, batch processing, and interactive analytics. It supports various job types to meet different data processing needs.
++
+Table 1 Job types supported by DLI Job Type
+Description
+Use Case
+SQL job
+This type is suitable for scenarios where standard SQL statements are used for querying. It is typically used for querying and analyzing structured data.
+For details, see Creating and Submitting a SQL Job.
+It applies to scenarios such as data warehouse query, report generation, and online analytical processing (OLAP).
+Flink job
+This type is specifically designed for real-time data stream processing, making it ideal for scenarios that require low latency and quick response. It is well-suited for real-time monitoring and online analysis.
+- Flink OpenSource job: DLI provides standard connectors and various APIs to facilitate quick integration with other data systems. For details, see Creating a Flink OpenSource SQL Job.
- Flink Jar job: allows you to submit Flink jobs compiled into JAR files, providing greater flexibility and customization capabilities.
It is suitable for complex data processing scenarios that require user-defined functions (UDFs) or specific library integration. The Flink ecosystem can be utilized to implement advanced stream processing logic and status management. For details, see Creating a Flink Jar Job.
+
It applies to scenarios that require quick response, such as real-time data monitoring and real-time recommender systems.
+Flink Jar jobs are suitable for data analysis scenarios that require custom stream processing logic, complex state management, or integration with specific libraries.
+Spark job
+Compute jobs can be submitted through interactive sessions or batch processing. Jobs are submitted to queues created within an elastic resource pool, simplifying resource management and job scheduling.
+It supports multiple data sources and formats, providing rich data processing capabilities, including but not limited to SQL queries and machine learning. For details, see Creating a Spark Job.
+It is suitable for large-scale data processing and analysis, such as machine learning training, log analysis, and large-scale data mining.
+ - Manage program packages of Jar jobs.
DLI allows you to submit Flink or Spark jobs compiled as JAR files, which contain the necessary code and dependency information for executing the job. These files are used for specific data processing tasks such as data query, analysis, and machine learning. You can manage program packages required for jobs on the DLI console.
+To submit a Spark Jar or Flink Jar job, you must first upload the program package to OBS, create a program package in DLI, and then submit the program package, data, and job parameters to run the job. For details, see Managing Program Packages of Jar Jobs.
++ For Spark 3.3.1 or later and Flink 1.15 or later, when creating a Jar job, you can directly configure the program package in OBS. Program packages cannot be read from DLI.
+
Using Cloud Eye to Monitor DLI
You can query DLI monitoring metrics and alarms through Cloud Eye management console or APIs.
+For example, you can monitor the resource usage and job status of a DLI queue. For details about DLI metrics, see Using Cloud Eye to Monitor DLI.
+Using CTS to Audit DLI
With CTS, you can log operations related to DLI, making it easier to search, audit, and trace in the future. For the supported operations, see Using CTS to Audit DLI.
+Creating and Submitting a Spark SQL Job
-Scenario
DLI can query data stored in OBS. This section describes how to us a Spark SQL job on DLI to query OBS data.
+Using DLI to Submit a SQL Job to Query OBS Data
+Scenario
DLI can query data stored in OBS. This section describes how to use DLI to submit a SQL job to query OBS data.
+To illustrate, create a new file called sampledata.csv and upload it to an OBS bucket. Then, create an elastic resource pool, create queues within it, and use DLI to create a database and table. Finally, use DLI's SQL editor to query 1,000 data records from the table.
Procedure
You can use DLI to submit a Spark SQL job to query data. The general procedure is as follows:
- - - - - +Procedure
Table 1 shows the process for submitting a SQL job to query OBS data.
+ +Procedure + |
+Description + |
+
|---|---|
| + | +Upload data files to OBS. + |
+
Step 2: Create an Elastic Resource Pool and Add Queues to the Pool + |
+Create compute resources required for submitting jobs. + |
+
| + | +DLI metadata forms the foundation for SQL job development. Before executing a job, you need to define databases and tables based on your business scenario. + |
+
| + | +Use the sample data stored in OBS to create tables in the db1 database. + |
+
| + | +Use standard SQL statements to query and analyze data. + |
+
Step 1: Upload Data to OBS
Before you use DLI to query and analyze data, upload data files to OBS.
-- Go to the DLI console.
- In the service list, click Object Storage Service under Storage. The OBS console page is displayed.
- Create a bucket. In this example, the bucket name is obs1.
- Click Create Bucket in the upper right corner.
- On the displayed Create Bucket page, enter the Bucket Name. Retain the default values for other parameters or adjust them as needed. +
You must select the same region as the DLI management console.
+-Step 1: Upload Data to OBS
Upload data files to OBS.
+- Log in to the OBS management console.
- Create a bucket. In this example, the bucket name is obs1.
- Click Create Bucket in the upper right corner.
- On the displayed Create Bucket page, enter the Bucket Name. Retain the default values for other parameters or adjust them as needed.
Select a region that matches the location of the DLI console.
- Click Create Now.
- Click obs1 to access its Objects tab page.
- Click Upload Object. In the displayed dialog box, drag a desired file or folder, for example, sampledata.csv to the Upload Object area. Then, click Upload.You can create a sampledata.txt file, copy the following content separated by commas (,), and save the file as sampledata.csv.+
12,test
- - Click obs1 to access its Objects tab page.
- Click Upload Object. In the displayed dialog box, drag a desired file or folder, for example, sampledata.csv to the Upload Object area. Then, click Upload.
You can create a sampledata.txt file, copy the following content separated by commas (,), and save the file as sampledata.csv.
+product_id,product_name +113,office_13 +22,book_2 +29,book_9
After the file is uploaded successfully, the file path is obs://obs1/sampledata.csv.
-+ - For more information about OBS operations, see the Object Storage Service Console Operation Guide.
- For more information about the tool, see the OBS Tool Guide.
- You are advised to use an OBS tool, such as OBS Browser+, to upload large files because OBS Console has restrictions on the file size and quantity.
- OBS Browser+ is a graphical tool that provides complete functions for managing your buckets and objects in OBS.
For more operations on the OBS console, see the Object Storage Service User Guide.
Step 2: Create a Queue
A queue is the basis for using DLI. Before executing a SQL job, you need to create a queue.
-- DLI provides a preconfigured queue named default.
- You can also create queues as needed.
- Log in to the DLI management console.
- In the left navigation pane of the DLI management console, choose SQL Editor.
- On the left pane, select the Queues tab, and click
next to Queues.For details, see Creating a Queue.
- -
Step 2: Create an Elastic Resource Pool and Add Queues to the Pool
In this example, the elastic resource pool dli_resource_pool and queue dli_queue_01 are created.- Log in to the DLI management console.
- In the navigation pane on the left, choose Resources > Resource Pool.
- On the displayed page, click Buy Resource Pool in the upper right corner.
- On the displayed page, set the parameters.Table 2 describes the parameters. +++
+Table 2 Parameters Parameter
+Description
+Example Value
+Region
+Select a region where you want to buy the elastic resource pool.
+_
+Project
+Project uniquely preset by the system for each region
+Default
+Name
+Name of the elastic resource pool
+dli_resource_pool
+Specifications
+Specifications of the elastic resource pool
+Standard
+CU Range
+The maximum and minimum CUs allowed for the elastic resource pool
+64-64
+CIDR Block
+CIDR block the elastic resource pool belongs to. If you use an enhanced datasource connection, this CIDR block cannot overlap that of the data source. Once set, this CIDR block cannot be changed.
+172.16.0.0/19
+Enterprise Project
+Select an enterprise project for the elastic resource pool.
+default
+ - Click Buy.
- Click Submit.
- In the elastic resource pool list, locate the pool you just created and click Add Queue in the Operation column.
- Set the basic parameters listed below.
++
+Table 3 Basic parameters for adding a queue Parameter
+Description
+Example Value
+Name
+Name of the queue to add
+dli_queue_01
+Type
+Type of the queue
+- To execute SQL jobs, select For SQL.
- To execute Flink or Spark jobs, select For general purpose.
_
+Enterprise Project
+Select an enterprise project.
+default
+ - Click Next and configure scaling policies for the queue.
Click Create to add a scaling policy with varying priority, period, minimum CUs, and maximum CUs.
+ + ++
+Table 4 Scaling policy parameters Parameter
+Description
+Example Value
+Priority
+Priority of the scaling policy in the current elastic resource pool. A larger value indicates a higher priority. In this example, only one scaling policy is configured, so its priority is set to 1 by default.
+1
+Period
+The first scaling policy is the default policy, and its Period parameter configuration cannot be deleted or modified.
+The period for the scaling policy is from 00 to 24.
+00–24
+Min CU
+Minimum number of CUs allowed by the scaling policy
+16
+Max CU
+Maximum number of CUs allowed by the scaling policy
+64
+ - Click OK.
Step 3: Create a Database
Before querying data, create a database, for example, db1.
The default database is a built-in database. You cannot create the default. database.
- In the left navigation pane of the DLI management console, choose SQL Editor.
- In the editing window on the right of the SQL Editor page, enter the following SQL statement and click Execute. Read and agree to the privacy agreement, and click OK.
create database db1;
-After the database is successfully created, click
-
in the middle pane to refresh the database list. The new database db1 is displayed in the list.+ When you execute a query on the DLI management console for the first time, you need to read the privacy agreement. You can perform operations only after you agree to the agreement. For later queries, you will not need to read the privacy agreement again.
-After the database is successfully created, click
in the middle pane to refresh the database list. The new database db1 is displayed in the list.
diff --git a/docs/dli/umn/dli_01_0003.html b/docs/dli/umn/dli_01_0003.html index 35c6b2bf1..67edbadd3 100644 --- a/docs/dli/umn/dli_01_0003.html +++ b/docs/dli/umn/dli_01_0003.html @@ -1,18 +1,47 @@ -Step 4: Create a Table
After database db1 is created, create a table (for example, table1) containing data in the sample file obs://obs1/sampledata.csv stored on OBS in db1.
-- In the SQL editing window of the SQL Editor page, select the default queue and database db1.
- Enter the following SQL statement in the job editor window and click Execute:
create table table1 (id int, name string) using csv options (path 'obs://obs1/sampledata.csv');
+- In the SQL editing window of the SQL Editor page, select the default queue and database db1.
- Enter the following SQL statement in the job editor window and click Execute:
create table table1 (product_id int, product_name string) using csv options (path 'obs://obs1');
+When creating a table, you only need to specify the OBS storage path where the data file is located, without specifying the file name at the end of the directory.
After the table is successfully created, click the Databases tab then db1. The created table table1 is displayed in the table list.
Overview
-What Is Enhanced Datasource Connection?
In cross-source data analysis scenarios, DLI needs to connect to external data sources. However, due to the different VPCs between the data source and DLI, the network cannot be connected, which results in DLI being unable to read data from the data source. DLI's enhanced datasource connection feature enables network connectivity between DLI and the data source.
+Overview of Enhanced Datasource Connections
+Why Create Enhanced Datasource Connections?
In cross-source data analysis scenarios, DLI needs to connect to external data sources. However, due to the different VPCs between the data source and DLI, the network cannot be connected, which results in DLI being unable to read data from the data source. DLI's enhanced datasource connection feature enables network connectivity between DLI and the data source.
This section will introduce a solution for cross-VPC data source network connectivity:
- Creating an enhanced datasource connection: Establish a VPC peering connection to connect DLI and the data source's VPC network.
- Testing network connectivity: Verify the connectivity between the queue and the data source's network.
For details about the data sources that support cross-source access, see Cross-Source Analysis Development Methods.
-
-In cross-source development scenarios, there is a risk of password leakage if datasource authentication information is directly configured. You are advised to use the datasource authentication provided by DLI. For details, see Overview.
+For details about the data sources that support cross-source access, see Common Development Methods for DLI Cross-Source Analysis.
+ In cross-source development scenarios, there is a risk of password leakage if datasource authentication information is directly configured. You are advised to use Data Encryption Workshop (DEW) to store authentication information of data sources when Spark 3.3.1 or later and Flink 1.15 or later jobs access data sources using datasource connections. This will help you address issues related to data security, key security, and complex key management. For details, see Using DEW to Manage Access Credentials for Data Sources.
Constraints
- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
- When checking the connectivity of datasource connections, the notes and constraints on IP addresses are:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
For example, 192.168.xx.xx or 192.168.xx.xx:8181.
+Notes and Constraints
+
+Table 1 Notes and constraints on enhanced datasource connections Item
+Description
+Use case
+- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
Permission
+- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
Usage
+- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
Connectivity check
+- When checking the connectivity of datasource connections, the notes and constraints on IP addresses are:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
- When checking the connectivity of datasource connections, the notes and constraints on domain names are:
- The domain name can contain 1 to 255 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535. +
- When checking the connectivity of datasource connections, the notes and constraints on domain names are:
- The domain name can contain 1 to 255 characters. Only letters, numbers, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
Cross-Source Analysis Process
To use DLI for cross-source analysis, you need to create a datasource connection to connect DLI to the data source, and then develop jobs to access the data source.
Figure 1 Cross-source analysis flowchart@@ -20,7 +49,7 @@
diff --git a/docs/dli/umn/dli_01_0004.html b/docs/dli/umn/dli_01_0004.html index 1d24cba4b..1f4245345 100644 --- a/docs/dli/umn/dli_01_0004.html +++ b/docs/dli/umn/dli_01_0004.html @@ -1,13 +1,21 @@ -Data Management
+Managing Database Resources on the DLI Console
diff --git a/docs/dli/umn/dli_01_0005.html b/docs/dli/umn/dli_01_0005.html index 6f1a0004b..35dd222c2 100644 --- a/docs/dli/umn/dli_01_0005.html +++ b/docs/dli/umn/dli_01_0005.html @@ -1,46 +1,45 @@ --
-
- Databases and Tables
+ - Configuring Database Permissions on the DLI Console
- - Package Management
+ - Deleting a Database on the DLI Console
+
+ - Changing the Database Owner on the DLI Console
+
+ - Managing Tags
+Parent topic: Creating Databases and Tables+Creating a Database or a Table
-Definition of Database and Table in DLI
A database, built on the computer storage device, is a data warehouse where data is organized, stored, and managed based on its structure.
+Creating a Database and Table on the DLI Console
+A database, built on the computer storage device, is a data warehouse where data is organized, stored, and managed based on its structure.
The table is an important part of the database. It consists of rows and columns. Each column functions as a field. Each value in a field (column) represents a type of data.
The database is a framework and the table contains data content. A database has one or more tables.
You can create databases and tables on the management console or using SQL statements. This section describes how to create a database and a table on the management console.
- A view can be created only by using SQL statements. You cannot create a view on the Create Table page.
Precautions
- If a folder and a file have the same name in the OBS directory, the file path is preferred as the path of the OBS table to be created.
Creating a Database
- You can create a database on either the Data Management page or the SQL Editor page.
- To create a database on the Data Management page:
- On the left of the management console, choose Data Management > Databases and Tables.
- In the upper right corner of the Databases and Tables page, click Create Database to create a database.
- To create a database on the SQL Editor page:
- On the left of the management console, click SQL Editor.
- In the navigation pane on the left, click
next to Databases.
- To create a database on the Data Management page:
- In the displayed Create Database dialog box, specify Name and Description by referring to Table 1.
-
Table 1 Description Parameter
+Table 1 Description Parameter
Description
+Description
Database Name
+Database Name
- The database name can contain only digits, letters, and underscores (_), but cannot contain only digits or start with an underscore (_).
- The database name is case insensitive and cannot be left blank.
- The length of the database name cannot exceed 128 characters.
- Only numbers, letters, and underscores (_) are allowed. The value cannot contain only numbers or start with an underscore (_).
- The database name is case insensitive and cannot be left blank.
- The value can contain a maximum of 128 characters.
NOTE:The default database is a built-in database. You cannot create the default. database.
Description
+Description
Description of a database.
+Description of a database.
Tag
+Tags
Tags used to identify cloud resources. A tag includes the tag key and tag value. If you want to use the same tag to identify multiple cloud resources, that is, to select the same tag from the drop-down list box for all services, you are advised to create predefined tags on the Tag Management Service (TMS).
+Tags used to identify cloud resources. A tag includes the tag key and tag value. If you want to use the same tag to identify multiple cloud resources, that is, to select the same tag from the drop-down list box for all services, you are advised to create predefined tags on the Tag Management Service (TMS).
NOTE:-- A maximum of 20 tags can be added.
- Only one tag value can be added to a tag key.
- The key name in each resource must be unique.
- Tag key: Enter a tag key name in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
+- Tag key: Enter a tag key name in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
- Tag value: Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed. The value cannot start or end with a space.
+- Tag value: Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed.
- Tag value: Enter a tag value in the text box.
- Tag key: Enter a tag key name in the text box.
Creating a Table
Before creating a table, ensure that a database has been created.
-- You can create a table on either the Databases and Tables page or the SQL Editor page.
Datasource connection tables, such as View tables, HBase (MRS) tables, OpenTSDB (MRS) tables, GaussDB(DWS) tables, RDS tables, and CSS tables, cannot be created. You can use SQL to create views and datasource connection tables. For details, see sections Creating a View and Creating a Datasource Connection Table in the Data Lake Insight SQL Syntax Reference.
+- You can create a table on either the Databases and Tables page or the SQL Editor page.-
Datasource connection tables, such as View tables, HBase (MRS) tables, OpenTSDB (MRS) tables, GaussDB(DWS) tables, RDS tables, and CSS tables, cannot be created. You can use SQL to create views and datasource connection tables. For details, see sections Creating a View and Creating a Datasource Connection Table in the Data Lake Insight SQL Syntax Reference.
- To create a table on the Data Management page:
- On the left of the management console, choose Data Management > Databases and Tables.
- On the Databases and Tables page, select the database for which you want to create a table. In the Operation column, click More > Create Table to create a table in the current database.
- To create a table on the Data Management page:
- On the left of the management console, choose Data Management > Databases and Tables.
- On the Databases and Tables page, select the database for which you want to create a table. In the Operation column, click More > Create Table to create a table in the current database.
- To create a table on the SQL Editor page:
- On the left of the management console, click SQL Editor.
- In the navigation pane of the displayed SQL Editor page, click Databases. You can create a table in either of the following ways:
- Click a database name. In the Tables area, click
on the right to create a table in the current database. - Click
on the right of the database and choose Create Table from the shortcut menu to create a table in the current database.
- Click a database name. In the Tables area, click
- In the displayed Create Table dialog box, set parameters as required.
- If you set Data Location to DLI, set related parameters by referring to Table 2.
- If you set Data Location to OBS, set related parameters by referring to Table 2 and Table 3.
-
Table 2 Common parameters Parameter
+-Table 2 Common parameters Parameter
Description
+Description
Example
+Example
Table Name
+Table Name
- The table name can contain only digits, letters, and underscores (_), but cannot contain only digits or start with an underscore (_).
- The table name is case insensitive and cannot be left unspecified.
- The table name can contain the dollar sign ($). An example value is $test.
- The length of the table name cannot exceed 128 characters.
- Only numbers, letters, and underscores (_) are allowed. The value cannot contain only numbers or start with an underscore (_).
- The table name is case insensitive and cannot be left unspecified.
- The table name can contain the dollar sign ($). An example value is $test.
- The value can contain a maximum of 128 characters.
table01
+table01
Data Location
+Data Location
Data storage location. Currently, DLI and OBS are supported.
+Data storage location. Currently, DLI and OBS are supported.
DLI
+DLI
Description
+Description
Description of the table.
+Description of the table.
-
+-
Column Type
+Column Type
Available values: Normal or Partition
+Available values: Normal or Partition
Normal
+Normal
Column
+Column
Name of a column in a table. The column name must contain at least one letter and can contain underscores (_). It cannot contain only digits.
+Name of a column in a table. The column name must contain at least one letter and can contain underscores (_). It cannot contain only numbers.
You can select Normal or Partition. Partition columns are dedicated to partition tables. User data is partitioned to improve query efficiency.
NOTE:The column name is case-insensitive and must be unique.
name
+name
Type
+Type
Data type of a column. This parameter corresponds to Column Name.
-- string: The data is of the string type.
- int: Each integer is stored on four bytes.
- date: The value ranges from 0000-01-01 to 9999-12-31.
- double: Each number is stored on eight bytes.
- boolean: Each value is stored on one byte.
- decimal: The valid bits are positive integers between 1 to 38, including 1 and 38. The decimal digits are integers less than 10.
- smallint/short: The number is stored on two bytes.
- bigint/long: The number is stored on eight bytes.
- timestamp: The data indicates a date and time. The value can be accurate to six decimal points.
- float: Each number is stored on four bytes.
- tinyint: Each number is stored on one byte. Only OBS tables support this data type.
Data type of a column. This parameter corresponds to Column Name.
+- string: The data is of the string type.
- int: Each integer is stored on four bytes.
- date: The value ranges from 0000-01-01 to 9999-12-31.
- double: Each number is stored on eight bytes.
- boolean: Each value is stored on one byte.
- decimal: The valid bits are positive integers between 1 to 38, including 1 and 38. The decimal digits are integers less than 10.
- smallint/short: The number is stored on two bytes.
- bigint/long: The number is stored on eight bytes.
- timestamp: The data indicates a date and time. The value can be accurate to six decimal points.
- float: Each number is stored on four bytes.
- tinyint: Each number is stored on one byte. Only OBS tables support this data type.
string
+string
Column Description
+Column Description
Description of a column.
+Description of a column.
-
+-
Operation
+Operation
- Add Column
- DeleteNOTE:
If the table to be created includes a great number of columns, you are advised to use SQL statements to create the table or import column information from the local EXCEL file.
+- Add Column
- DeleteNOTE:
If the table to be created includes a great number of columns, you are advised to use SQL statements to create the table or import column information from the local EXCEL file.
-
+-
Table 3 Parameter description when Data Location is set to OBS Parameter
+-Table 3 Parameter description when Data Location is set to OBS Parameter
Description
+Description
Example
+Example
Data Format
+Data Format
DLI supports the following data formats:
-- Parquet: DLI can read non-compressed data or data that is compressed using Snappy and gzip.
- CSV: DLI can read non-compressed data or data that is compressed using gzip.
- ORC: DLI can read non-compressed data or data that is compressed using Snappy.
- JSON: DLI can read non-compressed data or data that is compressed using gzip.
- Avro: DLI can read uncompressed Avro data.
DLI supports the following data formats:
+- Parquet: DLI can read non-compressed data or data that is compressed using Snappy and gzip.
- CSV: DLI can read non-compressed data or data that is compressed using gzip.
- ORC: DLI can read non-compressed data or data that is compressed using Snappy.
- JSON: DLI can read non-compressed data or data that is compressed using gzip.
- Avro: DLI can read uncompressed Avro data.
CSV
+CSV
Storage Path
+Storage Path
Enter or select an OBS path. The path can be a folder or a path.
+Enter or select an OBS path. The path can be a folder or a path.
NOTE:If you need to import data stored in OBS to the OBS table, set this parameter to the path of a folder. If the table creation path is a file, data fails to be imported.
obs://obs1/sampledata.csv
+obs://obs1/sampledata.csv
Table Header: No/Yes
+Table Header: No/Yes
This parameter is valid only when Data Format is set to CSV. Whether the data source to be imported contains the table header.
-Click Advanced Settings and select the check box next to Table Header: No. If the check box is selected, the table header is displayed. If the check box is deselected, no table header is displayed.
+This parameter is valid only when Data Format is set to CSV. Whether the data source to be imported contains the table header.
+Click Advanced Settings and select the checkbox next to Table Header: No. If the checkbox is selected, the table header is displayed. If the checkbox is deselected, no table header is displayed.
-
+-
User-defined Delimiter
+User-defined Delimiter
This parameter is valid only when Data Format is set to CSV and you select User-defined Delimiter.
+This parameter is valid only when Data Format is set to CSV and you select User-defined Delimiter.
The following delimiters are supported:
- Comma (,)
- Vertical bar (|)
- Tab character (\t)
- Others: Enter a user-defined delimiter.
Comma (,)
+Comma (,)
User-defined Quotation Character
+User-defined Quotation Character
This parameter is valid only when Data Format is set to CSV and you select User-defined Quotation Character.
+This parameter is valid only when Data Format is set to CSV and you select User-defined Quotation Character.
The following quotation characters are supported:
- Single quotation mark (')
- Double quotation marks (")
- Others: Enter a user-defined quotation character.
Single quotation mark (')
+Single quotation mark (')
User-defined Escape Character
+User-defined Escape Character
This parameter is valid only when Data Format is set to CSV and you select User-defined Escape Character.
+This parameter is valid only when Data Format is set to CSV and you select User-defined Escape Character.
The following escape characters are supported:
- Backslash (\)
- Others: Enter a user-defined escape character.
Backslash (\)
+Backslash (\)
Date Format
+Date Format
This parameter is valid only when Data Format is set to CSV or JSON.
+This parameter is valid only when Data Format is set to CSV or JSON.
This parameter specifies the format of the date in the table and is valid only Advanced Settings is selected. The default value is yyyy-MM-dd. For definition of characters involved in the date pattern, see Table 3 in .
2000-01-01
+2000-01-01
Timestamp Format
+Timestamp Format
This parameter is valid only when Data Format is set to CSV or JSON.
+This parameter is valid only when Data Format is set to CSV or JSON.
This parameter specifies the format of the timestamp in the table and is valid only Advanced Settings is selected. The default value is yyyy-MM-dd HH:mm:ss. For definition of characters involved in the time pattern, see Table 3 in .
2000-01-01 09:00:00
+2000-01-01 09:00:00
- Click OK.
After a table is created, you can view and select the table for use on the Data Management page or SQL Editor page.
+- Click OK.
After a table is created, you can view and select the table for use on the Data Management page or SQL Editor page.
- (Optional) After a DLI table is created, you can decide whether to directly import data to the table.
diff --git a/docs/dli/umn/dli_01_0006.html b/docs/dli/umn/dli_01_0006.html index 0973a4bb7..9fe5f682f 100644 --- a/docs/dli/umn/dli_01_0006.html +++ b/docs/dli/umn/dli_01_0006.html @@ -5,19 +5,48 @@-Parent topic: Databases and Tables+Parent topic: Creating Databases and TablesFor example, to connect DLI to the MRS, RDS, CSS, Kafka, or GaussDB(DWS) data source, you need to enable the network between DLI and the VPC of the data source.
Create an enhanced datasource connection on the console.
-Constraints
- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
- When checking the connectivity of datasource connections, the notes and constraints on IP addresses are:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
For example, 192.168.xx.xx or 192.168.xx.xx:8181.
+Notes and Constraints
+
+Table 1 Notes and constraints on enhanced datasource connections Item
+Description
+Use case
+- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
Permission
+- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
Usage
+- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
Connectivity check
+- When checking the connectivity of datasource connections, the notes and constraints on IP addresses are:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
- When checking the connectivity of datasource connections, the notes and constraints on domain names are:
- The domain name can contain 1 to 255 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535. +
- When checking the connectivity of datasource connections, the notes and constraints on domain names are:
- The domain name can contain 1 to 255 characters. Only letters, numbers, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
Process
Figure 1 Enhanced datasource connection creation flowchart
-Prerequisites
- An elastic resource pool or queue has been created.
- You have obtained the VPC, subnet, private IP address, port, and security group information of the external data source.
- The security group of the external data source has allowed access from the CIDR block of the elastic resource pool or queue.
Procedure
- Create an Enhanced Datasource Connection
- Log in to the DLI management console.
- In the left navigation pane, choose Datasource Connections.
- On the Enhanced tab page displayed, click Create.
Configure parameters according to Table 1.
+Procedure
- Create an Enhanced Datasource Connection
- Log in to the DLI management console.
- In the navigation pane on the left, choose Datasource Connections.
- On the displayed Enhanced tab, click Create.
Set parameters based on Table 2.
-Table 1 Parameters Parameter
+Table 2 Parameters Parameter
Description
Connection Name
Name of the created datasource connection.
-- The name can contain only letters, digits, and underscores (_). The parameter must be specified.
- A maximum of 64 characters are allowed.
- Only letters, numbers, and underscores (_) are allowed. The parameter must be specified.
- A maximum of 64 characters are allowed.
Resource Pool
@@ -62,10 +91,10 @@Tags used to identify cloud resources. A tag includes the tag key and tag value. If you want to use the same tag to identify multiple cloud resources, that is, to select the same tag from the drop-down list box for all services, you are advised to create predefined tags on the Tag Management Service (TMS).
NOTE:-- A maximum of 20 tags can be added.
- Only one tag value can be added to a tag key.
- The key name in each resource must be unique.
- Tag key: Enter a tag key name in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
+- Tag key: Enter a tag key name in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
- Tag value: Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed. The value cannot start or end with a space.
+- Tag value: Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed.
- Tag value: Enter a tag value in the text box.
- Tag key: Enter a tag key name in the text box.
- Click OK.
After the creation is complete, the enhanced datasource connection is in the Active state, indicating that the connection is successfully created.
- Security Group Where the Data Source Belongs Allows Access from the CIDR Block of the Elastic Resource Pool
- On the DLI management console, obtain the network segment of the elastic resource pool or queue.
Choose Resources > Queue Management from the left navigation pane. On the page displayed, locate the queue on which jobs are running, and click the button next to the queue name to obtain the CIDR block of the queue.
- - Log in to the VPC console and find the VPC the data source belongs to.
- On the network console, choose Virtual Private Cloud > Network Interfaces. On the Network Interfaces tab page displayed, search for the security group name, click More in the Operation column, and select Change Security Group.
- In the navigation pane on the left, choose Access Control > Security Groups.
- Click the name of the security group to which the external data source belongs.
- Click the Inbound Rules tab and add a rule to allow access from the CIDR block of the queue.
Configure the inbound rule parameters according to Table 2.
+ - Log in to the VPC console and find the VPC the data source belongs to.
- On the network console, choose Virtual Private Cloud > Network Interfaces. On the Network Interfaces tab page displayed, search for the security group name, click More in the Operation column, and select Change Security Group.
- In the navigation pane on the left, choose Access Control > Security Groups.
- Click the name of the security group to which the external data source belongs.
- Click the Inbound Rules tab and add a rule to allow access from the CIDR block of the queue.
Configure the inbound rule parameters according to Table 3.
-Table 2 Inbound rule parameters Parameter
+Table 3 Inbound rule parameters Parameter
Description
diff --git a/docs/dli/umn/dli_01_0007.html b/docs/dli/umn/dli_01_0007.html index 263204d85..bbf784e8b 100644 --- a/docs/dli/umn/dli_01_0007.html +++ b/docs/dli/umn/dli_01_0007.html @@ -1,6 +1,6 @@ -Previewing Data
+Previewing Table Data on the DLI Console
The Preview page displays the first 10 records in the table.
Procedure
You can preview data on either the Data Management page or the SQL Editor page.- To preview data on the Data Management page:
- On the left of the management console, choose Data Management > Databases and Tables.
- On the displayed Data Management page, click the name of the database where the target table whose data you want to export resides to switch to the Manage Tables page.
- Click More in the Operation column of the target table and select View Properties.
- Click the Preview tab to preview the table data.
- To preview data on the SQL Editor page:
- On the left of the management console, click SQL Editor.
- In the navigation pane of the displayed SQL Editor page, click Databases.
- Click the corresponding database name to view the tables in the database.
- Click
on the right of the corresponding table, choose View Properties from the list menu, and click the Preview tab to preview the data of the table.
diff --git a/docs/dli/umn/dli_01_0008.html b/docs/dli/umn/dli_01_0008.html index 21f5410a6..127a8e145 100644 --- a/docs/dli/umn/dli_01_0008.html +++ b/docs/dli/umn/dli_01_0008.html @@ -1,6 +1,6 @@ --Parent topic: Databases and Tables+Parent topic: Managing Table Resources on the DLI ConsoleViewing Metadata
+Viewing Table Metadata
Metadata Description
- Metadata is used to define data types. It describes information about the data, including the source, size, format, and other data features. In database fields, metadata interprets data content in the data warehouse.
- When you create a table, metadata is defined, consisting of the column name, type, and description.
- The Metadata page displays information about the target table, including Column Name, Column Type, Data Type, and Description.
Procedure
You can view metadata on either the Data Management page or the SQL Editor page.- To view metadata on the Data Management page:
- On the left of the management console, choose Data Management > Databases and Tables.
- On the displayed Data Management page, click the name of the database where the target table whose data you want to export resides to switch to the Manage Tables page.
- Click More in the Operation column of the target table and select View Properties. In the Metadata tab, view the metadata of the table.
diff --git a/docs/dli/umn/dli_01_0010.html b/docs/dli/umn/dli_01_0010.html index 9a3e62870..b64a19cf7 100644 --- a/docs/dli/umn/dli_01_0010.html +++ b/docs/dli/umn/dli_01_0010.html @@ -1,6 +1,6 @@ --Parent topic: Databases and Tables+Parent topic: Creating Databases and TablesExporting Data from DLI to OBS
+Exporting DLI Table Data to OBS
You can export data from a DLI table to OBS. During the export, a folder is created in OBS or the content in the existing folder is overwritten.
@@ -66,7 +66,7 @@Precautions
- The exported file can be in JSON format, and the text format can only be UTF-8.
- Only the data in the DLI table (the table type is Managed) can be exported to the OBS bucket, and the export path must be specified to the folder level.
- Data can be exported across accounts. That is, after account B authorizes account A, account A can export data to the OBS path of account B if account A has the permission to read the metadata and permission information about the OBS bucket of account B and read and write the path.
diff --git a/docs/dli/umn/dli_01_0011.html b/docs/dli/umn/dli_01_0011.html index 87400259d..9edb98166 100644 --- a/docs/dli/umn/dli_01_0011.html +++ b/docs/dli/umn/dli_01_0011.html @@ -1,16 +1,11 @@ --Parent topic: Databases and Tables+Parent topic: Managing Table Resources on the DLI ConsoleDeleting a Database or a Table
-You can delete unnecessary databases and tables based on actual conditions.
-Precautions
- You are not allowed to delete databases or tables that are being used for running jobs.
- The administrator, database owner, and users with the database deletion permission can delete the database. The administrator, database owner, and users with the table deletion permission can delete the table.
If a database or table is deleted, it cannot be recovered. Exercise caution when performing this operation.
+Deleting a Database on the DLI Console
+You can delete databases as needed.
+-Precautions
- You are not allowed to delete databases or tables that are being used for running jobs.
- The administrator, database owner, and users with the database deletion permission can delete the database.
If a database or table is deleted, it cannot be recovered. Exercise caution when performing this operation.
Deleting a Table
You can delete a table on either the Data Management page or the SQL Editor page.-- Delete the table on the Data Management page.
- On the left of the management console, choose Data Management > Databases and Tables.
- Locate the row where the database whose tables you want to delete, click the database name to switch to the Table Management page.
- Locate the row where the target table locates and click More > Delete in the Operation column.
- In the displayed dialog box, click Yes.
- Delete a table on the SQL Editor page.
- On the top menu bar of the DLI management console, click SQL Editor.
- In the navigation tree on the left, click Databases. Click the name of a database where the table you want belongs. The tables of the selected database are displayed.
- Click
on the right of the table and choose Delete from the shortcut menu. - In the dialog box that is displayed, click OK.
Deleting a Database
- On the left of the management console, choose Data Management > Databases and Tables.
- Locate the row where the target database locates and click More > Drop Database in the Operation column.
You cannot delete databases that contain tables. To delete a database containing tables, delete the tables first.
- In the displayed dialog box, click Yes.
diff --git a/docs/dli/umn/dli_01_0012.html b/docs/dli/umn/dli_01_0012.html deleted file mode 100644 index 88f5d69f0..000000000 --- a/docs/dli/umn/dli_01_0012.html +++ /dev/null @@ -1,33 +0,0 @@ - - --Parent topic: Databases and Tables+Parent topic: Managing Database Resources on the DLI ConsoleQueue Management
- --- diff --git a/docs/dli/umn/dli_01_0013.html b/docs/dli/umn/dli_01_0013.html index c2d3396bc..dad68476b 100644 --- a/docs/dli/umn/dli_01_0013.html +++ b/docs/dli/umn/dli_01_0013.html @@ -25,7 +25,7 @@-
-
- Overview
-
- - Queue Permission Management
-
- - Creating a Queue
-
- - Deleting a Queue
-
- - Allocating a Queue to an Enterprise Project
-
- - Modifying the CIDR Block
-
- - Elastic Scaling of Queues
-
- - Scheduling CU Changes
-
- - Testing Address Connectivity
-
- - Creating an SMN Topic
-
- - Managing Queue Tags
-
- - Setting Queue Properties
-
-
diff --git a/docs/dli/umn/dli_01_0014.html b/docs/dli/umn/dli_01_0014.html index 649c1fc17..c3b0062ed 100644 --- a/docs/dli/umn/dli_01_0014.html +++ b/docs/dli/umn/dli_01_0014.html @@ -1,6 +1,6 @@ --Parent topic: Enhanced Datasource Connections+Parent topic: Managing Enhanced Datasource ConnectionsAdding a Route
+Adding a Route for an Enhanced Datasource Connection
Scenario
A route is configured with the destination, next hop type, and next hop to determine where the network traffic is directed. Routes are classified into system routes and custom routes.
After an enhanced connection is created, the subnet is automatically associated with the default route. You can add custom routes as needed to forward traffic destined for the destination to the specified next hop.
- When an enhanced connection is created, the associated route table is the one associated with the subnet of the data source.
- The route to be added in the Add Route dialog box must be one in the route table associated with the subnet of the resource pool.
- The subnet of the data source must be different from that used by the resource pool. Otherwise, a network segment conflict occurs.
diff --git a/docs/dli/umn/dli_01_0015.html b/docs/dli/umn/dli_01_0015.html index 7c343f561..9c512a3e9 100644 --- a/docs/dli/umn/dli_01_0015.html +++ b/docs/dli/umn/dli_01_0015.html @@ -1,27 +1,31 @@-Parent topic: Enhanced Datasource Connections+Parent topic: Managing Enhanced Datasource ConnectionsQueue Permission Management
-Scenario
- You can isolate queues allocated to different users by setting permissions to ensure data query performance.
- The administrator and queue owner have all permissions, which cannot be set or modified by other users.
Administrators and queue owners have full operation permissions on queues. They can grant operation permissions to other users based on service needs. This ensures that users can execute their jobs independently without any impact on the performance of other users' job execution. This section describes how to manage queue permissions.
+-Operation Precautions
- The administrator and queue owner have all permissions, which cannot be set or modified by other users.
- When setting queue permissions for a new user, ensure that the region of the user group to which the user belongs has the Tenant Guest permission. + + +
Operations
- On the top menu bar of the DLI management console, click Resources > Queue Management.
- Select the queue to be configured and choose Manage Permissions in the Operation column. The User Permission Info area displays the list of users who have permissions on the queue.
You can assign queue permissions to new users, modify permissions for users who have some permissions of a queue, and revoke all permissions of a user on a queue.
-- Assign permissions to a new user.A new user does not have permissions on the queue.
- Click Set Permission on the right of User Permissions page. The Set Permission dialog box is displayed.
- Specify Username and select corresponding permissions.
- Click OK.
Table 1 describes the related parameters.
+Operations
- On the top menu bar of the DLI management console, click Resources > Queue Management.
- Select the queue to be configured and choose Manage Permissions in the Operation column. The User Permission Info area displays the list of users who have permissions on the queue.
You can grant queue permissions to new users, modify permissions for users who have some permissions on a queue, and revoke all permissions for a user on a queue.
+- Grant permissions to a new user.A new user does not have permissions on the queue.
- Click Set Permission on the right of User Permissions page. The Set Permission dialog box is displayed.
- Specify Username and select corresponding permissions.
- Click OK.
Table 1 describes the related parameters.
-Table 1 Parameter description Parameter
+Table 1 Parameter description Parameter
Description
+Description
Username
+Username
Name of the authorized user.
+Name of the authorized user.
NOTE:The username is an existing IAM user name and has logged in to the DLI management console.
Permission Settings
+Permission Settings
- Delete Queues: This permission allows you to delete the queue.
- Submit Jobs: This permission allows you to submit jobs using this queue.
- Terminate Jobs: This permission allows you to terminate jobs submitted using this queue.
- Grant Permission: This permission allows you to grant queue permissions to other users.
- Revoke Permission: This permission allows you to revoke the queue permissions that other users have but cannot revoke the queue owner's permissions.
- View Other User's Permissions: This permission allows you to view the queue permissions of other users.
- Restart Queues: This permission allows you to restart queues.
- Modify Queue Specifications: This permission allows you to modify queue specifications.
- Delete Queues: This permission allows you to delete the queue.
- Submit Jobs: This permission allows you to submit jobs using this queue.
- Terminate Jobs: This permission allows you to terminate jobs submitted using this queue.
- Grant Permission: This permission allows you to grant queue permissions to other users.
- Revoke Permission: This permission allows you to revoke the queue permissions that other users have but cannot revoke the queue owner's permissions.
- View Other User's Permissions: This permission allows you to view the queue permissions of other users.
- Restart Queues: This permission allows you to restart queues.
- Modify Queue Specifications: This permission allows you to modify queue specifications.
- To assign or revoke permissions of a user who has some permissions on the queue, perform the following steps:
- In the list under User Permission Info for a queue, select the user whose permissions need to be modified and click Set Permission in the Operation column.
- In the displayed Set Permission dialog box, modify the permissions of the current user. Table 1 lists the detailed permission descriptions.
If all options under Set Permission are gray, you are not allowed to change permissions on this queue. You can apply to the administrator, queue owner, or other authorized users for queue permission granting and revoking.
+ - To grant or revoke permissions for a user who already has certain permissions on a queue, perform the following steps:
- In the list under User Permission Info for a queue, select the user whose permissions need to be modified and click Set Permission in the Operation column.
- In the displayed Set Permission dialog box, modify the permissions of the current user. Table 1 lists the detailed permission descriptions.
If all options under Set Permission are gray, you are not allowed to change permissions on this queue. You can apply to the administrator, queue owner, or other authorized users for queue permission granting and revoking.
- Click OK.
- To revoke all permission of a user on the queue, perform the following steps:
In the user list under Permission Info, select the user whose permission needs to be revoked and click Revoke Permission under Operation. In the Revoke Permission dialog box, click OK. All permissions on this queue are revoked.
+ - To revoke all permissions for a user on a queue, perform the following steps:
In the user list under Permission Info, select the user whose permission needs to be revoked and click Revoke Permission under Operation. In the Revoke Permission dialog box, click OK. All permissions on this queue are revoked.
diff --git a/docs/dli/umn/dli_01_0016.html b/docs/dli/umn/dli_01_0016.html index f99b73a52..fadc18c45 100644 --- a/docs/dli/umn/dli_01_0016.html +++ b/docs/dli/umn/dli_01_0016.html @@ -4,14 +4,14 @@-Parent topic: Queue Management+Parent topic: Managing QueuesYou can delete a queue based on actual conditions.
- - This operation will fail if there are jobs in the Submitting or Running state on this queue.
- Deleting a queue does not cause table data loss in your database.
Procedure
- On the left of the DLI management console, click Resources >Queue Management.
- Locate the row where the target queue locates and click Delete in the Operation column.
If Delete in the Operation column is gray, the current user does not have the permission of deleting the queue. You can apply to the administrator for the permission.
+Procedure
- In the navigation pane on the left of the DLI management console, choose Resources > Queue Management.
- Locate the row where the target queue locates and click Delete in the Operation column.
If Delete in the Operation column is gray, the current user does not have the permission of deleting the queue. You can apply to the administrator for the permission.
- In the displayed dialog box, click OK.
diff --git a/docs/dli/umn/dli_01_0017.html b/docs/dli/umn/dli_01_0017.html index fe1a9b054..f609b07a3 100644 --- a/docs/dli/umn/dli_01_0017.html +++ b/docs/dli/umn/dli_01_0017.html @@ -1,59 +1,56 @@ --Parent topic: Queue Management+Parent topic: Managing QueuesSQL Job Management
-SQL jobs allow you to execute SQL statements in the SQL job editing window, import data, and export data.
-SQL job management provides the following functions:
-- Searching for Jobs: Search for jobs that meet the search criteria.
- Viewing Job Details: Display job details.
- Terminating a Job: Stop a job in the Submitting or Running status.
- Exporting Query Results: A maximum of 1000 records can be displayed in the query result on the console. To view more or all data, you can export the data to OBS.
SQL Jobs Page
On the Overview page of the DLI console, click SQL Jobs to go to the SQL job management page. Alternatively, you can click Job Management > SQL Jobs. The job list displays all SQL jobs. If there are a large number of jobs, they will be displayed on multiple pages. You can switch to the specified page as needed. DLI allows you to view jobs in all statuses. By default, jobs in the job list are displayed in descending order of the job creation time.
+Managing SQL Jobs
+Viewing Basic Job Information on the SQL Jobs Page
The SQL Jobs page displays all SQL jobs, which may span multiple pages if there are many jobs. You can navigate to a specific page as needed. DLI allows you to view jobs in all statuses. Jobs in the job list are sorted by creation time in descending order by default.
-Table 1 SQL Job management parameters Parameter
+-Table 1 SQL Job management parameters Parameter
Description
+Description
Queues
+Queues
Name of the queue to which a job belongs
+Name of the queue to which a job belongs
Username
+Username
Name of the user who executed the job.
+Name of the user who executed the job.
Type
+Type
Job type. The following types are supported:
+Job type. The following types are supported:
- IMPORT: A job that imports data to DLI
- EXPORT: A job that exports data from DLI
- DCL: Conventional DCLs and operations related to queue permissions
- DDL:Conventional DDLs, including creating and deleting databases and tables
- QUERY: A job that queries data by running SQL statements
- INSERT: A job that inserts data by running SQL statements
- UPDATE: A job that updates data.
- DELETE: A job that deletes a SQL job.
- DATA_MIGRATION: A job that migrates data.
- RESTART_QUEUE: A job that restarts a queue.
- SCALE_QUEUE: A job that changes queue specifications, including sale-out and scale-in.
Status
+Status
Job status. Possible values are as follows:
+Job status. Possible values are as follows:
- Submitting
- Running
- Finished
- Canceled
- Failed
- Scaling
Query
+Query
SQL statements for operations such as exporting and creating tables
+SQL statements for operations such as exporting and creating tables
You can click
to copy the query statement.Duration
+Duration
Running duration of a job
+Running duration of a job
Created
+Created
Time when a job is created. Jobs can be displayed in ascending or descending order of the job creation time.
+Time when a job is created. Jobs can be displayed in ascending or descending order of the job creation time.
Operation
+Operation
- Edit: Edit the job.
- Cancel
- You can terminate a job only when the job is in Submitting or Running status.
- A job whose status is Finished, Failed, or Canceled cannot be terminated.
- If the Cancel button is gray, you are not allowed to perform this operation.
- Re-execute: Execute the job again.
- SparkUI: Display the Spark job execution page.NOTE:
- When you execute a job on a created queue, the cluster is restarted. It takes about 10 minutes. If you click SparkUI before the cluster is created, an empty projectID will be cached. The SparkUI page cannot be displayed. You are advised to use a dedicated queue so that the cluster will not be released. Alternatively, wait for a while after the job is submitted (the cluster is created), and then check SparkUI.
- Currently, only the latest 100 job information records are displayed on the SparkUI of DLI.
- Edit: Edit the job.
- Cancel
- You can terminate a job only when the job is in Submitting or Running status.
- A job whose status is Finished, Failed, or Canceled cannot be terminated.
- If the Cancel button is gray, you are not allowed to perform this operation.
- Re-execute: Execute the job again.
- SparkUI: Display the Spark job execution page.NOTE:
- When you execute a job on a created queue, the cluster is restarted. It takes about 10 minutes. If you click SparkUI before the cluster is created, an empty projectID will be cached. The SparkUI page cannot be displayed. You are advised to use a dedicated queue so that the cluster will not be released. Alternatively, wait for a while after the job is submitted (the cluster is created), and then check SparkUI.
- SparkUI can currently only display the latest 100 jobs.
- In addition to the preceding operations, the following operations are available for QUERY jobs and asynchronous DDL jobs.
- View Result: View the job running result.
- Export Result: Export the job running result to the created OBS bucket.
-Searching for a Job
On the SQL Jobs page, you can search jobs with any of the following operations.
-- Select a queue name.
- Set the date range.
- Enter a username, statement, or job ID.
- Select the creation time in ascending or descending order.
- Select a job type.
- Select a job status.
- Select the job execution duration in ascending or descending order.
Viewing Job Details
On the SQL Jobs page, you can click
+
in front of a job record to view details about the job.-Viewing Job Details
On the SQL Jobs page, you can click
in front of a job record to view details about the job.Job details vary with job types. The job details vary depending on the job types, status, and configuration options. The following describes how to load data, create a table, and select a job. For details about other job types, see the information on the management console.
- Load data (job type: IMPORT) include the following information: queue, job ID, username, type, status, execution statement, running duration, creation time, end time, parameter settings, label, number of results, scanned data, number of scanned data, number of error records, storage path, data format, database, table, table header, separator, reference character, escape character, date format, timestamp format, total CPU used, and output bytes.
- Create table (job type: DDL) include the following information: queue, job ID, username, type, status, execution statement, running duration, creation time, end time, parameter settings, tags, number of results, scanned data, and database.
- Select (job type: QUERY) include the following information: queue, job ID, username, type, status, execution statement, running duration, creation time, end time, parameter setting, label, number of results (results of successful executions can be exported), and scanned data, username, result status (results of successful tasks can be viewed. Failure causes of failed tasks are displayed), database, total CPU used, and output bytes.
- Total CPU Used (Core x ms): total CPU used during job execution.
- Output Bytes: number of output bytes after the job is executed.
-Terminating a Job
On the SQL Jobs page, you can click Terminate in the Operation column to stop a submitting or running job.
-Exporting Query Results
A maximum of 1000 records can be displayed in the query result on the console. To view more or all data, you can export the data to OBS. The procedure is as follows:
-You can export results on the SQL Jobs page or the SQL Editor page.
-- On the Job Management > SQL Jobs page, you can click More > Export Result in the Operation column to export the query result.
- After the query statements are successfully executed on the SQL Editor page, click
next to the View Result tab page to export query results.
- - - If there are no numerical columns in the query results, job results cannot be exported.
-
+Table 2 Parameters Parameter
-Description
-Data Format
-Format of the exported query result file. This parameter can be set to json or csv.
-Queues
-The queue where the jobs are executed. SQL jobs can only be executed on SQL queues. For details about how to create a queue, see Creating a Queue.
-Compression Format
-Compression format of the data to be exported. The options are:
-- none
- bzip2
- deflate
- gzip
Storage Path
-OBS path to store the result.
-NOTE:-- After selecting an OBS bucket, enter a name for the folder. If the folder does not exist, it will be created in OBS.
- The folder name cannot contain the special characters of \ / : * ? "< > |, and cannot start or end with a dot (.).
Export Mode
-Mode for saving the exported query result data.
-- New OBS directory: If the specified export directory exists, an error is reported and the export operation cannot be performed.
- Existing OBS directory (Overwritten): If you create a file in the specified directory, the existing file will be overwritten.
Number of Results
-Number of exported query results. If no value is entered or the value is 0, all results are exported.
-Table Header
-Whether the data to be exported contains table headers.
-+Searching for a Job
On the SQL Jobs page, you can search jobs with any of the following operations.
+- Select a queue name.
- Set the date range.
- Enter a username, statement, or job ID.
- Select the creation time in ascending or descending order.
- Select a job type.
- Select a job status.
- Select the job execution duration in ascending or descending order.
Terminating a SQL Job
On the SQL Jobs page, you can click Terminate in the Operation column to stop a submitting or running job.
diff --git a/docs/dli/umn/dli_01_0018.html b/docs/dli/umn/dli_01_0018.html index 9ca6ac814..853fd3bfa 100644 --- a/docs/dli/umn/dli_01_0018.html +++ b/docs/dli/umn/dli_01_0018.html @@ -20,7 +20,7 @@-Parent topic: Job Management+Parent topic: Submitting a SQL Job Using DLIdiff --git a/docs/dli/umn/dli_01_0019.html b/docs/dli/umn/dli_01_0019.html index cf8a514c2..a0b962d90 100644 --- a/docs/dli/umn/dli_01_0019.html +++ b/docs/dli/umn/dli_01_0019.html @@ -22,7 +22,7 @@-Parent topic: Enhanced Datasource Connections+Parent topic: Managing Enhanced Datasource Connections- Click the text box and select a predefined tag key from the drop-down list.
To add a predefined tag, you need to create one on TMS and then select it from the Tag key drop-down list. You can click View predefined tags to go to the Predefined Tags page of the TMS console. Then, click Create Tag in the upper corner of the page to create a predefined tag.
- Enter a tag key in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
+- Enter a tag key in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
Tag value
You can perform the following operations:
-- Click the text box and select a predefined tag value from the drop-down list.
- Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed. The value cannot start or end with a space.
+- Click the text box and select a predefined tag value from the drop-down list.
- Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed.
diff --git a/docs/dli/umn/dli_01_0020.html b/docs/dli/umn/dli_01_0020.html new file mode 100644 index 000000000..03c857a7f --- /dev/null +++ b/docs/dli/umn/dli_01_0020.html @@ -0,0 +1,441 @@ + + +-Parent topic: Enhanced Datasource Connections+Parent topic: Managing Enhanced Datasource ConnectionsUsing DLI to Submit a SQL Job to Query RDS for MySQL Data
+++Scenario
DLI can query data stored in RDS. This section describes how to use DLI to submit a SQL job to query RDS for MySQL data.
+In this example, we will create an RDS for MySQL DB instance, create a database and table, create a DLI elastic resource pool and add a queue to it, create an enhanced datasource connection to connect the DLI elastic resource pool and the RDS for MySQL DB instance, and submit a SQL job to access the data in the RDS table.
++Procedure
Table 1 shows the process for submitting a SQL job to query RDS for MySQL data.
+ ++
+Table 1 Procedure for using DLI to submit a SQL job to query RDS for MySQL data Procedure
+Description
++ +Create an RDS for MySQL instance for the example scenario.
++ +Log in to the RDS for MySQL DB instance and create a database and a table.
+Step 3: Create an Elastic Resource Pool and Add Queues to the Pool
+Create compute resources required for submitting jobs.
++ +Create an enhanced datasource connection to connect the DLI elastic resource pool and the RDS for MySQL DB instance.
++ +Create a datasource authentication to store the access credentials required by DLI to read from and write to RDS for MySQL data.
++ +Use standard SQL statements to query and analyze data.
++Step 1: Create an RDS MySQL Instance
In this example, assuming the job name is JobSample, RDS is used as the data source to create an RDS for MySQL DB instance.
+For details, see "RDS for MySQL Getting Started" in the Relational Database Service Getting Started.
+- Log in to the RDS management console.
- Select a region and a project in the upper left corner.
- In the navigation pane on the left, choose Instances. On the displayed page, click Buy DB Instance in the upper right corner.
- On the Buy DB Instance page, set instance parameters, and click Buy.
Set the parameters listed below based on your service planning.
+ ++
+Table 2 RDS for MySQL instance parameters Parameter
+Description
+Example Value
+Region
+Region where you want to create the instance
+_
+DB Instance Name
+Instance name
+rds-demo
+DB Engine
+MySQL
+MySQL
+DB Engine Version
+If you select MySQL for DB Engine, select an engine version that best suits your service needs. You are advised to select the latest available version for more stable performance, higher security, and greater reliability.
+8.0
+DB Instance Type
+Primary/standby mode of the instance
+Single
+Storage Type
+Determines the instance read/write speed. The higher the maximum throughput, the faster the read and write speeds.
+Cloud SSD
+AZ
+For a single DB instance, you only need to select a single AZ.
+-
+Time Zone
+Select a time zone based on the region you selected. You can change it after the DB instance is created.
+Retain the default value.
+Instance Class
+vCPUs and memory. These instance classes support varying number of connections and maximum IOPS.
+2 vCPUs | 4 GB
+Storage Space
+If the storage type is cloud SSD or extreme SSD, you can enable storage autoscaling. If the available storage drops to a specified threshold, autoscaling is triggered.
+40 GB
+Disk Encryption
+Determine whether to enable disk encryption.
+Disable
+VPC
+Select an existing VPC.
+For how to recreate a VPC and subnet, refer to "Creating a VPC and Subnet" in the Virtual Private Cloud User Guide.
+NOTE:+In datasource scenarios, the CIDR block of the data source cannot overlap that of the elastic resource pool.
+-
+Database Port
+Port 3306 is used by default.
+3306
+Security Group
+Enhances security by providing rules that control access to RDS from other services.
+The security group where the data source is must allow access from the CIDR block of the DLI elastic resource pool.
+-
+Password
+Set a password for logging in to the DB instance.
+-
+Administrator
+root
+root
+Administrator Password
+Administrator password
+-
+Parameter Template
+A template of parameters for creating an instance. The template contains engine configuration values that are applied to one or more instances.
+Default-MySQL-5.7
+Table Name
+Determines whether the table name is case-insensitive.
+Case insensitive
+Enterprise Project
+If the instance has been associated with an enterprise project, select the target project from the Enterprise Project drop-down list.
+default
+Quantity
+Number of instances to buy
+1
+ - Click Next. The confirmation page is displayed.
- Click Submit.
- To view and manage the DB instance, go to the Instance Management page.
During the creation process, the DB instance status is Creating. When the creation process is complete, the instance status will change to Available. You can view the detailed progress and result of the task on the Task Center page.
+
+Step 2: Create an RDS Database Table
- Log in to the RDS management console.
- In the upper left corner of the management console, select the target region and project.
- On the Instances page, locate the DB instance you just created and record its floating IP address.
- Locate the RDS for MySQL DB instance you created, click More in the Operation column, and select Log In. On the displayed page, enter the username and password for logging in to the instance and click Test Connection. After Connection is successful is displayed, click Log In.
- Click Create Database. In the displayed dialog box, enter database name dli_demo. Then, click OK.
- Click SQL Query and run the following SQL statement to create a table:
CREATE TABLE `dli_demo`.`tabletest` ( + `id` VARCHAR(32) NOT NULL, + `name` VARCHAR(32) NOT NULL, + PRIMARY KEY (`id`) +) ENGINE = InnoDB + DEFAULT CHARACTER SET = utf8mb4;
+
+Step 3: Create an Elastic Resource Pool and Add Queues to the Pool
To execute SQL jobs in datasource scenarios, you must use your own SQL queue as the existing default queue cannot be used. In this example, create an elastic resource pool named dli_resource_pool and a queue named dli_queue_01.
+- Log in to the DLI management console.
- In the navigation pane on the left, choose Resources > Resource Pool.
- On the displayed page, click Buy Resource Pool in the upper right corner.
- On the displayed page, set the parameters.Table 3 describes the parameters. +++
+Table 3 Parameters Parameter
+Description
+Example Value
+Region
+Select a region where you want to buy the elastic resource pool.
+_
+Project
+Project uniquely preset by the system for each region
+Default
+Name
+Name of the elastic resource pool
+dli_resource_pool
+Specifications
+Specifications of the elastic resource pool
+Standard
+CU Range
+The maximum and minimum CUs allowed for the elastic resource pool
+64-64
+CIDR Block
+CIDR block the elastic resource pool belongs to. If you use an enhanced datasource connection, this CIDR block cannot overlap that of the data source. Once set, this CIDR block cannot be changed.
+172.16.0.0/19
+Enterprise Project
+Select an enterprise project for the elastic resource pool.
+default
+ - Click Buy.
- Click Submit.
- In the elastic resource pool list, locate the pool you just created and click Add Queue in the Operation column.
- Set the basic parameters listed below.
++
+Table 4 Basic parameters for adding a queue Parameter
+Description
+Example Value
+Name
+Name of the queue to add
+dli_queue_01
+Type
+Type of the queue
+- To execute SQL jobs, select For SQL.
- To execute Flink or Spark jobs, select For general purpose.
_
+Enterprise Project
+Select an enterprise project.
+default
+ - Click Next and configure scaling policies for the queue.
Click Create to add a scaling policy with varying priority, period, minimum CUs, and maximum CUs.
+ + ++
+Table 5 Scaling policy parameters Parameter
+Description
+Example Value
+Priority
+Priority of the scaling policy in the current elastic resource pool. A larger value indicates a higher priority. In this example, only one scaling policy is configured, so its priority is set to 1 by default.
+1
+Period
+The first scaling policy is the default policy, and its Period parameter configuration cannot be deleted or modified.
+The period for the scaling policy is from 00 to 24.
+00–24
+Min CU
+Minimum number of CUs allowed by the scaling policy
+16
+Max CU
+Maximum number of CUs allowed by the scaling policy
+64
+ - Click OK.
+Step 4: Create an Enhanced Datasource Connection
- Create a rule on the security group of the RDS DB instance to allow access from the CIDR block of the DLI queue.
- Go to the RDS management console and choose Instances in the navigation pane on the left. In the instance list, click the name of the RDS for MySQL DB instance you created to access its basic information page.
- In the navigation pane on the left, choose Connectivity & Security. In the Security Group Rules area, find the Inbound Rules tab and click Add Inbound Rule.
For example, if the CIDR block of the queue is 172.16.0.0/19, add the rule as follows:
+- Type: Select IPv4.
- Protocol & Port: Select Protocols/TCP (Custom) as the protocol and leave the port number blank.
- Source: Select IP Address and enter 172.16.0.0/19.
Click OK. The security group rule is added.
+
- Create an enhanced datasource connection between RDS for MySQL and DLI.+
The CIDR block of the elastic resource pool bound to a datasource connection cannot overlap that of the data source.
+- Go to the DLI management console and choose Datasource Connections in the navigation pane on the left.
- On the displayed Enhanced tab, click Create. Set the following parameters:
- Connection Name: dlirds
- Resource Pool: Select the elastic resource pool created in Step 3: Create an Elastic Resource Pool and Add Queues to the Pool.
- VPC: Select the VPC where the RDS for MySQL DB instance is, that is, the VPC selected in Step 2: Create an RDS Database Table.
- Subnet: Select the subnet where the RDS for MySQL DB instance is, that is, the subnet selected in Step 2: Create an RDS Database Table.
To check the subnet information, go to the RDS console, choose Instances in the navigation pane on the left, click the name of the RDS for MySQL DB instance you created, and find Subnet in the Connection Information area on the displayed page.
+
- Click OK.
- In the Enhanced tab, click the created connection dlirds to view its VPC Peering ID and Connection Status. If the connection status is Active, the connection is successful.
- Test if the queue can connect to the RDS for MySQL DB instance.
- In the navigation pane on the left, choose Resources > Queue Management. On the displayed page, locate the queue added in Step 3: Create an Elastic Resource Pool and Add Queues to the Pool, click More in the Operation column, and select Test Address Connectivity.
- Enter the floating IP address of the RDS for MySQL DB instance recorded in Step 2: Create an RDS Database Table.+
On the Instance Management page, click the target DB instance. On the displayed page, choose Connection Information > Floating IP Address to obtain the floating IP address.
+- If the IP address is reachable, the DLI queue can connect to the RDS for MySQL DB instance through the created enhanced datasource connection.
- If the IP address is unreachable, check if the security group where the data source is allows access from the IP address of the elastic resource pool. Once the fault is rectified, retest the network connectivity.
+Step 5: Create a Datasource Authentication
When analyzing across multiple sources, you are advised not to configure authentication information directly in a job as it can lead to password leakage. Instead, you are advised to use datasource authentication provided by DLI to securely store data source authentication information.
+To connect a Spark SQL job to an RDS for MySQL data source, you can create a password-type datasource authentication.
+- Log in to the DLI management console.
- In the navigation pane on the left, choose Datasource Connections. On the displayed page, click Datasource Authentication.
- Click Create.
Set authentication parameters based on Table 6.
+ ++
+Table 6 Datasource authentication parameters Parameter
+Description
+Type
+Select Password.
+Authentication Certificate
+Name of the datasource authentication to create
+- Only numbers, letters, and underscores (_) are allowed. The name cannot contain only numbers or start with an underscore (_).
- The value can contain a maximum of 128 characters.
Username
+Username for logging in to the RDS for MySQL DB instance
+Password
+Password for logging in to the RDS for MySQL DB instance
+
+Step 6: Submit a SQL Job
In this example, a SQL job accesses an RDS table using a datasource connection.
+- On the DLI management console, choose SQL Editor in the navigation pane on the left.
- In the editing window on the right of the SQL Editor page, enter the following SQL statement to create database db1 and click Execute.
create database db1;
+ - On the top of the editing window, choose the dli_queue_01 queue and the db1 database. Enter the following SQL statements to create a table, insert data to the table, and query the data. Click Execute.View the query result to verify that the query is successful and the datasource connection works.+
CREATE TABLE IF NOT EXISTS rds_test USING JDBC OPTIONS ( +'url' = 'jdbc:mysql://{{ip}}:{{port}}', // Private IP address and port of RDS + 'driver' = 'com.mysql.jdbc.Driver', +'dbtable' = 'dli_demo.tabletest', // Name of the created DB instance and table name + 'passwdauth'="xxxxx" // Name of the datasource authentication of the password type created on DLI. If datasource authentication is used, you do not need to set the username and password for the job. +) + +insert into rds_test VALUES ('123','abc'); + + +SELECT * from rds_test;+
++ diff --git a/docs/dli/umn/dli_01_0021.html b/docs/dli/umn/dli_01_0021.html index 17ca1a768..7309d5c9e 100644 --- a/docs/dli/umn/dli_01_0021.html +++ b/docs/dli/umn/dli_01_0021.html @@ -1,139 +1,68 @@ -++Parent topic: Getting Started+Managing SQL Templates
+Creating a SQL Job Template
To facilitate SQL operation execution, DLI allows you to customize query templates or save the SQL statements in use as templates. After templates are saved, you do not need to compile SQL statements. Instead, you can directly perform the SQL operations using the templates.
-SQL templates include sample templates and custom templates. The default sample template contains 22 standard TPC-H query statements, which can meet most TPC-H test requirements. For details, see TPC-H Sample Data in the SQL Template.
-SQL template management provides the following functions:
-- Sample Templates
- Custom Templates
- Creating a Template
- Executing the Template
- Searching for a Template
- Modifying a Template
- Deleting a Template
-Table Settings
In the upper right corner of the SQL Template page, click Set Property to determine whether to display templates by group.
-If you select Display by Group, the following display modes are available:
-- Expand the first group
- Expand all
- Collapse All
Sample Templates
The current sample template contains 22 standard TPC-H query statements. You can view the template name, description, and statements. For details about TPC-H examples, see TPC-H Sample Data in the SQL Template.
+SQL templates include sample templates and custom templates. The default sample template contains 22 standard TPC-H query statements, which can meet most TPC-H test requirements. For details, see TPC-H Sample Data in the SQL Templates Preset on DLI.
++ In the navigation pane on the left, choose Job Templates > SQL Templates. In the upper right corner of the displayed page, click Settings. In the displayed Settings dialog box, choose whether to display templates by group.
+If you enable Display by Group, the display options are Expand the first group, Expand all, and Collapse all.
+Creating a SQL Job Template
You can create a template on either the Job Templates or the SQL Editor page.
+- To create a template on the Job Templates page:
- On the left of the management console, choose Job Templates > SQL Templates.
- On the SQL Templates page, click Create Template to create a template.
Enter the template name, SQL statement, and description information. For details, see Table 1.
-Table 1 Template management parameters Parameter
+-
-Table 1 Parameter description Parameter
Description
+Description
Name
+Name
Indicates the template name.
-- A template name can contain only digits, letters, and underscores (_), but cannot start with an underscore (_) or contain only digits. It cannot be left empty.
- The template name can contain a maximum of 50 characters.
Description
-Description of the template you create.
-Statement
-SQL statement created as the template.
-Operation
-Execute: After you click this button, the system switches to the SQL Editor page, where you can modify or directly perform the statement as required. For details, see Executing the Template.
-The existing sample templates apply to the following scenarios:
-- Price summary report query
- Minimum cost supplier analysis
- Shipping priority analysis
- Analysis of order priority check
- Analysis of the number of local suppliers
- Analysis of forecasted income changes
- Freight volume analysis
- National market share analysis
- Profit estimation analysis by product type
- Analysis of returned parts
- Analysis of key inventory indicators
- Freight mode and command priority analysis
- Consumer allocation analysis
- Promotion effect analysis
- Analysis of the supplier with the largest contribution
- Analysis of the relationship between parts and suppliers
- Revenue analysis of small-lot orders
- Customer analysis for large orders
- Discounted revenue analysis
- Analysis of potential component improvements
- Analysis of suppliers who fail to deliver goods on time
- Global sales opportunity analysis
-Custom Templates
The custom template list displays all templates you have created. You can view the template name, description, statements, and more.
- --
-Table 2 Template management parameters Parameter
-Description
-Name
-Indicates the template name.
-- A template name can contain only digits, letters, and underscores (_), but cannot start with an underscore (_) or contain only digits. It cannot be left empty.
- The template name can contain a maximum of 50 characters.
Description
-Description of the template you create.
-Statement
-SQL statement created as the template.
-Operation
-- Execute: After you click this button, the system switches to the SQL Editor page, where you can modify or directly perform the statement as required. For details, see Executing the Template.
- Modify: Click Modify. In the displayed Modify Template dialog box, modify the template information as required. For details, see Modifying a Template.
-Creating a Template
You can create a template on either the Job Templates or the SQL Editor page.
-- To create a template on the Job Templates page:
- On the left of the management console, choose Job Templates > SQL Templates.
- On the SQL Templates page, click Create Template to create a template.
Enter the template name, SQL statement, and description information. For details, see Table 3.
- -Table 3 Parameter description Parameter
-Description
-Name
-Indicates the template name.
+Indicates the template name.
- A template name can contain only digits, letters, and underscores (_), but cannot start with an underscore (_) or contain only digits. It cannot be left empty.
- The template name can contain a maximum of 50 characters.
Statement
+Statement
SQL statement to be saved as a template.
+SQL statement to be saved as a template.
Description
+Description
Description of the template you create.
+Description of the template you create.
Group
+Group
- Use existing
- Use new
- Do not use
- Use existing
- Use new
- Do not use
Group Name
+Group Name
If you select Use existing or Use new, you need to enter the group name.
+If you select Use existing or Use new, you need to enter the group name.
- Click OK.
- To create a template on the SQL Editor page:
- On the left of the management console, click SQL Editor.
- In the SQL job editing area of the displayed SQL Editor page, click More in the upper right corner and choose Save as Template.
Enter the template name, SQL statement, and description information. For details, see Table 3.
+ - To create a template on the SQL Editor page:
- On the left of the management console, click SQL Editor.
- In the SQL job editing area of the displayed SQL Editor page, click More in the upper right corner and choose Save as Template.
Enter the template name, SQL statement, and description information. For details, see Table 1.
- Click OK.
Executing the Template
Perform the template operation as follows:- On the left of the management console, choose Job Templates > SQL Templates.
- On the SQL Templates page, select a template and click Execute in the Operation column. The SQL Editor page is displayed, and the corresponding SQL statement is automatically entered in the SQL job editing window.
- In the upper right corner of the SQL job editing window, Click Execute to run the SQL statement. After the execution is complete, you can view the execution result below the current SQL job editing window.
-Submitting a SQL Job Using a Template
Perform the template operation as follows:- On the left of the management console, choose Job Templates > SQL Templates.
- On the SQL Templates page, select a template and click Execute in the Operation column. The SQL Editor page is displayed, and the corresponding SQL statement is automatically entered in the SQL job editing window.
- In the upper right corner of the SQL job editing window, Click Execute to run the SQL statement. After the execution is complete, you can view the execution result below the current SQL job editing window.
Searching for a Template
On the SQL Templates page, you can enter the template name keyword in the search box on the upper right corner to search for the desired template.
+-Searching for a SQL Job Template
On the SQL Templates page, you can enter the template name keyword in the search box on the upper right corner to search for the desired template.
Modifying a Template
Only custom templates can be modified. To modify a template, perform the following steps:
+-Modifying a SQL Job Template
Only custom templates can be modified. To modify a template, perform the following steps:
- On the SQL Templates page, locate the target template and click Modify in the Operation column.
- In the displayed Modify Template dialog box, modify the template name, statement, and description as required.
- Click OK.
Deleting a Template
On the SQL Templates page, select one or more templates to be deleted and click Delete to delete the selected templates.
+Deleting a Template
On the SQL Templates page, select one or more templates to be deleted and click Delete to delete the selected templates.
diff --git a/docs/dli/umn/dli_01_0022.html b/docs/dli/umn/dli_01_0022.html index c4ab96483..b3889e91c 100644 --- a/docs/dli/umn/dli_01_0022.html +++ b/docs/dli/umn/dli_01_0022.html @@ -21,7 +21,7 @@-Parent topic: Job Templates+Parent topic: Creating and Managing SQL Job Templates- Click the text box and select a predefined tag key from the drop-down list.
To add a predefined tag, you need to create one on TMS and then select it from the Tag key drop-down list. You can click View predefined tags to go to the Predefined Tags page of the TMS console. Then, click Create Tag in the upper corner of the page to create a predefined tag.
- Enter a tag key in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
+- Enter a tag key in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
Tag value
You can specify the tag value in either of the following ways:
-- Click the text box and select a predefined tag value from the drop-down list.
- Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed. The value cannot start or end with a space.
+- Click the text box and select a predefined tag value from the drop-down list.
- Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed.
diff --git a/docs/dli/umn/dli_01_0220.html b/docs/dli/umn/dli_01_0220.html index 3ffaab0dc..b092753a7 100644 --- a/docs/dli/umn/dli_01_0220.html +++ b/docs/dli/umn/dli_01_0220.html @@ -4,13 +4,15 @@-Parent topic: Queue Management+Parent topic: Managing Queuesdiff --git a/docs/dli/umn/dli_01_0228.html b/docs/dli/umn/dli_01_0228.html index f75ecdbe8..dc6c0ddda 100644 --- a/docs/dli/umn/dli_01_0228.html +++ b/docs/dli/umn/dli_01_0228.html @@ -1,126 +1,78 @@ --
-
- Creating and Submitting a Spark SQL Job
+ - Using DLI to Submit a SQL Job to Query OBS Data
- - Developing and Submitting a Spark SQL Job Using the TPC-H Sample Template
+ - Using DLI to Submit a SQL Job to Query RDS for MySQL Data
- - Creating and Submitting a Spark Jar Job
+ - Using DLI to Submit a Flink OpenSource SQL Job to Query RDS for MySQL Data
- - Creating and Submitting a Flink OpenSource SQL Job
+ - Using DLI to Submit a Flink Jar Job
+
+ - Using DLI to Submit a Spark Jar Job
Overview
-DLI database and table management provide the following functions:
-- Database Permission Management
- Table Permission Management
- Creating a Database or a Table
- Deleting a Database or a Table
- Changing the Owners of Databases and Tables
- Importing Data
- Exporting Data
- Viewing Metadata
- Previewing Data
-Difference Between DLI Tables and OBS Tables
- Data stored in DLI tables is applicable to delay-sensitive services, such as interactive queries.
- Data stored in OBS tables is applicable to delay-insensitive services, such as historical data statistics and analysis.
-Notes and Constraints
- Database
- default is the database built in DLI. You cannot create a database named default.
- DLI supports a maximum of 50 databases.
- Table
- DLI supports a maximum of 5,000 tables.
- DLI supports the following table types:
- MANAGED: Data is stored in a DLI table.
- EXTERNAL: Data is stored in an OBS table.
- View: A view can only be created using SQL statements.
- Datasource table: The table type is also EXTERNAL.
- You cannot specify a storage path when creating a DLI table.
- Data import
- Only OBS data can be imported to DLI or OBS.
- You can import data in CSV, Parquet, ORC, JSON, or Avro format from OBS to tables created on DLI.
- To import data in CSV format to a partitioned table, place the partition column in the last column of the data source.
- The encoding format of imported data can only be UTF-8.
- Data export
- Data in DLI tables (whose table type is MANAGED) can only be exported to OBS buckets, and the export path must contain a folder.
- The exported file is in JSON format, and the text format can only be UTF-8.
- Data can be exported across accounts. That is, after account B authorizes account A, account A has the permission to read the metadata and permission information of account B's OBS bucket as well as the read and write permissions on the path. Account A can export data to the OBS path of account B.
-Databases and Tables Page
The Databases and Tables page displays all created databases. You can view the database information, such as the owner and the number of tables.
- --
-Table 1 Database and table management parameters Parameter
-Description
-Database Name
-- The database name can contain only digits, letters, and underscores (_), but cannot contain only digits or start with an underscore (_).
- The database name is case insensitive and cannot be left unspecified.
- It cannot exceed 128 characters.
Username
-Database owner.
-Tables
-Number of tables in the database.
-Description
-Description of the database specified during database creation. If no description is provided, -- is displayed.
-Enterprise Project
-Enterprise project to which the database belongs. An enterprise project facilitates project-level management and grouping of cloud resources and users.
-Operation
-- Permissions: View the permission information and perform user authorization, permission settings, and user permission revocation.
- Tables: View the tables in the corresponding database. For details, see Table Management Page.
- Create Table: This permission allows you to create a table in the corresponding database.
- Modify Database. This permission allows you to change the owner of the database. The username must exist under the same account.
- Drop Database: This permission allows you to delete the selected database.
Table Management Page
From the Data Management page, click the database name or Tables in the Operation column to switch to the table management page.
-The displayed page lists all tables created in the current database. You can view the table type, data storage location, and other information. Tables are listed in chronological order by default, with the most recently created tables displayed at the top.
- -Table 2 Table management parameters Parameter
-Description
-Table Name
-- The table name can contain only digits, letters, and underscores (_), but cannot contain only digits or start with an underscore (_).
- The table name is case insensitive and cannot be left unspecified.
- The table name can contain the dollar sign ($). An example value is $test.
- It cannot exceed 128 characters.
Table Type
-Table type. Available options are as follows:
-- Managed: Indicates that data is stored in a DLI table.
- External: Indicates that data is stored in an OBS table.
- View: Indicates the view type. You can only create views using SQL statements.NOTE:
The table or view information contained in the view cannot be modified. If the table or view information is modified, the query may fail.
+Understanding Data Catalogs, Databases, and Tables
+Databases and tables are the basis for developing SQL and Spark jobs. Before running a job, you need to define databases and tables based on your service scenarios.++ Flink allows for dynamic data types, enabling the definition of data structures at runtime without the need for predefined metadata.
+Data Catalog
A data catalog is a metadata management object that can contain multiple databases.
+For how to create a database and a table in a DLI data catalog, see Creating a Database and Table on the DLI Console.
++Database
A database is a repository of data organized, stored, and managed on computer storage devices according to data structures. Databases are typically used to store, retrieve, and manage structured data, consisting of multiple data tables that are interrelated through keys and indexes.
++Table
Tables are one of the most important components of a database, consisting of rows and columns. Each row represents a data item, while each column represents a property or feature of the data. Tables are used to organize and store specific types of data, making it possible to query and analyze the data effectively.
+A database is a framework, and tables are its essential content. A database contains one or more tables.
+You can create databases and tables on the management console or using SQL statements. This section describes how to create a database and a table on the management console.
++ When creating a database or table, you need to grant permissions to other users so that they can view the new database or table.
++Table Metadata
Metadata is data used to define the type of data. It primarily describes information about the data itself, including its source, size, format, or other characteristics. In database fields, metadata is used to interpret the content of a data warehouse.
+When creating a table, metadata is defined by three columns: column name, type, and column description.
++Table Types Supported by DLI
- DLI table
DLI tables are data tables stored in a DLI data lake. They can store structured, semi-structured, and unstructured data.
+Data in DLI tables is stored internally within the DLI service, resulting in better query performance. This makes it suitable for time-sensitive services, such as interactive queries.
+In the navigation pane on the left of the DLI console, choose Data Management > Databases and Tables. On the displayed page, click the name of a database. In the displayed table list, tables whose Type is MANAGED are DLI tables.
+ - OBS table
Data in OBS tables is stored in the OBS service, which is suitable for latency-insensitive services, such as historical data statistics and analysis.
+An OBS table stores data in the form of objects. Each object contains data and related metadata.
+In the navigation pane on the left of the DLI console, choose Data Management > Databases and Tables. On the displayed page, click the name of a database. In the displayed table list, tables whose Type is EXTERNAL and Storage Location is OBS are OBS tables.
+ - View table
A view table is a virtual table that does not store actual data. Instead, it dynamically generates data based on the defined query logic. Views are typically used to simplify complex queries or provide customized data views for different users or applications.
+A view table can be created based on one or multiple tables, providing a flexible way to display data without affecting the storage and organization of the underlying data.
+In the navigation pane on the left of the DLI console, choose Data Management > Databases and Tables. On the displayed page, click the name of a database. In the displayed table list, tables whose Type is VIEW are view tables.
++ A view can only be created using SQL statements. You cannot create a view on the Create Table page. Table or view information in a view cannot be modified. Otherwise, the query may fail.
+ - Datasource table
A datasource table is a data table that can be queried and analyzed across multiple data sources. This type of tables can integrate data from varying data sources and provide a unified data view.
+Datasource tables are typically used in data warehouse and data lake architectures, allowing users to perform complex queries across multiple data sources.
+In the navigation pane on the left of the DLI console, choose Data Management > Databases and Tables. On the displayed page, click the name of a database. In the displayed table list, tables whose Type is EXTERNAL and Storage Location is not OBS are datasource tables.
+Notes and Constraints on Databases and Tables
+Table 1 Notes and constraints on DLI resources Item
+Description
+Database
+- default is the database built in DLI. You cannot create a database named default.
- DLI supports a maximum of 50 databases.
Owner
+Data table
User who creates the table.
+- DLI supports a maximum of 5,000 tables.
- DLI supports the following table types:
- MANAGED: Data is stored in a DLI table.
- EXTERNAL: Data is stored in an OBS table.
- View: A view can only be created using SQL statements.
- Datasource table: The table type is also EXTERNAL.
- You cannot specify a storage path when creating a DLI table.
Storage Location
+Data import
DLI, OBS, View, CloudTable, and CSS data location
+- Only OBS data can be imported to DLI or OBS.
- You can import data in CSV, Parquet, ORC, JSON, or Avro format from OBS to tables created on DLI.
- To import data in CSV format to a partitioned table, place the partition column in the last column of the data source.
- The encoding format of imported data can only be UTF-8.
Size
+Data export
Size of the data in the table. The value is displayed only for tables of the Managed type. For tables of other types, -- is displayed.
-Data Source Path
-- If Data Location is OBS, the corresponding OBS path is displayed.
- If Data Location is DLI and View, -- is displayed.
- When the data storage location is a datasource connection service such as CloudTable and CSS, the corresponding URL is displayed.
Created
-Time when the table is created.
-Last Accessed
-Last time when an operation was performed on the table.
-The last access time of a table refers only to the last time it was updated, not the time it was read (SELECT operation).
-Operation
-- Manage Permissions: This operation allows you to view the permission information and perform user authorization, permission settings, and user permission revocation.
- More:
- Delete: Delete a table from the corresponding database.
- Modify Owner: Change the owner of a table The username must exist under the same account.
- Import: Import data stored in an OBS bucket to a DLI or OBS table.
- Properties: View data in Metadata and Preview tabs.
- Data in DLI tables (whose table type is MANAGED) can only be exported to OBS buckets, and the export path must contain a folder.
- The exported file is in JSON format, and the text format can only be UTF-8.
- Data can be exported across accounts. That is, after account B authorizes account A, account A has the permission to read the metadata and permission information of account B's OBS bucket as well as the read and write permissions on the path. Account A can export data to the OBS path of account B.
Table Management Page
From the Data Management page, click the database name or Tables in the Operation column to switch to the table management page.
+The displayed page lists all tables created in the current database. You can view the table type, data storage location, and other information. Tables are listed in chronological order by default, with the most recently created tables displayed at the top.
+diff --git a/docs/dli/umn/dli_01_0253.html b/docs/dli/umn/dli_01_0253.html index 5efbeac4d..d87ef8470 100644 --- a/docs/dli/umn/dli_01_0253.html +++ b/docs/dli/umn/dli_01_0253.html @@ -1,7 +1,7 @@ --Parent topic: Databases and Tables+Parent topic: Creating Databases and TablesImporting Data to the Table
-You can import data from OBS to a table created in DLI.
+Importing OBS Data to DLI
+This section describes how to import data stored in OBS to a table on the DLI console.
Precautions
- Only one path can be specified during data import. The path cannot contain commas (,).
- To import data in CSV format to a partitioned table, place the column to be partitioned in the last column of the data source.
- You are advised not to concurrently import data in to a table. If you concurrently import data into a table, there is a possibility that conflicts occur, leading to failed data import.
- The imported file can be in CSV, Parquet, ORC, JSON, and Avro format. The encoding format must be UTF-8.
Prerequisites
The data to be imported has been stored on OBS.
@@ -60,7 +60,7 @@Table Header: No/Yes
This parameter is valid only when File Format is set to CSV. Whether the data source to be imported contains the table header.
-Click Advanced Settings and select the check box next to Table Header: No. If the check box is selected, the table header is displayed. If the check box is deselected, no table header is displayed.
+Click Advanced Settings and select the checkbox next to Table Header: No. If the checkbox is selected, the table header is displayed. If the checkbox is deselected, no table header is displayed.
-
diff --git a/docs/dli/umn/dli_01_0318.html b/docs/dli/umn/dli_01_0318.html index 2d5289e35..43c46d4e3 100644 --- a/docs/dli/umn/dli_01_0318.html +++ b/docs/dli/umn/dli_01_0318.html @@ -1,548 +1,403 @@ --Parent topic: Databases and Tables+Parent topic: Managing Table Resources on the DLI ConsoleDLI Operations That Can Be Recorded by CTS
-With CTS, you can record operations associated with DLI for later query, audit, and backtrack operations.
+Using CTS to Audit DLI
+With CTS, you can log operations related to DLI, making it easier to search, audit, and trace in the future.
-Table 1 DLI operations that can be recorded by CTS Operation
+-Table 1 DLI operations that can be recorded by CTS Operation
Resource Type
+Resource Type
Trace Name
+Trace Name
Creating a database
+Creating a database
database
+database
createDatabase
+createDatabase
Deleting a database
+Deleting a database
database
+database
deleteDatabase
+deleteDatabase
Changing the database owner
+Changing the database owner
database
+database
alterDatabaseOwner
+alterDatabaseOwner
Creating a table
+Creating a table
table
+table
createTable
+createTable
Deleting tables
+Deleting tables
table
+table
deleteTable
+deleteTable
Exporting table data
+Exporting table data
table
+table
exportData
+exportData
Importing table data
+Importing table data
table
+table
importData
+importData
Modifying the owner of a table
+Changing the owner of a table
table
+table
alterTableOwner
+alterTableOwner
Creating a queue
+Creating a queue
queue
+queue
createQueue
+createQueue
Deleting a queue
+Deleting a queue
queue
+queue
dropQueue
+deleteQueue
Granting permissions to a queue
+Granting permissions to a queue
queue
+queue
shareQueue
+queueAuthorize
Modifying the CIDR block of a queue
+Modifying the CIDR block of a queue
queue
+queue
replaceQueue
+replaceQueue
Restarting a queue
+Restarting a queue
queue
+queue
queueActions
+queueActions
Scaling out/in a queue
+Scaling out/in a queue
queue
+queue
queueActions
+queueActions
Submitting a job (SQL)
+Submitting a job (SQL)
queue
+queue
submitJob
+submitJob
Canceling a job (SQL)
+Canceling a job (SQL)
queue
+jobs
cancelJob
+cancelJob
Granting DLI the permission to access OBS buckets
+Granting DLI the permission to access OBS buckets
obs
+obs
obsAuthorize
+authorizeObsBucketsForStream
Checking the SQL syntax
+Checking SQL syntax
job
+jobs
checkSQL
+checkSQL
Creating a job
+Deleting a job
job
+jobs
createJob
+deleteStreamJob
Updating a job
+Creating a Flink OpenSource SQL job
job
+jobs
updateJob
+createStreamSqlJob
Deleting a job
+Updating a Flink OpenSource SQL job
job
+jobs
deleteJob
+updateStreamSqlJob
Creating a Flink OpenSource SQL job
+Deleting Flink jobs in batches
jobs
+jobs
createStreamSqlJob
+deleteStreamJobs
Updating a Flink OpenSource SQL job
+Stopping a Flink job
jobs
+jobs
updateStreamSqlJob
+stopStreamJobs
Deleting a Flink job
+Submitting a Flink job
jobs
+jobs
deleteStreamJobs
+submitStreamJobs
Stopping a Flink job
+Creating a Flink Jar job
jobs
+jobs
stopStreamJobs
+createStreamJarJob
Submitting a Flink job
+Updating a Flink Jar job
jobs
+jobs
submitStreamJobs
+updateStreamJarJob
Creating a Flink Jar job
+Viewing Flink jobs
jobs
+jobs
createStreamJarJob
+checkStreamJob
Updating a Flink Jar job
+Importing a savepoint
jobs
+jobs
updateStreamJarJob
+dealSavepoint
Checking Flink jobs
+Granting data permissions
jobs
+authorization
checkStreamJob
+dataAuthorize
Importing a savepoint
+Granting permissions on other projects
jobs
+authorization
dealSavepoint
+projectDataAuthorize
Purchasing CUH packages
+Exporting query results
order
+jobs
orderPackage
+storeJobResult
Freezing resources
+Saving a SQL template
resource
+template
freezeResource
+createTemplate
Unfreezing resources
+Updating a SQL template
resource
+template
unfreezeResource
+updateTemplate
Terminating resources
+Deleting a SQL template
resource
+template
deleteResource
+deleteTemplates
Clearing resources
+Creating a Flink template
resource
+template
cleanResource
+createStreamTemplate
Granting data permissions
+Updating a Flink template
data
+template
dataAuthorize
+updateStreamTemplate
Granting permissions on other projects
+Viewing Flink templates
data
+template
authorizeProjectData
+checkStreamTemplate
Exporting query results
+Deleting a Flink template
data
+template
storeJobResult
+deleteStreamTemplate
Saving a SQL template
+Creating a datasource authentication and uploading a certificate
sqlTemplate
+datasourceauth
saveSQLTemplate
+uploadAuthInfo
Updating a SQL template
+Updating datasource authentication information
sqlTemplate
+datasourceauth
updateSQLTemplate
+updateAuthInfo
Deleting a SQL template
+Deleting datasource authentication information
sqlTemplate
+datasourceauth
deleteSQLTemplate
+deleteAuthInfo
Creating a Flink template
+Uploading a resource package
template
+resource
createStreamTemplate
+uploadResources
Updating a Flink template
+Deleting a resource package
template
+resource
updateStreamTemplate
+deleteResource
Checking Flink templates
+Creating an enhanced datasource connection
template
+edsconnection
checkStreamTemplate
+createConnection
Deleting a Flink template
+Deleting an enhanced datasource connection
flinkTemplate
+edsconnection
deleteStreamTemplate
+deleteConnection
Creating a data upload task
+Binding a queue
uploader
+edsconnection
createUploadJob
+associateQueueToConnection
Obtaining the authentication to perform a data upload task
+Unbinding a queue
uploader
+edsconnection
getUploadAuthInfo
+disassociateQueueToConnection
Submitting a data upload task
+Modifying host information
uploader
+edsconnection
commitUploadJob
+updateHostInfo
Creating a datasource authentication and uploading a certificate
+Adding a route
authInfo
+edsconnection
uploadAuthInfo
+addRoute
Updating a datasource authentication
+Deleting a route
authInfo
+edsconnection
updateAuthInfop
+deleteRoute
Deleting a datasource authentication
+Creating a batch processing job
authInfo
+jobs
deleteAuthInfo
+createBatch
Updating the quota
+Canceling a batch processing job
quota
+jobs
updateQuota
+cancelBatch
Uploading a resource package
+Creating a global variable
pkgResource
+variable
uploadResources
+createGlobalVariable
Deleting a resource package
+Deleting a global variable
pkgResource
+variable
deleteResource
+deleteGlobalVariable
Creating a basic datasource connection
+Modifying a global variable
datasource
+variable
createDatasourceConn
-Deleting a basic datasource connection
-datasource
-deleteDatasourceConn
-Reactivating a basic datasource connection
-datasource
-reactivateDSConnection
-Creating an enhanced datasource connection
-datasource
-createConnection
-Deleting an enhanced datasource connection
-datasource
-getConnection
-Binding a queue
-datasource
-associateQueueToDatasourceConn
-Unbinding a queue
-datasource
-disassociateQueueToDatasourceConn
-Modifying the host information
-datasource
-updateHostInfo
-Adding a route
-datasource
-addRoute
-Deleting a route
-datasource
-deleteRoute
-Creating a topic
-smn
-createTopic
-Creating an agency
-agency
-createAgencyV2
-Creating a batch processing job
-batch
-createBatch
-Canceling a batch processing job
-batch
-cancelBatch
-Creating a session
-session
-createSession
-Deleting a session
-session
-deleteSession
-Creating a statement
-statement
-createStatement
-Canceling execution of a statement
-statement
-cancelStatement
-Creating a global variable
-globalVar
-createGlobalVariable
-Deleting a global variable
-globalVar
-deleteGlobalVariable
-Modifying a global variable
-globalVar
-updateGlobalVariable
+updateGlobalVariable
-diff --git a/docs/dli/umn/dli_01_0320.html b/docs/dli/umn/dli_01_0320.html index 066b8c93d..003ce509b 100644 --- a/docs/dli/umn/dli_01_0320.html +++ b/docs/dli/umn/dli_01_0320.html @@ -1,10 +1,10 @@ ---Parent topic: Other Common Operations-SQL Editor
-Introduction
You can edit and run SQL statements in the SQL job editor to execute data query.
-The editor supports SQL:2003 and is compatible with Spark SQL. For details about the syntax, see .
+Creating and Submitting a SQL Job
+Introduction
The SQL Editor allows you to execute data query operations using SQL statements.
+It supports SQL:2003 and is compatible with Spark SQL.
To access the SQL editor, choose SQL Editor in the left navigation pane of the DLI console, or click Create Job in the upper right corner of the Job Management > SQL Jobs page.
-This topic describes the main functions of the SQL editor.
+This section describes how to create and submit a SQL job using the DLI SQL editor.
-Notes
- If you access the SQL editor for the first time, the system prompts you to set a bucket for DLI jobs. The created bucket is used to store temporary data generated by DLI, such as job logs.
You cannot view job logs if you choose not to create the bucket. The bucket name will be set by the system.
On the OBS console, you can configure lifecycle rules for a bucket to periodically delete objects in it or change object storage classes.
@@ -12,222 +12,175 @@- SQL statements can be executed in batches on the SQL editor page.
- Commonly used syntax in the job editing window is highlighted in different colors.
- Both single-line comment and multi-line comment are allowed. Use two consecutive hyphens (--) in each line to comment your statements.
Navigation pane
The navigation pane on the left consists of Databases, Queues, and Templates tabs.
--Table 1 Navigation pane buttons No.
+- -Creating and Submitting a SQL Job Using the SQL Editor
- Log in to the DLI management console. In the navigation pane on the left, choose SQL Editor.+
On the SQL editor page, the system prompts you to create an OBS bucket to store temporary data generated by DLI jobs. In the Set Job Bucket dialog box, click Setting. On the page displayed, click the edit button in the upper right corner of the job bucket card. In the displayed Set Job Bucket dialog box, enter the job bucket path and click OK.
+ - Above the SQL job editing window, set the parameters required for running a SQL job, such as the queue and database. For how to set the parameters, refer to Table 1.
+-
-Table 1 Setting SQL job parameters Button & Drop-Down List
Name
-Description
+Description
1
+Queues
Databases
-Displays all the existing databases and tables in these databases.
-- Click a database name to view the tables in the database.
- Click a table name to view the metadata in the table. A maximum of 20 metadata records can be displayed.
- After you double-click a table name, a SQL query statement is automatically entered in the editing window.
Select a queue from the drop-down list box. If there is no queue available, the default queue is displayed. However, this queue is shared among all users for experience only. This may lead to resource contention, preventing you from obtaining the required resources for your jobs. So, you are advised to create your own queue for executing your jobs. For how to create a queue, see Creating an Elastic Resource Pool and Creating Queues Within It.
+SQL jobs can only be executed on SQL queues.
2
+Databases
Queues
-Displays existing queues.
-3
-Templates
-Click the drop-down button to view 22 built-in standard TPC-H query templates and custom templates.
-
-SQL Editing Window
SQL job editing window is displayed in the upper right part of the page. For details about the parameters, see Table 2.
-The SQL statement editing area is below the operation bar. For details about keyboard shortcuts, see Table 3.
---- -Table 2 Components of the SQL job editing window No.
-Button & Drop-Down List
-Description
-2
-Queues
-Select a queue from the drop-down list box. If no queue is available, the default queue is displayed. Refer to Creating a Queue and create a queue.
-SQL jobs can only be executed on SQL queues.
-3
-Database
-Select a database from the drop-down list box. If no database is available, the default database is displayed. For details about how to create a database, see Creating a Database or a Table.
-NOTE:If you specify the database in the SQL statements, the database you choose from the drop-down list will not be used.
+Select a database from the drop-down list box. If no database is available, the default database is displayed. For how to create a database, see Creating a Database and Table on the DLI Console.
+NOTE:If you have specified a database where tables are located in SQL statements, the database you choose here does not apply.
4
+Settings
Execute
-Click this button to run the SQL statements in the job editing window.
-5
-Format
-Click this button to format the SQL statements.
-6
-Syntax Reference
-Click this button to view the Data Lake Insight SQL Syntax Reference.
-7
-Settings
-Add parameters and tags.
-Parameter Settings: Set parameters in key/value format for SQL jobs.
-Tags: Set tags in key/value format for SQL jobs.
-8
-More
-The drop-down list includes the following options:
-- Click Verify Syntax to check whether the SQL statements are correct.
- Click Set as Template to set SQL statements as a template. For details, see Managing SQL Templates.
- Click Change Theme to switch between dark and light modes.
Add parameters and tags.
+Parameter Settings: Set parameters in key/value format for SQL jobs.
+Tags: Assign key-value pairs as tags to a SQL job.
-
-Table 3 Keyboard shortcut description Shortcut
-Description
-Ctrl+Enter
-Execute SQL statements. You can run SQL statements by pressing Ctrl+R or Ctrl + Enter on the keyboard.
-Ctrl+F
-Search for SQL statements. You can press Ctrl+F to search for a required SQL statement.
-Shift+Alt+F
-Format SQL statements. You can press Shift + Alt + F to format a SQL statement.
-Ctrl+Q
-Syntax verification. You can press Ctrl + Q to verify the syntax of SQL statements.
-F11
-Full screen. You can press F11 to display the SQL Job Editor window in full screen. Press F11 again to leave the full screen.
--Executed Queries (Last Day) and View Result
After the SQL job is executed, you can view the execution history and result in the lower part of the editing area.
-- Executed Queries (Last Day)
You can filter the execution history in the following ways:
-- In the search box in the upper right corner of the Executed Queries (Last Day) pane, select a queue name or enter an execution statement in the search box.
- In the list, click the icon next to Created and choose Ascending or Descending.
- Select a job status from the Status list.
-
-Table 4 Area description Area
-Description
-Executed Queries (Last Day)
-- Queues: Queue name
- Username: User who executes the SQL statements
- Type: Type of the SQL job
- Status: Execution status of the SQL job
- Query
- Created
- Operation
- Edit: Edit the SQL statement.
- SparkUI: Switch to the SparkUI page to view the SQL statement execution process.NOTE:-
When you execute a job on a created queue, the cluster is restarted. It takes about 10 minutes. If you click SparkUI before the cluster is created, an empty projectID will be cached. The SparkUI page cannot be displayed. You are advised to use a dedicated queue so that the cluster will not be released. Alternatively, wait for a while after the job is submitted (the cluster is created), and then check SparkUI.
-Currently, only the latest 100 job information records are displayed on the SparkUI of DLI.
-This function is not supported for synchronization jobs and jobs running on the default queue.
- - More: The following operations vary depending on the SQL job types and running status.
Cancel: Cancel a SQL job that is running or being submitted.
-Re-execute: Execute the SQL statement again.
-View Result: View the execution result of a QUERY job.
-Export Result: Export the execution results of a QUERY job to a specified OBS path.
-View Log: View the OBS path for storing SQL statement execution logs.
-NOTE:-The View Log button is not available for synchronization jobs and jobs running on the default queue.
-
- View Result
--
-Table 5 Operations on the result tab Operation
-Description
-Clear the result
-Clear the displayed SQL statement query results.
-View chart/table
-Click
-
to view the query result in a chart or table.Export job results
-Export the job execution results to the created OBS bucket.
-
SQL Query Procedure
- Log in to the DLI management console. On the page displayed, choose Job Management > SQL Jobs. On the page displayed, click Create Job.-
On the SQL editor page, the system prompts you to create an OBS bucket to store temporary data generated by DLI jobs. In the Set Job Bucket dialog box, click Setting. On the page displayed, click the edit button in the upper right corner of the job bucket card. In the Set Job Bucket dialog box displayed, enter the job bucket path and click OK.
- - Select a queue from the queue list in the upper left corner of the SQL job editing window. For details about how to create a queue, see Creating a Queue.
- In the upper right corner of the SQL job editing window, select a database, for example, qw, from the Databases drop-down list.
- Create a table, for example, qw. For details about how to create a database and table, see Creating a Database or a Table.
- In the SQL job editing window, enter the following SQL statement:
1SELECT * FROM qw.qw LIMIT 10; +
- Create a database and a table.
Create them in advance by referring to Creating a Database and Table on the DLI Console. For example, create a table named qw.
+- In the SQL job editing window, enter the following SQL statement:
1SELECT * FROM qw.qw LIMIT 10;
Alternatively, you can double-click the table name qw. The query statement is automatically entered in the SQL job editing window.
-- On top of the editing window, click More > Verify Syntax to check whether the SQL statement is correct.
- If the verification fails, check the SQL statement syntax by referring to Data Lake Insight SQL Syntax Reference.
- If the syntax verification is successful, click Execute. Read and agree to the privacy agreement. Click OK to execute the SQL statement.
- After the execution is complete, you can view the execution result in the area under the SQL job editing window.
- (Optional) A maximum of 1000 records can be displayed in the query result on the current console. To view more or all data, click
to export the data to OBS.- (Optional) In the View Result tab, click
to display the query result in a chart. Click 
to switch back to the table view. -- If no column of the numeric type is displayed in the execution result, the result cannot be represented in charts.
- You can view the data in a bar chart, line chart, or fan chart.
- In the bar chart and line chart, the X axis can be any column, while the Y axis can only be columns of the numeric type. The fan chart displays the corresponding legends and indicators.
DLI offers a range of SQL templates that come with use cases, code examples, and usage guides. You can also use these templates to quickly implement your service logic. For more information about templates, see Creating a SQL Job Template.
+- On top of the editing window, click More > Verify Syntax to check whether the SQL statement is correct.
- If the verification fails, check the SQL statement syntax by referring to Data Lake Insight SQL Syntax Reference.
- If the syntax verification is successful, click Execute. Read and agree to the privacy agreement. Click OK to execute the SQL statement.
Once successfully executed, you can check the execution result on the View Result tab below the SQL job editing window.
+- View job execution results.
On the View Result tab, click
+
to display execution results in a chart. Click 
to switch back to the table view.You can view a maximum of 1,000 data records on this View Result tab. To view more or full data, click to export the data to OBS.+ - If no column of the numeric type is displayed in the execution result, the result cannot be represented in charts.
- You can view the data in a bar chart, line chart, or fan chart.
- In the bar chart and line chart, the X axis can be any column, while the Y axis can only be columns of the numeric type. The fan chart displays the corresponding legends and indicators.
- +Quickly Importing SQL Statements
- Double-click a table name in the navigation pane on the left to import the query statement of the selected table into the SQL statement editing window, and then click Execute to query.
- You can click More and choose Save as Template to save the SQL statement as a template for future use.
To use the SQL statement template, click Templates from the left pane of the SQL editor page. Double-click the required template in the template list, and modify it as required before executing the SQL statements.
+Functions of SQL Editor
- Parameter settings for SQL jobs
Click Settings in the upper right corner of the SQL Editor page. You can set parameters and tags for the SQL job.
+- Parameter Settings: Assign key-value pairs as parameter settings. + +
- Tags: Assign key-value pairs as tags to a SQL job.
+
+Table 2 Parameters for SQL job running Parameter
+Default Value
+Description
+spark.sql.files.maxRecordsPerFile
+0
+Maximum number of records to be written into a single file. If the value is zero or negative, there is no limit.
+spark.sql.autoBroadcastJoinThreshold
+209715200
+Maximum size, in bytes, of the table that displays all working nodes when a connection is executed. You can set this parameter to -1 to disable the display.
+NOTE:+Currently, only configuration units that store tables analyzed using the ANALYZE TABLE COMPUTE statistics noscan command and file-based data source tables that calculate statistics directly from data files are supported.
+spark.sql.shuffle.partitions
+200
+Default number of partitions used to filter data for join or aggregation.
+spark.sql.dynamicPartitionOverwrite.enabled
+false
+When set to false, DLI will delete all partitions that meet the conditions before overwriting them. For example, if there is a partition named 2021-01 in a partitioned table and you use the INSERT OVERWRITE statement to write data to the 2021-02 partition, the data in the 2021-01 partition will also be overwritten.
+When set to true, DLI will not delete partitions in advance, but will overwrite partitions with data written during runtime.
+spark.sql.files.maxPartitionBytes
+134217728
+Maximum number of bytes to be packed into a single partition when a file is read.
+spark.sql.badRecordsPath
+-
+Path of bad records.
+dli.sql.sqlasync.enabled
+true
+Whether DDL and DCL statements are executed asynchronously. The value true indicates that asynchronous execution is enabled.
+dli.sql.job.timeout
+-
+Job running timeout interval, in seconds. If the job times out, it will be canceled.
+ - Switching to the SparkUI page to view the SQL statement execution process
The SQL editor allows you to switch to the SparkUI to view the SQL statement execution process.
+- You can view only the latest 100 job records on DLI's SparkUI.
- If a job is running on the default queue or is a synchronization one, you cannot switch to the SparkUI to view the SQL statement execution process.
+ When you execute a job on a created queue, the cluster is restarted. It takes about 10 minutes. If you click SparkUI before the cluster is created, an empty projectID will be cached. The SparkUI page cannot be displayed. You are advised to use a dedicated queue so that the cluster will not be released. Alternatively, wait for a while after the job is submitted (the cluster is created), and then check SparkUI.
+ - Archiving SQL run logs
On the Executed Queries (Last Day) tab of the SQL Editor page, click More and select View Log in the Operation column of the SQL job. The system automatically switches to the OBS path where logs are stored. You can download logs as needed.
++ The View Log button is not available for synchronization jobs and jobs running on the default queue.
+ - SQL Editor shortcuts
+
+Table 3 Keyboard shortcuts Shortcut
+Description
+Ctrl+Enter
+Execute SQL statements. You can run SQL statements by pressing Ctrl+R or Ctrl + Enter on the keyboard.
+Ctrl+F
+Search for SQL statements. You can press Ctrl+F to search for a required SQL statement.
+Shift+Alt+F
+Format SQL statements. You can press Shift + Alt + F to format a SQL statement.
+Ctrl+Q
+Syntax verification. You can press Ctrl + Q to verify the syntax of SQL statements.
+F11
+Full screen. You can press F11 to display the SQL Job Editor window in full screen. Press F11 again to leave the full screen.
+
- Parameter settings for SQL jobs
+diff --git a/docs/dli/umn/dli_01_0363.html b/docs/dli/umn/dli_01_0363.html index 4100a31b4..c66e75103 100644 --- a/docs/dli/umn/dli_01_0363.html +++ b/docs/dli/umn/dli_01_0363.html @@ -1,60 +1,89 @@ -++Parent topic: Submitting a SQL Job Using DLI+Creating a Queue
-Before executing a job, you need to create a queue.
+Creating a Non-Elastic Resource Pool Queue (Discarded and Not Recommended)
+Queues in the non-elastic resource pool mode are the previous-gen of resource management for DLI. It involved purchasing and releasing resources based on usage demands, requiring estimation of resource needs before making purchases.
+Queues in an elastic resource pool are recommended, as they offer the flexibility to use resources with high utilization as needed. For how to buy an elastic resource pool and create queues within it, see Creating an Elastic Resource Pool and Creating Queues Within It.
- - If you use a sub-account to create a queue for the first time, log in to the DLI management console using the main account and keep records in the DLI database before creating a queue.
- It takes 6 to 10 minutes for a job running on a new queue for the first time.
- After a queue is created, if no job is run within one hour, the system releases the queue.
- Queues with 16 CUs do not support scale-out or scale-in.
- Queues with 64 CUs do not support scale-in.
Procedure
- You can create a queue on the Overview, SQL Editor, or Queue Management page.
- In the upper right corner of the Overview page, click Create Queue.
- To create a queue on the Queue Management page:
- In the navigation pane of the DLI management console, choose Resources >Queue Management.
- In the upper right corner of the Queue Management page, click Create Queue to create a queue.
- To create a queue on the SQL Editor page:
- In the navigation pane of the DLI management console, click SQL Editor.
- Click Queues. On the tab page displayed, click
on the right to create a queue.
- On the Create Queue page displayed, set the parameters according to Table 1.
-
Table 1 Parameters Parameter
++Notes and Constraints
++
+Table 1 Notes and constraints on queues Item
Description
+Description
Name
+Resource type
Name of a queue.
-- The queue name can contain only digits, letters, and underscores (_), but cannot contain only digits, start with an underscore (_), or be left unspecified.
- The length of the name cannot exceed 128 characters.
- Queue types:
- default queue: A queue named default is preset in DLI, where you can use resources on demand.
- For SQL: Spark SQL jobs can be submitted to SQL queues.
- For general purpose: The queue is used to run Spark programs, Flink SQL jobs, and Flink Jar jobs.
- You cannot change the queue type once a queue is purchased. To use another queue type, purchase a new queue.
Managing queues
+- The region of a queue cannot be changed.
- DLI queues cannot access the Internet.
Queue scaling
+- Queues with 16 CUs do not support scale-out or scale-in.
- Queues with 64 CUs do not support scale-in.
- Newly created queues need to run jobs before they can be scaled in or out.
Procedure
- You can create a queue on the Overview, SQL Editor, or Queue Management page.
- In the upper right corner of the Overview page, click Create Queue.
- To create a queue on the Queue Management page:
- In the navigation pane on the left of the DLI management console, choose Resources > Queue Management.
- In the upper right corner of the Queue Management page, click Create Queue to create a queue.
- To create a queue on the SQL Editor page:
- In the navigation pane of the DLI management console, click SQL Editor.
- Click Queues. On the tab page displayed, click on the right to create a queue.
- On the Create Queue page displayed, set the parameters according to Table 2.
+-
Table 2 Parameters Parameter
+Description
+Name
+Queue name
+- Only numbers, letters, and underscores (_) are allowed. The value cannot contain only numbers, start with an underscore (_), or be left unspecified.
- The value can contain a maximum of 128 characters.
NOTE:The queue name is case-insensitive. Uppercase letters will be automatically converted to lowercase letters.
Type
+Type
- For SQL: compute resources used for SQL jobs.
- For general purpose: compute resources used for Spark and Flink jobs.NOTE:
Selecting Dedicated Resource Mode enables you to create a dedicated queue. Enhanced datasource connections can only be created for dedicated queues.
+- For SQL: compute resources used for SQL jobs.
- For general purpose: compute resources used for Spark and Flink jobs.NOTE:
When a dedicated queue is not in use, its resources are still reserved and not released. This means that resources are constantly being held, regardless of whether the queue is being utilized. By using a dedicated queue, resources can be guaranteed to be available whenever jobs are submitted. You can create enhanced datasource connections for a dedicated queue.
+
Specifications
+Specifications
The compute nodes' total number of CUs. One CU equals one vCPU and 4 GB of memory. DLI automatically assigns CPU and memory resources to each compute node, and the client does not need to know how many compute nodes are being used.
+The compute nodes' total number of CUs. One CU equals one vCPU and 4 GB of memory. DLI automatically assigns CPU and memory resources to each compute node, and the client does not need to know how many compute nodes are being used.
Description
+Description
Description of the queue to be created. The description can contain a maximum of 128 characters.
+Description of the queue to be created. The value can contain a maximum of 128 characters.
Advanced Settings
+Advanced Settings
In the Queue Type area, select Dedicated Resource Mode and then click Advanced Settings.- Default: The system automatically configures the parameter.
- Custom
CIDR Block: You can specify the CIDR block. For details, see Modifying the CIDR Block. If DLI enhanced datasource connection is used, the CIDR block of the DLI queue cannot overlap with that of the data source.
+In the Queue Type area, select Dedicated Resource Mode and then click Advanced Settings.- Default: The system automatically configures the parameter.
- Custom
CIDR Block: Enter a CIDR block range. If DLI enhanced datasource connection is used, the CIDR block of the DLI queue cannot overlap with that of the data source.
Queue Type: When running an AI-related SQL job, select AI-enhanced. When running other jobs, select Basic.
Tags
+Tags
Tags used to identify cloud resources. A tag includes the tag key and tag value. If you want to use the same tag to identify multiple cloud resources, that is, to select the same tag from the drop-down list box for all services, you are advised to create predefined tags on the Tag Management Service (TMS).
+Tags used to identify cloud resources. A tag includes the tag key and tag value. If you want to use the same tag to identify multiple cloud resources, that is, to select the same tag from the drop-down list box for all services, you are advised to create predefined tags on the Tag Management Service (TMS).
NOTE:-- A maximum of 20 tags can be added.
- Only one tag value can be added to a tag key.
- The key name in each resource must be unique.
- Tag key: Enter a tag key name in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
+- Tag key: Enter a tag key name in the text box.NOTE:-
A tag key can contain a maximum of 128 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
- Tag value: Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed. The value cannot start or end with a space.
+- Tag value: Enter a tag value in the text box.
NOTE:A tag value can contain a maximum of 255 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed.
- Tag value: Enter a tag value in the text box.
- Tag key: Enter a tag key name in the text box.
- Click Create Now to create a queue.
After a queue is created, you can view and select the queue for use on the Queue Management page.
+ - Click Create Now to create a queue.
Once created, you can check and use it on the Queue Management page.
It takes 6 to 10 minutes for a job running on a new queue for the first time.
diff --git a/docs/dli/umn/dli_01_0366.html b/docs/dli/umn/dli_01_0366.html index 1352042b4..ca1a92347 100644 --- a/docs/dli/umn/dli_01_0366.html +++ b/docs/dli/umn/dli_01_0366.html @@ -1,25 +1,25 @@ --Parent topic: Queue Management+Parent topic: Creating an Elastic Resource Pool and Queues Within ItPackage Management
+Managing Program Packages of Jar Jobs
diff --git a/docs/dli/umn/dli_01_0367.html b/docs/dli/umn/dli_01_0367.html index e4bb90f6c..be1da1f56 100644 --- a/docs/dli/umn/dli_01_0367.html +++ b/docs/dli/umn/dli_01_0367.html @@ -1,6 +1,6 @@ --
-
- Overview
+ - Package Management Overview
- - Managing Permissions on Packages and Package Groups
+ - Creating a DLI Package
- - Creating a Package
+ - Configuring DLI Package Permissions
- - Deleting a Package
+ - Changing the DLI Package Owner
- - Modifying the Owner
+ - Managing DLI Package Tags
- - Built-in Dependencies
+ - DLI Built-in Dependencies
-Parent topic: Data Management+Parent topic: Common DLI Management OperationsCreating a Package
+Creating a DLI Package
DLI allows you to submit program packages in batches to the general-use queue for running.
@@ -44,10 +44,10 @@ If you need to update a package, you can use the same package or file to upload it to the same location (in the same group) on DLI to overwrite the original package or file.
Tags used to identify cloud resources. A tag includes the tag key and tag value. If you want to use the same tag to identify multiple cloud resources, that is, to select the same tag from the drop-down list box for all services, you are advised to create predefined tags on the Tag Management Service (TMS).
NOTE:-- A maximum of 20 tags can be added.
- Only one tag value can be added to a tag key.
- The key name in each resource must be unique.
- Tag key: Enter a tag key name in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
+- Tag key: Enter a tag key name in the text box.NOTE:
A tag key can contain a maximum of 128 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
- Tag value: Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed. The value cannot start or end with a space.
+- Tag value: Enter a tag value in the text box.NOTE:
A tag value can contain a maximum of 255 characters. Only letters, numbers, spaces, and special characters (_.:+-@) are allowed.
- Tag value: Enter a tag value in the text box.
- Tag key: Enter a tag key name in the text box.
- Click OK.
After a package is created, you can view and select the package for use on the Package Management page.
+Once a job is executed, you can free up DLI storage space by promptly deleting the job package from the package management page if it is no longer needed.
diff --git a/docs/dli/umn/dli_01_0369.html b/docs/dli/umn/dli_01_0369.html index 3c5dbe60c..00cf39928 100644 --- a/docs/dli/umn/dli_01_0369.html +++ b/docs/dli/umn/dli_01_0369.html @@ -1,13 +1,44 @@ --Parent topic: Package Management+Parent topic: Managing Program Packages of Jar JobsDeleting a Package
-You can delete a package based on actual conditions.
-Procedure
- On the left of the management console, choose Data Management > Package Management.
- Click Delete in the Operation column of the package to be deleted.
- In the dialog box that is displayed, click Yes.
Managing DLI Package Tags
+Tags are key-value pairs that you can define to identify cloud resources. They assist you in categorizing and searching for cloud resources. A tag consists of a key and a value.
+DLI allows you to add tags to package groups or packages.
+- Log in to the DLI management console and choose Data Management > Package Management.
- On the Package Management page, locate the desired package, click More in the Operation column, and select Tags.
- On the page that appears, click Add/Edit Tag in the upper left corner.
- In the Add/Edit Tag dialog box, set parameters.
+++
- A maximum of 20 tags can be added.
- Only one tag value can be added to a tag key.
- The key name in each resource must be unique.
- Click OK.
- (Optional) To delete a tag, locate the tag in the tag list and click Delete in its Operation column.
diff --git a/docs/dli/umn/dli_01_0375.html b/docs/dli/umn/dli_01_0375.html index ed769d0a3..3a63422e1 100644 --- a/docs/dli/umn/dli_01_0375.html +++ b/docs/dli/umn/dli_01_0375.html @@ -1,40 +1,191 @@ --Parent topic: Package Management+Parent topic: Managing Program Packages of Jar JobsCreating and Submitting a Spark Jar Job
-Scenario
DLI can query data stored in OBS. This section describes how to use a Spark Jar job on DLI to query OBS data in real time.
+Using DLI to Submit a Spark Jar Job
+-Scenario
DLI allows you to submit Spark jobs compiled as JAR files, which contain the necessary code and dependency information for executing the job. These files are used for specific data processing tasks such as data query, analysis, and machine learning. Before submitting a Spark Jar job, upload the package to OBS and submit it along with the data and job parameters to run the job.
+This example introduces the basic process of submitting a Spark Jar job package through the DLI console. Due to different service requirements, the specific writing of the Jar package may vary. It is recommended that you refer to the sample code provided by DLI and edit and customize it according to your actual business scenario.
Procedure
You can use DLI to submit Spark jobs for real-time computing. The general procedure is as follows:
- - - - +Procedure
Table 1 describes the procedure for submitting a Spark Jar job using DLI.
+ +-Table 1 Procedure for submitting a Spark Jar job using DLI Step
+Description
++ +Prepare a Spark Jar job package and upload it to OBS.
+Step 2: Create an Elastic Resource Pool and Add Queues to the Pool
+Create compute resources required for submitting the Spark Jar job.
++ +Create a Spark Jar job to analyze data.
++Step 1: Upload Data to OBS
Write a Spark Jar job program, and compile and pack it as spark-examples.jar. Perform the following steps to submit the job:
+Step 1: Upload Data to OBS
Develop a Spark Jar job program, compile it, and pack it into spark-examples.jar. Perform the following steps to upload the program:
Before submitting Spark Jar jobs, upload data files to OBS.
-- Log in to the DLI console.
- In the service list, click Object Storage Service under Storage. The OBS console page is displayed.
- Create a bucket. In this example, name it dli-test-obs01.
- Click Create Bucket.
- On the displayed Create Bucket page, enter the Bucket Name. Retain the default values for other parameters or set them as required. -
When creating an OBS bucket, you must select the same region as the DLI management console.
+- Log in to the DLI console.
- In the service list, click Object Storage Service under Storage.
- Create a bucket. In this example, name it dli-test-obs01.
- On the displayed Buckets page, click Create Bucket in the upper right corner.
- On the displayed Create Bucket page, enter the Bucket Name. Retain the default values for other parameters or set them as required.
Select a region that matches the location of the DLI console.
- Click Create Now.
- Click dli-test-obs01 to switch to the Objects tab page.
- Click Upload Object. In the dialog box displayed, drag or add files or folders, for example, spark-examples.jar, to the upload area. Then, click Upload.
After the file is uploaded successfully, the file path is obs://dli-test-obs01/spark-examples.jar.
-+ - For more information about OBS operations, see the Object Storage Service Console Operation Guide.
- For more information about the tool, see the OBS Tool Guide.
- You are advised to use an OBS tool, such as OBS Browser+, to upload large files because OBS Console has restrictions on the file size and quantity.
- OBS Browser+ is a graphical tool that provides complete functions for managing your buckets and objects in OBS. You are advised to use this tool to create buckets or upload objects.
- In the bucket list, click the name of the dli-test-obs01 bucket you just created to access its Objects tab.
- Click Upload Object. In the dialog box displayed, drag or add files or folders, for example, spark-examples.jar, to the upload area. Then, click Upload.
In this example, the path after upload is obs://dli-test-obs01/spark-examples.jar.
+For more operations on the OBS console, see the Object Storage Service User Guide.
Step 2: Create a Queue
If you submit a Spark job for the first time, you need to create a queue first. For example, create a queue named sparktest and set Queue Type to General Queue.
-- Log in to the DLI management console.
- In the navigation pane of the DLI management console, choose Resources > Queue Management.
- In the upper right corner of the Queue Management page, click Create Queue to create a queue.
- Create a queue, name it sparktest, and set the queue usage to for general purpose. For details, see Creating a Queue. -
- Click Create Now to create a queue.
Step 2: Create an Elastic Resource Pool and Add Queues to the Pool
In this example, the elastic resource pool dli_resource_pool and queue dli_queue_01 are created.- Log in to the DLI management console.
- In the navigation pane on the left, choose Resources > Resource Pool.
- On the displayed page, click Buy Resource Pool in the upper right corner.
- On the displayed page, set the parameters.Table 2 describes the parameters. ++-
Table 2 Parameters Parameter
+Description
+Example Value
+Region
+Select a region where you want to buy the elastic resource pool.
+_
+Project
+Project uniquely preset by the system for each region
+Default
+Name
+Name of the elastic resource pool
+dli_resource_pool
+Specifications
+Specifications of the elastic resource pool
+Standard
+CU Range
+The maximum and minimum CUs allowed for the elastic resource pool
+64-64
+CIDR Block
+CIDR block the elastic resource pool belongs to. If you use an enhanced datasource connection, this CIDR block cannot overlap that of the data source. Once set, this CIDR block cannot be changed.
+172.16.0.0/19
+Enterprise Project
+Select an enterprise project for the elastic resource pool.
+default
+-Step 3: Create a Package
Before submitting a Spark job, you need to create a package, for example, spark-examples.jar.
-- In the navigation pane on the left of the DLI console, choose Data Management > Package Management.
- On the Package Management page, click Create in the upper right corner to create a package.
- In the Create Package dialog box, set Type to JAR, OBS Path to the path of the spark-examples.jar package in Step 1: Upload Data to OBS, and Group to Do not use. -
- Click OK.
You can view and select the package on the Package Management page.
-
For details about how to create a package, see "Creating a Package".
+Step 4: Submit a Spark Job
- On the DLI management console, choose Job Management > Spark Jobs in the navigation pane on the left. On the displayed page, click Create Job.
- On the Spark job editing page, set Queues to the queue created in Step 2: Create a Queue and Application to the package created in Step 3: Create a Package.
For details about other parameters, see the description of the Spark job editing page in "Creating a Spark Job".
+ - Click Buy.
- Click Submit.
- In the elastic resource pool list, locate the pool you just created and click Add Queue in the Operation column.
- Set the basic parameters listed below.
++
+Table 3 Basic parameters for adding a queue Parameter
+Description
+Example Value
+Name
+Name of the queue to add
+dli_queue_01
+Type
+Type of the queue
+- To execute SQL jobs, select For SQL.
- To execute Flink or Spark jobs, select For general purpose.
_
+Enterprise Project
+Select an enterprise project.
+default
+ - Click Next and configure scaling policies for the queue.
Click Create to add a scaling policy with varying priority, period, minimum CUs, and maximum CUs.
+ + ++
+Table 4 Scaling policy parameters Parameter
+Description
+Example Value
+Priority
+Priority of the scaling policy in the current elastic resource pool. A larger value indicates a higher priority. In this example, only one scaling policy is configured, so its priority is set to 1 by default.
+1
+Period
+The first scaling policy is the default policy, and its Period parameter configuration cannot be deleted or modified.
+The period for the scaling policy is from 00 to 24.
+00–24
+Min CU
+Minimum number of CUs allowed by the scaling policy
+16
+Max CU
+Maximum number of CUs allowed by the scaling policy
+64
+ - Click OK.
Step 3: Submit a Spark Job
- On the DLI management console, choose Job Management > Spark Jobs in the navigation pane on the left. On the displayed page, click Create Job in the upper right corner.
- Set the following Spark job parameters:
- Queue: Select the queue created in Step 2: Create an Elastic Resource Pool and Add Queues to the Pool.
- Spark Version: Select a Spark engine version.
- Application: Select the package created in Step 1: Upload Data to OBS.
For other parameters, refer to the description about the Spark job editing page in "Creating a Spark Job" in the Data Lake Insight User Guide.
- Click Execute in the upper right corner of the Spark job editing window, read and agree to the privacy agreement, and click OK. Submit the job. A message is displayed, indicating that the job is submitted. -
- (Optional) Switch to the Job Management > Spark Jobs page to view the status and logs of the submitted Spark job.-
When you click Execute on the DLI management console for the first time, you need to read the privacy agreement. Once agreed to the agreement, you will not receive any privacy agreement messages for subsequent operations.
-
- (Optional) Switch to the Job Management > Spark Jobs page to view the status and logs of the submitted Spark job.
diff --git a/docs/dli/umn/dli_01_0376.html b/docs/dli/umn/dli_01_0376.html index b48216389..51151ca9e 100644 --- a/docs/dli/umn/dli_01_0376.html +++ b/docs/dli/umn/dli_01_0376.html @@ -1,18 +1,16 @@ -Modifying the Owners of Databases and Tables
-During actual use, developers create databases and tables and submit them to test personnel for testing. After the test is complete, the databases and tables are transferred to O&M personnel for user experience. In this case, you can change the owner of the databases and tables to transfer data to other owners.
-Modifying the Database Owner
You can change the owner of a database on either the Data Management page or the SQL Editor page.- On the Data Management page, change the database owner.
- On the left of the management console, choose Data Management > Databases and Tables.
- On the Databases and Tables page, locate the database you want and click More > Modify Database in the Operation column.
- In the displayed dialog box, enter a new owner name (an existing username) and click OK.
Changing the Database Owner on the DLI Console
+In practical use, developers create databases and tables, which are then handed over to testers for testing. Once testing is complete, the databases and tables are handed over to O&M personnel for experience. In this scenario, ownership of the data can be transferred to another owner by changing the database owner.
+-Changing the Database Owner
You can change the owner of a database on either the Data Management page or the SQL Editor page.- On the Data Management page, change the database owner.
- On the left of the management console, choose Data Management > Databases and Tables.
- On the Databases and Tables page, locate the database you want and click More > Modify Database in the Operation column.
- In the displayed dialog box, enter a new owner name (an existing username) and click OK.
- Change the database owner on the SQL Editor page.
- On the left of the management console, click SQL Editor.
- In the navigation tree on the left, click Databases, click
on the right of the database you want to modify, and choose Modify Database from the shortcut menu. - In the displayed dialog box, enter a new owner name (an existing username) and click OK.
Modifying the Table Owner
- On the left of the management console, choose Data Management > Databases and Tables.
- Click the name of the database corresponding to the table to be modified. The Manage Tables page of the database is displayed.
- In the Operation column of the target table, choose More > Modify Owner.
- In the displayed dialog box, enter a new owner name (an existing username) and click OK.
diff --git a/docs/dli/umn/dli_01_0377.html b/docs/dli/umn/dli_01_0377.html index 1d060c238..c15319069 100644 --- a/docs/dli/umn/dli_01_0377.html +++ b/docs/dli/umn/dli_01_0377.html @@ -1,17 +1,27 @@ --Parent topic: Databases and Tables+Parent topic: Managing Database Resources on the DLI ConsoleDLI Console Overview
-The Overview page of the DLI console provides you with the DLI workflow and resource usage.
-How to Use DLI
The process of using DLI is as follows:
-- Create a queue.
Queues are DLI's compute resources. There are SQL queues and general-purpose queues. For a SQL queue, you can only submit Spark SQL jobs. For a general-purpose queue, you can submit Spark programs, Flink OpenSource SQL jobs, and Flink Jar jobs.
- - Prepare data.
Create databases and tables before you run a Spark SQL job. Upload a program package before you run a Spark job or a Flink Jar job.
- - Edit and submit a job.
After you set the job parameters, you can submit the job.
- - View job status.
Go to the Job Management page to view the job execution status.
- -
Managing Flink Jobs
+ +-Queue Usage (hours)
The overview page provides usage durations of all queues or a specific queue.
-- Usage Usage (hours): an overview of the usage of all resources.
- Selected Queue Usage (hours): recent usage of a single queue.
+diff --git a/docs/dli/umn/dli_01_0378.html b/docs/dli/umn/dli_01_0378.html index c4ef4d49d..bfc8fba10 100644 --- a/docs/dli/umn/dli_01_0378.html +++ b/docs/dli/umn/dli_01_0378.html @@ -1,18 +1,19 @@ --
+
- Viewing Flink Job Details
+
+ - Setting the Priority for a Flink Job
+
+ - Enabling Dynamic Scaling for Flink Jobs
+
+ - Querying Logs for Flink Jobs
+
+ - Common Operations of Flink Jobs
+
+
+Parent topic: Submitting a Flink Job Using DLIWhat Is Data Lake Insight?
+What Is Data Lake Insight
DLI Introduction
Data Lake Insight (DLI) is a serverless data processing and analysis service fully compatible with Apache Spark and Apache Flink ecosystems. It frees you from managing any servers.
DLI supports standard SQL and is compatible with Spark SQL and Flink SQL. It also supports multiple access modes, and is compatible with mainstream data formats. DLI supports SQL statements and Spark applications for heterogeneous data sources, including CloudTable, RDS, GaussDB(DWS), CSS, OBS, custom databases on ECSs, and offline databases.
Functions
You can query and analyze heterogeneous data sources such as RDS, and GaussDB(DWS) on the cloud using access methods, such as visualized interface, RESTful API, JDBC, and Beeline. The data format is compatible with five mainstream data formats: CSV, JSON, Parquet, and ORC.
-- Basic functions
- You can use standard SQL statements to query in SQL jobs.
- Flink jobs support Flink SQL online analysis. Aggregation functions such as Window and Join, geographic functions, and CEP functions are supported. SQL is used to express service logic, facilitating service implementation.
- For spark jobs, fully-managed Spark computing can be performed. You can submit computing tasks through interactive sessions or in batch to analyze data in the fully managed Spark queues.
- Federated analysis of heterogeneous data sources
- Spark datasource connection: Data sources such as GaussDB(DWS), RDS, and CSS can be accessed through DLI.
- Interconnection with multiple cloud services is supported in Flink jobs to form a rich stream ecosystem. The DLI stream ecosystem consists of cloud service ecosystems and open source ecosystems.
- Cloud service ecosystem: DLI can interconnect with other services in Flink SQL. You can directly use SQL to read and write data from cloud services.
- Open-source ecosystems: After connections to other VPCs are established through datasource connections, you can access all data sources and output targets (such as Kafka, HBase, and Elasticsearch) supported by Flink and Spark in your dedicated DLI queue.
- Basic functions
- Create a database and a table.
- DLI table
- Enter a tag key in the text box.
- Enter a tag key in the text box.
- Grant permissions to a new user.
- Assign permissions to a new user.
- To preview data on the Data Management page:
- Create an Enhanced Datasource Connection
- To create a table on the Data Management page:
- You can create a table on either the Databases and Tables page or the SQL Editor page.