Files
doc-exports/docs/dli/dev/CLASS.TXT.json
Su, Xiaomeng 89b6bedc33 dli_dev_0104_version
Reviewed-by: Rechenburg, Matthias <matthias.rechenburg@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-01-08 15:25:35 +00:00

560 lines
21 KiB
JSON

[
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Spark SQL Jobs",
"uri":"dli_09_0120.html",
"doc_type":"devg",
"p_code":"",
"code":"1"
},
{
"desc":"DLI allows you to use data stored on OBS. You can create OBS tables on DLI to access and process data in your OBS bucket.This section describes how to create an OBS table",
"product_code":"dli",
"title":"Using Spark SQL Jobs to Analyze OBS Data",
"uri":"dli_05_0044.html",
"doc_type":"devg",
"p_code":"1",
"code":"2"
},
{
"desc":"DLI allows you to use Hive user-defined functions (UDFs) to query data. UDFs take effect only on a single row of data and are applicable to inserting and deleting a singl",
"product_code":"dli",
"title":"Calling UDFs in Spark SQL Jobs",
"uri":"dli_09_0171.html",
"doc_type":"devg",
"p_code":"1",
"code":"3"
},
{
"desc":"You can use Hive User-Defined Table-Generating Functions (UDTF) to customize table-valued functions. Hive UDTFs are used for the one-in-multiple-out data operations. UDTF",
"product_code":"dli",
"title":"Calling UDTFs in Spark SQL Jobs",
"uri":"dli_09_0204.html",
"doc_type":"devg",
"p_code":"1",
"code":"4"
},
{
"desc":"DLI allows you to use a Hive User Defined Aggregation Function (UDAF) to process multiple rows of data. Hive UDAF is usually used together with groupBy. It is equivalent ",
"product_code":"dli",
"title":"Calling UDAFs in Spark SQL Jobs",
"uri":"dli_05_0062.html",
"doc_type":"devg",
"p_code":"1",
"code":"5"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Submitting a Spark SQL Job Using JDBC",
"uri":"dli_09_0123.html",
"doc_type":"devg",
"p_code":"1",
"code":"6"
},
{
"desc":"On DLI, you can connect to the server for data query in the Internet environment. In this case, you need to first obtain the connection information, including the endpoin",
"product_code":"dli",
"title":"Obtaining the Server Connection Address",
"uri":"dli_09_0124.html",
"doc_type":"devg",
"p_code":"6",
"code":"7"
},
{
"desc":"To connect to DLI using JDBC, download the specific driver file on the DLI management console.JDBC driver packageFor example, click dli-jdbc-1.1.2 to download the JDBC dr",
"product_code":"dli",
"title":"Downloading the JDBC Driver Package",
"uri":"dli_09_0125.html",
"doc_type":"devg",
"p_code":"6",
"code":"8"
},
{
"desc":"You need to be authenticated when using JDBC to create DLI driver connections.Currently, the JDBC supports authentication using the Access Key/Secret Key (AK/SK) or token",
"product_code":"dli",
"title":"Performing Authentication",
"uri":"dli_09_0126.html",
"doc_type":"devg",
"p_code":"6",
"code":"9"
},
{
"desc":"In Linux or Windows, you can connect to the DLI server using JDBC.The job submitted by using JDBC to connect to DLI runs on the Spark engine.DLI supports 13 data types. E",
"product_code":"dli",
"title":"Submitting a Job Using JDBC",
"uri":"dli_09_0127.html",
"doc_type":"devg",
"p_code":"6",
"code":"10"
},
{
"desc":"Relational Database Service (RDS) is a cloud-based web service that is reliable, scalable, easy to manage, and immediately ready for use. It can be deployed in single-nod",
"product_code":"dli",
"title":"Introduction to RDS",
"uri":"dli_09_0129.html",
"doc_type":"devg",
"p_code":"6",
"code":"11"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Flink OpenSource SQL Jobs",
"uri":"dli_09_0006.html",
"doc_type":"devg",
"p_code":"",
"code":"12"
},
{
"desc":"This guide provides reference for Flink 1.12 only.In this example, we aim to query information about top three most-clicked offerings in each hour from a set of real-time",
"product_code":"dli",
"title":"Reading Data from Kafka and Writing Data to RDS",
"uri":"dli_09_0009.html",
"doc_type":"devg",
"p_code":"12",
"code":"13"
},
{
"desc":"This guide provides reference for Flink 1.12 only.This example analyzes real-time vehicle driving data and collects statistics on data results that meet specific conditio",
"product_code":"dli",
"title":"Reading Data from Kafka and Writing Data to GaussDB(DWS)",
"uri":"dli_09_0010.html",
"doc_type":"devg",
"p_code":"12",
"code":"14"
},
{
"desc":"This guide provides reference for Flink 1.12 only.This example analyzes offering purchase data and collects statistics on data results that meet specific conditions. The ",
"product_code":"dli",
"title":"Reading Data from Kafka and Writing Data to Elasticsearch",
"uri":"dli_09_0011.html",
"doc_type":"devg",
"p_code":"12",
"code":"15"
},
{
"desc":"This guide provides reference for Flink 1.12 only.Change Data Capture (CDC) can synchronize incremental changes from the source database to one or more destinations. Duri",
"product_code":"dli",
"title":"Reading Data from MySQL CDC and Writing Data to GaussDB(DWS)",
"uri":"dli_09_0012.html",
"doc_type":"devg",
"p_code":"12",
"code":"16"
},
{
"desc":"This guide provides reference for Flink 1.12 only.Change Data Capture (CDC) can synchronize incremental changes from the source database to one or more destinations. Duri",
"product_code":"dli",
"title":"Reading Data from PostgreSQL CDC and Writing Data to GaussDB(DWS)",
"uri":"dli_09_0013.html",
"doc_type":"devg",
"p_code":"12",
"code":"17"
},
{
"desc":"If you need to configure high reliability for a Flink application, you can set the parameters when creating your Flink jobs.Create an SMN topic and add an email address o",
"product_code":"dli",
"title":"Configuring High-Reliability Flink Jobs (Automatic Restart upon Exceptions)",
"uri":"dli_09_0207.html",
"doc_type":"devg",
"p_code":"12",
"code":"18"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Flink Jar Jobs",
"uri":"dli_09_0202.html",
"doc_type":"devg",
"p_code":"",
"code":"19"
},
{
"desc":"Built on Flink and Spark, the stream ecosystem is fully compatible with the open-source Flink, Storm, and Spark APIs. It is enhanced in features and improved in performan",
"product_code":"dli",
"title":"Stream Ecosystem",
"uri":"dli_09_0162.html",
"doc_type":"devg",
"p_code":"19",
"code":"20"
},
{
"desc":"You can perform secondary development based on Flink APIs to build your own Jar packages and submit them to the DLI queues to interact with data sources such as MRS Kafka",
"product_code":"dli",
"title":"Flink Jar Job Examples",
"uri":"dli_09_0150.html",
"doc_type":"devg",
"p_code":"19",
"code":"21"
},
{
"desc":"DLI allows you to use a custom JAR package to run Flink jobs and write data to OBS. This section describes how to write processed Kafka data to OBS. You need to modify th",
"product_code":"dli",
"title":"Writing Data to OBS Using Flink Jar",
"uri":"dli_09_0191.html",
"doc_type":"devg",
"p_code":"19",
"code":"22"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Spark Jar Jobs",
"uri":"dli_09_0203.html",
"doc_type":"devg",
"p_code":"",
"code":"23"
},
{
"desc":"DLI is fully compatible with open-source Apache Spark and allows you to import, query, analyze, and process job data by programming. This section describes how to write a",
"product_code":"dli",
"title":"Using Spark Jar Jobs to Read and Query OBS Data",
"uri":"dli_09_0205.html",
"doc_type":"devg",
"p_code":"23",
"code":"24"
},
{
"desc":"DLI allows you to develop a program to create Spark jobs for operations related to databases, DLI or OBS tables, and table data. This example demonstrates how to develop ",
"product_code":"dli",
"title":"Using the Spark Job to Access DLI Metadata",
"uri":"dli_09_0176.html",
"doc_type":"devg",
"p_code":"23",
"code":"25"
},
{
"desc":"DLI Spark-submit is a command line tool used to submit Spark jobs to the DLI server. This tool provides command lines compatible with open-source Spark.Before using DLI B",
"product_code":"dli",
"title":"Using Spark-submit to Submit a Spark Jar Job",
"uri":"dli_09_0122.html",
"doc_type":"devg",
"p_code":"23",
"code":"26"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Using Spark Jobs to Access Data Sources of Datasource Connections",
"uri":"dli_09_0019.html",
"doc_type":"devg",
"p_code":"23",
"code":"27"
},
{
"desc":"DLI supports the native Spark DataSource capability and other extended capabilities. You can use SQL statements or Spark jobs to access other data storage services and im",
"product_code":"dli",
"title":"Overview",
"uri":"dli_09_0020.html",
"doc_type":"devg",
"p_code":"27",
"code":"28"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Connecting to CSS",
"uri":"dli_09_0089.html",
"doc_type":"devg",
"p_code":"27",
"code":"29"
},
{
"desc":"The Elasticsearch 6.5.4 and later versions provided by CSS provides the security settings. Once the function is enabled, CSS provides identity authentication, authorizati",
"product_code":"dli",
"title":"CSS Security Cluster Configuration",
"uri":"dli_09_0189.html",
"doc_type":"devg",
"p_code":"29",
"code":"30"
},
{
"desc":"A datasource connection has been created on the DLI management console.Development descriptionConstructing dependency information and creating a Spark sessionImport depen",
"product_code":"dli",
"title":"Scala Example Code",
"uri":"dli_09_0061.html",
"doc_type":"devg",
"p_code":"29",
"code":"31"
},
{
"desc":"A datasource connection has been created on the DLI management console.Development descriptionCode implementationImport dependency packages.from __future__ import print_f",
"product_code":"dli",
"title":"PySpark Example Code",
"uri":"dli_09_0090.html",
"doc_type":"devg",
"p_code":"29",
"code":"32"
},
{
"desc":"A datasource connection has been created on the DLI management console.Development descriptionCode implementationConstructing dependency information and creating a Spark ",
"product_code":"dli",
"title":"Java Example Code",
"uri":"dli_09_0190.html",
"doc_type":"devg",
"p_code":"29",
"code":"33"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Connecting to GaussDB(DWS)",
"uri":"dli_09_0086.html",
"doc_type":"devg",
"p_code":"27",
"code":"34"
},
{
"desc":"This section provides Scala example code that demonstrates how to use a Spark job to access data from the GaussDB(DWS) data source.A datasource connection has been create",
"product_code":"dli",
"title":"Scala Example Code",
"uri":"dli_09_0069.html",
"doc_type":"devg",
"p_code":"34",
"code":"35"
},
{
"desc":"This section provides PySpark example code that demonstrates how to use a Spark job to access data from the GaussDB(DWS) data source.A datasource connection has been crea",
"product_code":"dli",
"title":"PySpark Example Code",
"uri":"dli_09_0087.html",
"doc_type":"devg",
"p_code":"34",
"code":"36"
},
{
"desc":"This section provides Java example code that demonstrates how to use a Spark job to access data from the GaussDB(DWS) data source.A datasource connection has been created",
"product_code":"dli",
"title":"Java Example Code",
"uri":"dli_09_0199.html",
"doc_type":"devg",
"p_code":"34",
"code":"37"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Connecting to HBase",
"uri":"dli_09_0077.html",
"doc_type":"devg",
"p_code":"27",
"code":"38"
},
{
"desc":"Create a datasource connection on the DLI management console.Add the /etc/hosts information of MRS cluster nodes to the host file of the DLI queue.For details, see sectio",
"product_code":"dli",
"title":"MRS Configuration",
"uri":"dli_09_0196.html",
"doc_type":"devg",
"p_code":"38",
"code":"39"
},
{
"desc":"The CloudTable HBase and MRS HBase can be connected to DLI as data sources.PrerequisitesA datasource connection has been created on the DLI management console. For detail",
"product_code":"dli",
"title":"Scala Example Code",
"uri":"dli_09_0063.html",
"doc_type":"devg",
"p_code":"38",
"code":"40"
},
{
"desc":"The CloudTable HBase and MRS HBase can be connected to DLI as data sources.PrerequisitesA datasource connection has been created on the DLI management console.Hard-coded ",
"product_code":"dli",
"title":"PySpark Example Code",
"uri":"dli_09_0078.html",
"doc_type":"devg",
"p_code":"38",
"code":"41"
},
{
"desc":"This example applies only to MRS HBase.PrerequisitesA datasource connection has been created and bound to a queue on the DLI management console.Hard-coded or plaintext pa",
"product_code":"dli",
"title":"Java Example Code",
"uri":"dli_09_0197.html",
"doc_type":"devg",
"p_code":"38",
"code":"42"
},
{
"desc":"SymptomThe Spark job fails to be executed, and the job log indicates that the Java server connection or container fails to be started.The Spark job fails to be executed, ",
"product_code":"dli",
"title":"Troubleshooting",
"uri":"dli_09_0198.html",
"doc_type":"devg",
"p_code":"38",
"code":"43"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Connecting to OpenTSDB",
"uri":"dli_09_0080.html",
"doc_type":"devg",
"p_code":"27",
"code":"44"
},
{
"desc":"The CloudTable OpenTSDB and MRS OpenTSDB can be connected to DLI as data sources.PrerequisitesA datasource connection has been created on the DLI management console.Hard-",
"product_code":"dli",
"title":"Scala Example Code",
"uri":"dli_09_0065.html",
"doc_type":"devg",
"p_code":"44",
"code":"45"
},
{
"desc":"The CloudTable OpenTSDB and MRS OpenTSDB can be connected to DLI as data sources.PrerequisitesA datasource connection has been created on the DLI management console.Hard-",
"product_code":"dli",
"title":"PySpark Example Code",
"uri":"dli_09_0081.html",
"doc_type":"devg",
"p_code":"44",
"code":"46"
},
{
"desc":"This example applies only to MRS OpenTSDB.PrerequisitesA datasource connection has been created and bound to a queue on the DLI management console.Hard-coded or plaintext",
"product_code":"dli",
"title":"Java Example Code",
"uri":"dli_09_0193.html",
"doc_type":"devg",
"p_code":"44",
"code":"47"
},
{
"desc":"SymptomA Spark job fails to be executed and \"No respond\" is displayed in the job log.A Spark job fails to be executed and \"No respond\" is displayed in the job log.Solutio",
"product_code":"dli",
"title":"Troubleshooting",
"uri":"dli_09_0195.html",
"doc_type":"devg",
"p_code":"44",
"code":"48"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Connecting to RDS",
"uri":"dli_09_0083.html",
"doc_type":"devg",
"p_code":"27",
"code":"49"
},
{
"desc":"PrerequisitesA datasource connection has been created and bound to a queue on the DLI management console.Hard-coded or plaintext passwords pose significant security risks",
"product_code":"dli",
"title":"Scala Example Code",
"uri":"dli_09_0067.html",
"doc_type":"devg",
"p_code":"49",
"code":"50"
},
{
"desc":"PrerequisitesA datasource connection has been created and bound to a queue on the DLI management console.Hard-coded or plaintext passwords pose significant security risks",
"product_code":"dli",
"title":"PySpark Example Code",
"uri":"dli_09_0084.html",
"doc_type":"devg",
"p_code":"49",
"code":"51"
},
{
"desc":"PrerequisitesA datasource connection has been created and bound to a queue on the DLI management console.Hard-coded or plaintext passwords pose significant security risks",
"product_code":"dli",
"title":"Java Example Code",
"uri":"dli_09_0187.html",
"doc_type":"devg",
"p_code":"49",
"code":"52"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Connecting to Redis",
"uri":"dli_09_0093.html",
"doc_type":"devg",
"p_code":"27",
"code":"53"
},
{
"desc":"Redis supports only enhanced datasource connections.PrerequisitesAn enhanced datasource connection has been created on the DLI management console and bound to a queue in ",
"product_code":"dli",
"title":"Scala Example Code",
"uri":"dli_09_0094.html",
"doc_type":"devg",
"p_code":"53",
"code":"54"
},
{
"desc":"Redis supports only enhanced datasource connections.PrerequisitesAn enhanced datasource connection has been created on the DLI management console and bound to a queue in ",
"product_code":"dli",
"title":"PySpark Example Code",
"uri":"dli_09_0097.html",
"doc_type":"devg",
"p_code":"53",
"code":"55"
},
{
"desc":"Redis supports only enhanced datasource connections.PrerequisitesAn enhanced datasource connection has been created on the DLI management console and bound to a queue in ",
"product_code":"dli",
"title":"Java Example Code",
"uri":"dli_09_0100.html",
"doc_type":"devg",
"p_code":"53",
"code":"56"
},
{
"desc":"SymptomAfter the code is directly copied to the .py file, unexpected characters may exist after the backslashes (\\).After the code is directly copied to the .py file, une",
"product_code":"dli",
"title":"Troubleshooting",
"uri":"dli_09_0188.html",
"doc_type":"devg",
"p_code":"53",
"code":"57"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"dli",
"title":"Connecting to Mongo",
"uri":"dli_09_0113.html",
"doc_type":"devg",
"p_code":"27",
"code":"58"
},
{
"desc":"Mongo can be connected only through enhanced datasource connections.PrerequisitesAn enhanced datasource connection has been created on the DLI management console and boun",
"product_code":"dli",
"title":"Scala Example Code",
"uri":"dli_09_0114.html",
"doc_type":"devg",
"p_code":"58",
"code":"59"
},
{
"desc":"Mongo can be connected only through enhanced datasource connections.PrerequisitesAn enhanced datasource connection has been created on the DLI management console and boun",
"product_code":"dli",
"title":"PySpark Example Code",
"uri":"dli_09_0117.html",
"doc_type":"devg",
"p_code":"58",
"code":"60"
},
{
"desc":"Mongo can be connected only through enhanced datasource connections.PrerequisitesAn enhanced datasource connection has been created on the DLI management console and boun",
"product_code":"dli",
"title":"Java Example Code",
"uri":"dli_09_0110.html",
"doc_type":"devg",
"p_code":"58",
"code":"61"
},
{
"desc":"HUAWEI CLOUD Help Center presents technical documents to help you quickly get started with HUAWEI CLOUD services. The technical documents include Service Overview, Price Details, Purchase Guide, User Guide, API Reference, Best Practices, FAQs, and Videos.",
"product_code":"",
"title":"Change History",
"uri":"dli_09_00001.html",
"doc_type":"",
"p_code":"",
"code":"62"
}
]