forked from docs/doc-exports
Su, Xiaomeng
04ae6632a7
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
114 lines
10 KiB
HTML
114 lines
10 KiB
HTML
<a name="dli_02_0310"></a><a name="dli_02_0310"></a>
|
|
|
|
<h1 class="topictitle1">Creating and Submitting a Flink Job</h1>
|
|
<div id="body1602473883611"><div class="section" id="dli_02_0310__section851316282599"><h4 class="sectiontitle">Scenario</h4><p id="dli_02_0310__p1822151903">This section describes how to create and run a user-defined Flink job using APIs.</p>
|
|
</div>
|
|
<div class="section" id="dli_02_0310__section155144213214"><h4 class="sectiontitle">Notes and Constraints</h4><ul id="dli_02_0310__ul759612191669"><li id="dli_02_0310__li45963199610">It takes 6 to 10 minutes to start a job using a new queue for the first time.</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_02_0310__section13856162971"><h4 class="sectiontitle">Involved APIs</h4><ul id="dli_02_0310__ul1862251633812"><li id="dli_02_0310__li13929164064312"><a href="dli_02_0326.html">Creating an Elastic Resource Pool</a>: Create an elastic resource pool.</li><li id="dli_02_0310__li492974014318"><a href="dli_02_0194.html">Creating a Queue</a>: Create queues within the elastic resource pool.</li><li id="dli_02_0310__li46223161388"><a href="dli_02_0130.html">Uploading a Package Group (Deprecated)</a>: Upload the resource package required by the Flink custom job.</li><li id="dli_02_0310__li1137901761318"><a href="dli_02_0172.html">Querying Resource Packages in a Group (Deprecated)</a>: Check whether the uploaded resource package is correct.</li><li id="dli_02_0310__li9641746152414"><a href="dli_02_0230.html">Creating a Flink Jar Job</a> Create a user-defined Flink job.</li><li id="dli_02_0310__li2034416201925"><a href="dli_02_0233.html">Batch Running Jobs</a>: Run a user-defined Flink job.</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_02_0310__section2742155213719"><h4 class="sectiontitle">Procedure</h4><ol id="dli_02_0310__ol454620011213"><li id="dli_02_0310__li33424532067">Create an elastic resource pool named <strong id="dli_02_0310__b16365134201514">elastic_pool_dli</strong>.<ul id="dli_02_0310__ul12410110912"><li id="dli_02_0310__li11419115919">API<p id="dli_02_0310__p164191891"><a name="dli_02_0310__li11419115919"></a><a name="li11419115919"></a>URI format: POST /v3/{project_id}/elastic-resource-pools</p>
|
|
<ul id="dli_02_0310__ul1241011696"><li id="dli_02_0310__li859113919">Obtain the value of <em id="dli_02_0310__i1948963821519">{project_id}</em> by referring to <a href="dli_02_0183.html">Obtaining a Project ID</a>.</li><li id="dli_02_0310__li15516118917">For details about the request parameters, see <a href="dli_02_0326.html">Creating an Elastic Resource Pool</a>.</li></ul>
|
|
</li><li id="dli_02_0310__li12516114916">Example request<ul id="dli_02_0310__ul751915920"><li id="dli_02_0310__li566016160313">Description: Create an elastic resource pool named <strong id="dli_02_0310__b17530644141518">elastic_pool_dli</strong> in the project whose ID is <strong id="dli_02_0310__b135311044171516">48cc2c48765f481480c7db940d6409d1</strong>.</li><li id="dli_02_0310__li35614911">Example URL: POST https://{endpoint}/v3/48cc2c48765f481480c7db940d6409d1/elastic-resource-pools</li><li id="dli_02_0310__li45411199">Body:<pre class="screen" id="dli_02_0310__screen12372049928">{
|
|
"elastic_resource_pool_name" : "elastic_pool_dli",
|
|
"description" : "test",
|
|
"cidr_in_vpc" : "172.16.0.0/14",
|
|
"charging_mode" : "1",
|
|
"max_cu" : 64,
|
|
"min_cu" : 64
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0310__li151915918">Example response<pre class="screen" id="dli_02_0310__screen75911798">{
|
|
"is_success": true,
|
|
"message": ""
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0310__li1353820471116">Create a queue named <strong id="dli_02_0310__b1986345141516">queue1</strong> in the elastic resource pool.<ul id="dli_02_0310__ul25387411111"><li id="dli_02_0310__li3538940114">API<p id="dli_02_0310__p353814419114"><a name="dli_02_0310__li3538940114"></a><a name="li3538940114"></a>URI format: POST /v1.0/{project_id}/queues</p>
|
|
<ul id="dli_02_0310__ul1853810411114"><li id="dli_02_0310__li55381045113">Obtain the value of <em id="dli_02_0310__i1511913573153">{project_id}</em> by referring to <a href="dli_02_0183.html">Obtaining a Project ID</a>.</li><li id="dli_02_0310__li35388431118">For details about the request parameters, see <a href="dli_02_0194.html">Creating a Queue</a>.</li></ul>
|
|
</li><li id="dli_02_0310__li7538164171114">Example request<ul id="dli_02_0310__ul125381147119"><li id="dli_02_0310__li1853864171117">Description: Create an elastic resource pool named <strong id="dli_02_0310__b196516241611">queue1</strong> in the project whose ID is <strong id="dli_02_0310__b1065119214161">48cc2c48765f481480c7db940d6409d1</strong>.</li><li id="dli_02_0310__li1053817421112">Example URL: POST https://{<em id="dli_02_0310__i9799241163">endpoint</em>}/v1.0/48cc2c48765f481480c7db940d6409d1/queues</li><li id="dli_02_0310__li15383461112">Body:<pre class="screen" id="dli_02_0310__screen105384441117">{
|
|
"queue_name": "queue1",
|
|
"queue_type": "sql",
|
|
"description": "test",
|
|
"cu_count": 16,
|
|
"enterprise_project_id": "elastic_pool_dli"
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0310__li753818411110">Example response<pre class="screen" id="dli_02_0310__screen253811441117">{
|
|
"is_success": true,
|
|
"message": ""
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0310__li117291344122510">Upload the resource package of the user-defined Flink job. For details, see <a href="dli_02_0309.html#dli_02_0309__li117291344122510">3</a>.</li><li id="dli_02_0310__li970315312304">Query resource packages in a group. For details, see <a href="dli_02_0309.html#dli_02_0309__li970315312304">4</a>.</li><li id="dli_02_0310__li125211016104016">Create a custom flink job.<ul id="dli_02_0310__ul26571210114414"><li id="dli_02_0310__li146576102444">API<p id="dli_02_0310__p10657111019442"><a name="dli_02_0310__li146576102444"></a><a name="li146576102444"></a>URI format: POST /v1.0/{<em id="dli_02_0310__i18298164714552">project_id</em>}/streaming/flink-jobs</p>
|
|
<ul id="dli_02_0310__ul56581107449"><li id="dli_02_0310__li17658191013442">Obtain the value of {<em id="dli_02_0310__i10529144915517">project_id</em>} from <a href="dli_02_0183.html">Obtaining a Project ID</a>.</li><li id="dli_02_0310__li3658201018442">For details about the request parameters, see <a href="dli_02_0028.html">Creating a Database (Deprecated)</a>.</li></ul>
|
|
</li><li id="dli_02_0310__li10658410164410">Example request<ul id="dli_02_0310__ul14658161015442"><li id="dli_02_0310__li865851014449">Description: Create a user-defined Flink job in the project whose ID is <strong id="dli_02_0310__b2315144145612">48cc2c48765f481480c7db940d6409d1</strong>.</li><li id="dli_02_0310__li6658171074411">Example URL: POST https://{<em id="dli_02_0310__i353764845616">endpoint</em>}/v1.0/48cc2c48765f481480c7db940d6409d1/streaming/flink-jobs</li><li id="dli_02_0310__li165811084419">Body:<pre class="screen" id="dli_02_0310__screen3658131064420">{
|
|
"name": "test",
|
|
"desc": "job for test",
|
|
"queue_name": "queue1",
|
|
"manager_cu_number": 1,
|
|
"cu_number": 2,
|
|
"parallel_number": 1,
|
|
"tm_cus": 1,
|
|
"tm_slot_num": 1,
|
|
"log_enabled": true,
|
|
"obs_bucket": "bucketName",
|
|
"smn_topic": "topic",
|
|
"main_class": "org.apache.flink.examples.streaming.JavaQueueStream",
|
|
"restart_when_exception": false,
|
|
"entrypoint": "javaQueueStream.jar",
|
|
"entrypoint_args":"-windowSize 2000 -rate3",
|
|
"dependency_jars": [
|
|
"myGroup/test.jar",
|
|
"myGroup/test1.jar"
|
|
],
|
|
"dependency_files": [
|
|
"myGroup/test.csv",
|
|
"myGroup/test1.csv"
|
|
]
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0310__li2065912101446">Example response<pre class="screen" id="dli_02_0310__screen765961024418">{
|
|
"is_success": true,
|
|
"message": "A Flink job is created successfully.",
|
|
"job": {
|
|
"job_id": 138,
|
|
"status_name": "job_init",
|
|
"status_desc": ""
|
|
}
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0310__li18904826102815">Run jobs in batches.<ul id="dli_02_0310__ul178791154132816"><li id="dli_02_0310__li1187919544287">API<p id="dli_02_0310__p198791154192815"><a name="dli_02_0310__li1187919544287"></a><a name="li1187919544287"></a>URI format: POST /v1.0/{<em id="dli_02_0310__i396792595714">project_id</em>}/streaming/jobs/run</p>
|
|
<ul id="dli_02_0310__ul158791254132815"><li id="dli_02_0310__li7879854172815">Obtain the value of {<em id="dli_02_0310__i6586133245719">project_id</em>} from <a href="dli_02_0183.html">Obtaining a Project ID</a>.</li><li id="dli_02_0310__li13879115411285">For details about the request parameters, see <a href="dli_02_0233.html">Batch Running Jobs</a>.</li></ul>
|
|
</li><li id="dli_02_0310__li7879185418289">Example request<ul id="dli_02_0310__ul1287995452812"><li id="dli_02_0310__li118791554122819">Description: Run the jobs whose <strong id="dli_02_0310__b24295912573">job_id</strong> is <strong id="dli_02_0310__b1195014415820">298765</strong> and <strong id="dli_02_0310__b1150357135815">298766</strong> in the project whose ID is <strong id="dli_02_0310__b1951611195810">48cc2c48765f481480c7db940d6409d1</strong>.</li><li id="dli_02_0310__li6879145416284">Example URL: POST https://{<em id="dli_02_0310__i1862852495815">endpoint</em>}/v1.0/48cc2c48765f481480c7db940d6409d1/streaming/jobs/run</li><li id="dli_02_0310__li16880154182814">Body:<pre class="screen" id="dli_02_0310__screen488075411287">{
|
|
"job_ids": [131,130,138,137],
|
|
"resume_savepoint": true
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0310__li198801754192819">Example response<pre class="screen" id="dli_02_0310__screen78801654182815">[
|
|
{
|
|
"is_success": "true",
|
|
"message": "The request for submitting DLI jobs is delivered successfully."
|
|
},
|
|
{
|
|
"is_success": "true",
|
|
"message": "The request for submitting DLI jobs is delivered successfully."
|
|
},
|
|
{
|
|
"is_success": "true",
|
|
"message": "The request for submitting DLI jobs is delivered successfully."
|
|
},
|
|
{
|
|
"is_success": "true",
|
|
"message": "The request for submitting DLI jobs is delivered successfully."
|
|
}
|
|
]</pre>
|
|
</li></ul>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_02_0306.html">Getting Started</a></div>
|
|
</div>
|
|
</div>
|
|
|