forked from docs/doc-exports
Hasko, Vladimir
cfc48b3aed
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-committed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
181 lines
28 KiB
HTML
181 lines
28 KiB
HTML
<a name="dli_09_0150"></a><a name="dli_09_0150"></a>
|
|
|
|
<h1 class="topictitle1">Flink Jar Job Examples</h1>
|
|
<div id="body1575018104801"><div class="section" id="dli_09_0150__section17471021191520"><h4 class="sectiontitle">Overview</h4><p id="dli_09_0150__p1273772691510">You can perform secondary development based on Flink APIs to build your own Jar packages and submit them to the DLI queues to interact with data sources such as MRS Kafka, HBase, Hive, HDFS, GaussDB(DWS), and DCS.</p>
|
|
<p id="dli_09_0150__p77421045172410">This section describes how to interact with MRS through a custom job.</p>
|
|
</div>
|
|
<div class="section" id="dli_09_0150__en-us_topic_0111499973_section13926185035213"><h4 class="sectiontitle">Environment Preparations</h4><ol id="dli_09_0150__ol99958223243"><li id="dli_09_0150__li10633441112310">Log in to the MRS management console and create an MRS cluster. During the creation, enable <strong id="dli_09_0150__b18224216569">Kerberos Authentication</strong> and select <strong id="dli_09_0150__b1213217195619">Kafka</strong>, <strong id="dli_09_0150__b439321845617">HBase</strong>, and <strong id="dli_09_0150__b1873501918560">HDFS</strong>. For details about how to create an MRS cluster, see "Buying a Custom Cluster" in .</li><li id="dli_09_0150__li87431314112514">Enable the UDP/TCP port in the security group rule. For details, see "Adding a Security Group Rule" in .</li><li id="dli_09_0150__li6866558122720">Log in to <strong id="dli_09_0150__b14575172010446">MRS Manager</strong>.<ol type="a" id="dli_09_0150__ol75828202811"><li id="dli_09_0150__li1918114242283">Create a machine-machine account. Ensure that you have the <span class="parmvalue" id="dli_09_0150__parmvalue13251114942815"><b>hdfs_admin</b></span> and <span class="parmvalue" id="dli_09_0150__parmvalue1392535510282"><b>hbase_admin</b></span> permissions. Download the user authentication credentials, including the <span class="filepath" id="dli_09_0150__filepath67547182917"><b>user.keytab</b></span> and <span class="parmvalue" id="dli_09_0150__parmvalue1966541492918"><b>krb5.conf</b></span> files.<div class="note" id="dli_09_0150__note16701839363"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0150__p2070839363">The <strong id="dli_09_0150__b17263175134117">.keytab</strong> file of a human-machine account becomes invalid when the user password expires. Use a machine-machine account for configuration.</p>
|
|
</div></div>
|
|
</li><li id="dli_09_0150__li189811543011">Click <span class="uicontrol" id="dli_09_0150__uicontrol181001358301"><b>Services</b></span>, download the client, and click <span class="uicontrol" id="dli_09_0150__uicontrol1710018519303"><b>OK</b></span>.</li><li id="dli_09_0150__li1813143052914">Download the configuration files from the MRS node, including <span class="filepath" id="dli_09_0150__filepath1129915320304"><b>hbase-site.xml</b></span> and <span class="filepath" id="dli_09_0150__filepath13117123883017"><b>hiveclient.properties</b></span>.</li></ol>
|
|
</li><li id="dli_09_0150__li16906162643110">Create a dedicated DLI queue.</li><li id="dli_09_0150__li149966221248">Set up an enhanced datasource connection between the DLI dedicated queue and the MRS cluster and configure security group rules based on the site requirements.<p id="dli_09_0150__p290820132116"><a name="dli_09_0150__li149966221248"></a><a name="li149966221248"></a>For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_09_0150__filepath423863716222"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_09_0150__cite56a435148cb24025b0a8eb65b6b72d61190240">Data Lake Insight User Guide</cite></i>.</p>
|
|
<p id="dli_09_0150__p18909401219">For details about how to configure security group rules, see "Security Group" in <em id="dli_09_0150__i151264520213938">Virtual Private Cloud User Guide</em>.</p>
|
|
</li><li id="dli_09_0150__li1899652210245">Obtain the IP address and domain name mapping of all nodes in the MRS cluster, and configure the host mapping in the host information of the DLI cross-source connection.<p id="dli_09_0150__p1290940152113"><a name="dli_09_0150__li1899652210245"></a><a name="li1899652210245"></a>For details about how to add an IP-domain mapping, see <span class="filepath" id="dli_09_0150__filepath790913042114"><b>Modifying the Host Information</b></span> in the <i><cite id="dli_09_0150__citebf8bd59b34f042e6b669557e1968e896190241">Data Lake Insight User Guide</cite></i>.</p>
|
|
<div class="note" id="dli_09_0150__note26371439184311"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0150__p4660123954319">If the Kafka server listens on the port using <strong id="dli_09_0150__b72566147213938">hostname</strong>, you need to add the mapping between the hostname and IP address of the Kafka Broker node to the DLI queue. Contact the Kafka service deployment personnel to obtain the hostname and IP address of the Kafka Broker node.</p>
|
|
</div></div>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0150__section13123183424115"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_09_0150__ul198451926172619"><li id="dli_09_0150__li51821947175614">Ensure that a dedicated queue has been created.</li><li id="dli_09_0150__li17845142616266">When running a Flink Jar job, you need to build the secondary development application code into a JAR package and upload it to the created OBS bucket. On the DLI console, choose <strong id="dli_09_0150__b1319012513154">Data Management</strong> > <strong id="dli_09_0150__b38611353101512">Package Management</strong> to create a package.<div class="note" id="dli_09_0150__note38679553417"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0150__p108692554412">DLI does not support the download function. If you need to modify the uploaded data file, edit the local file and upload it again.</p>
|
|
</div></div>
|
|
</li><li id="dli_09_0150__li18845126192614">Flink dependencies have been built in the DLI server and security hardening has been performed based on the open-source community version. <strong id="dli_09_0150__b1850973753816">To prevent dependency compatibility issues or log output and dump issues, exclude the following files during packaging</strong>:<ul id="dli_09_0150__ul963732711519"><li id="dli_09_0150__li189511751712">Built-in dependencies (or <strong id="dli_09_0150__b95909302395">set the package dependency scope to "provided" in Maven or sbt</strong>)</li><li id="dli_09_0150__li2764181114818">Log configuration files (for example, <strong id="dli_09_0150__b8644118112612">log4j.properties</strong> or <strong id="dli_09_0150__b1479282312269">logback.xml</strong>)</li><li id="dli_09_0150__li17126142512512">JAR packages for log output implementation (for example, <strong id="dli_09_0150__b864185113263">log4j</strong>)</li></ul>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_09_0150__section1297804413418"><h4 class="sectiontitle">How to Use</h4><ol id="dli_09_0150__ol16424145713139"><li id="dli_09_0150__li13664141317385"><span>In the left navigation pane of the DLI management console, choose <span class="menucascade" id="dli_09_0150__menucascade1580122372914"><b><span class="uicontrol" id="dli_09_0150__uicontrol77402311291">Job Management</span></b></span> > <span class="menucascade" id="dli_09_0150__menucascade58122312911"><b><span class="uicontrol" id="dli_09_0150__uicontrol38122316294">Flink Jobs</span></b></span>. The <span class="wintitle" id="dli_09_0150__wintitle1181152317296"><b>Flink Jobs</b></span> page is displayed.</span></li><li id="dli_09_0150__li1832715401765"><span>In the upper right corner of the <span class="wintitle" id="dli_09_0150__wintitle6130142611292"><b>Flink Jobs</b></span> page, click <span class="uicontrol" id="dli_09_0150__uicontrol713062615296"><b>Create Job</b></span>.</span></li><li id="dli_09_0150__li767432417017"><span>Configure job parameters.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_09_0150__table1967419241707" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Job parameters</caption><thead align="left"><tr id="dli_09_0150__row13672324503"><th align="left" class="cellrowborder" valign="top" width="14.48%" id="mcps1.3.4.2.3.2.1.2.3.1.1"><p id="dli_09_0150__p86725241806">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="85.52%" id="mcps1.3.4.2.3.2.1.2.3.1.2"><p id="dli_09_0150__p3672924102">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_09_0150__row206721244013"><td class="cellrowborder" valign="top" width="14.48%" headers="mcps1.3.4.2.3.2.1.2.3.1.1 "><p id="dli_09_0150__p176729240019">Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="85.52%" headers="mcps1.3.4.2.3.2.1.2.3.1.2 "><p id="dli_09_0150__p14672192412017">Select <strong id="dli_09_0150__b8785124163715">Flink Jar</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row96731524306"><td class="cellrowborder" valign="top" width="14.48%" headers="mcps1.3.4.2.3.2.1.2.3.1.1 "><p id="dli_09_0150__p6672824703">Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="85.52%" headers="mcps1.3.4.2.3.2.1.2.3.1.2 "><p id="dli_09_0150__p667202410018">Job name, which contains 1 to 57 characters and consists of only letters, digits, hyphens (-), and underscores (_).</p>
|
|
<div class="note" id="dli_09_0150__note10673624903"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_09_0150__p1167313243015">The job name must be globally unique.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row9673142411010"><td class="cellrowborder" valign="top" width="14.48%" headers="mcps1.3.4.2.3.2.1.2.3.1.1 "><p id="dli_09_0150__p26735248017">Description</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="85.52%" headers="mcps1.3.4.2.3.2.1.2.3.1.2 "><p id="dli_09_0150__p567317241018">Description of the job, which contains 0 to 512 characters.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="dli_09_0150__li569314247015"><span>Click <span class="uicontrol" id="dli_09_0150__uicontrol375464614579"><b>OK</b></span> to enter the <span class="menucascade" id="dli_09_0150__menucascade97601446105713"><b><span class="uicontrol" id="dli_09_0150__uicontrol376011464571">Edit</span></b></span> page.</span></li><li id="dli_09_0150__li106011835103311"><span>Select a queue. A Flink Jar job can run only on general queues.</span><p><div class="note" id="dli_09_0150__note218315571477"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0150__ul1483193318"><li id="dli_09_0150__li14853910315">A Flink Jar job can run only on a pre-created dedicated queue.</li><li id="dli_09_0150__li154851897310">If no dedicated queue is available in the <span class="parmname" id="dli_09_0150__parmname35132415015"><b>Queue</b></span> drop-down list, create a dedicated queue and bind it to the current user.</li></ul>
|
|
</div></div>
|
|
</p></li><li id="dli_09_0150__li8384853145814"><span>Upload the JAR package.</span><p><div class="p" id="dli_09_0150__p430083812272">The Flink version must be the same as that specified in the JAR package.
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_09_0150__table1513363610272" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Description</caption><thead align="left"><tr id="dli_09_0150__row1213213367278"><th align="left" class="cellrowborder" valign="top" width="18.94%" id="mcps1.3.4.2.6.2.1.1.2.3.1.1"><p id="dli_09_0150__p1213216363271">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="81.06%" id="mcps1.3.4.2.6.2.1.1.2.3.1.2"><p id="dli_09_0150__p1113273613277">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_09_0150__row131321036202712"><td class="cellrowborder" valign="top" width="18.94%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.1 "><p id="dli_09_0150__p113211369278">Application</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.06%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.2 "><p id="dli_09_0150__p18132133632713">User-defined package. Before selecting a package, upload the corresponding JAR package to the OBS bucket and create a package on the <strong id="dli_09_0150__b71371054328">Data Management</strong> > <strong id="dli_09_0150__b2098778123211">Package Management</strong> page.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row1213223616275"><td class="cellrowborder" valign="top" width="18.94%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.1 "><p id="dli_09_0150__p113273611278">Main Class</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.06%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.2 "><p id="dli_09_0150__p10132163612717">Name of the main class of the JAR package to be loaded, for example, <strong id="dli_09_0150__b34071137174620">KafkaMessageStreaming</strong>.</p>
|
|
<ul id="dli_09_0150__ul81321736172715"><li id="dli_09_0150__li181327364276"><strong id="dli_09_0150__b1527864216471">Default</strong>: The value is specified based on the <strong id="dli_09_0150__b11279144216471">Manifest</strong> file in the JAR package.</li><li id="dli_09_0150__li1313216365278"><strong id="dli_09_0150__b1891162017484">Manually assign</strong>: You must enter the class name and confirm the class arguments (separate arguments with spaces).</li></ul>
|
|
<div class="note" id="dli_09_0150__note12132183612716"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_09_0150__p1013214363279">When a class belongs to a package, the package path must be carried, for example, <strong id="dli_09_0150__b9719827164815">packagePath.KafkaMessageStreaming</strong>.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row121324365277"><td class="cellrowborder" valign="top" width="18.94%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.1 "><p id="dli_09_0150__p8132036132714">Class Arguments</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.06%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.2 "><p id="dli_09_0150__p1513293612277">List of arguments of a specified class. The arguments are separated by spaces.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row17133183602720"><td class="cellrowborder" valign="top" width="18.94%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.1 "><p id="dli_09_0150__p813211360271">JAR Package Dependencies</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.06%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.2 "><p id="dli_09_0150__p61321936152717">User-defined dependencies. Before selecting a package, upload the corresponding JAR package to the OBS bucket and create a JAR package on the <strong id="dli_09_0150__b19705111125511">Data Management</strong> > <strong id="dli_09_0150__b1370671119554">Package Management</strong> page.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row12133203615272"><td class="cellrowborder" valign="top" width="18.94%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.1 "><p id="dli_09_0150__p14133153692720">Other Dependencies</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.06%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.2 "><p id="dli_09_0150__p18133103619275">User-defined dependency files. Before selecting a file, upload the corresponding file to the OBS bucket and create a package of any type on the <strong id="dli_09_0150__b19303441105711">Data Management</strong> > <strong id="dli_09_0150__b230904165714">Package Management</strong> page.</p>
|
|
<p id="dli_09_0150__p31331536182710">You can add the following content to the application to access the corresponding dependency file: <span class="parmname" id="dli_09_0150__parmname17725101620119"><b>fileName</b></span> indicates the name of the file to be accessed, and <span class="parmname" id="dli_09_0150__parmname87251716915"><b>ClassName</b></span> indicates the name of the class that needs to access the file.</p>
|
|
<pre class="screen" id="dli_09_0150__screen3133183622713">ClassName.class.getClassLoader().getResource("userData/fileName")</pre>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row6133236122711"><td class="cellrowborder" valign="top" width="18.94%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.1 "><p id="dli_09_0150__p113343682716">Flink Version</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="81.06%" headers="mcps1.3.4.2.6.2.1.1.2.3.1.2 "><p id="dli_09_0150__p20133736142717">Before selecting a Flink version, you need to select the queue to which the Flink version belongs. Currently, the following versions are supported: 1.10.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</p></li><li id="dli_09_0150__li5501135844"><span>Configure job parameters.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_09_0150__table15596526413" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Parameter description</caption><thead align="left"><tr id="dli_09_0150__row145620521843"><th align="left" class="cellrowborder" valign="top" width="16.24%" id="mcps1.3.4.2.7.2.1.2.3.1.1"><p id="dli_09_0150__p35637523415">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="83.76%" id="mcps1.3.4.2.7.2.1.2.3.1.2"><p id="dli_09_0150__p156517526416">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_09_0150__row359118526411"><td class="cellrowborder" valign="top" width="16.24%" headers="mcps1.3.4.2.7.2.1.2.3.1.1 "><p id="dli_09_0150__p13592195214418">CUs</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.76%" headers="mcps1.3.4.2.7.2.1.2.3.1.2 "><p id="dli_09_0150__p135951252143">One CU has one vCPU and 4 GB memory. The number of CUs ranges from 2 to 400.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row14595175219417"><td class="cellrowborder" valign="top" width="16.24%" headers="mcps1.3.4.2.7.2.1.2.3.1.1 "><p id="dli_09_0150__p12598165214420">Job Manager CUs</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.76%" headers="mcps1.3.4.2.7.2.1.2.3.1.2 "><p id="dli_09_0150__p19598952746">Set the number of CUs on a management unit. The value ranges from 1 to 4. The default value is <strong id="dli_09_0150__b1624613111611">1</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row959914521147"><td class="cellrowborder" valign="top" width="16.24%" headers="mcps1.3.4.2.7.2.1.2.3.1.1 "><p id="dli_09_0150__p860013529420">Parallelism</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.76%" headers="mcps1.3.4.2.7.2.1.2.3.1.2 "><p id="dli_09_0150__p060216521040">Maximum number of parallel operators in a job.</p>
|
|
<div class="note" id="dli_09_0150__note14602195219420"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_09_0150__ul1660513521547"><li id="dli_09_0150__li9605135211413">The value must be less than or equal to four times the number of compute units (CUs minus the number of job manager CUs).</li><li id="dli_09_0150__li1160518521148">You are advised to set this parameter to a value greater than that configured in the code. Otherwise, job submission may fail.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row1533272464411"><td class="cellrowborder" valign="top" width="16.24%" headers="mcps1.3.4.2.7.2.1.2.3.1.1 "><p id="dli_09_0150__p10188125819444">Task Manager Configuration</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.76%" headers="mcps1.3.4.2.7.2.1.2.3.1.2 "><p id="dli_09_0150__p164861439599">Whether to set Task Manager resource parameters.</p>
|
|
<p id="dli_09_0150__p656318135218">If this option is selected, you need to set the following parameters:</p>
|
|
<ul id="dli_09_0150__ul206731739155015"><li id="dli_09_0150__li1767343920508"><span class="parmname" id="dli_09_0150__parmname7511522575"><b>CU(s) per TM</b></span>: Number of resources occupied by each Task Manager.</li><li id="dli_09_0150__li8782174945010"><span class="parmname" id="dli_09_0150__parmname16744525979"><b>Slot(s) per TM</b></span>: Number of slots contained in each Task Manager.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row1760615210415"><td class="cellrowborder" valign="top" width="16.24%" headers="mcps1.3.4.2.7.2.1.2.3.1.1 "><p id="dli_09_0150__p97589379456">Save Job Log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.76%" headers="mcps1.3.4.2.7.2.1.2.3.1.2 "><p id="dli_09_0150__p175893714515">Whether to save the job running logs to OBS.</p>
|
|
<p id="dli_09_0150__p9822495517">If this option is selected, you need to set the following parameters:</p>
|
|
<p id="dli_09_0150__p135569492554"><span class="parmname" id="dli_09_0150__parmname23035318715"><b>OBS Bucket</b></span>: Select an OBS bucket to store user job logs. If the selected OBS bucket is not authorized, click <span class="uicontrol" id="dli_09_0150__uicontrol12063347713"><b>Authorize</b></span>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row13189114720391"><td class="cellrowborder" valign="top" width="16.24%" headers="mcps1.3.4.2.7.2.1.2.3.1.1 "><p id="dli_09_0150__p1178862351913">Alarm Generation upon Job Exception</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.76%" headers="mcps1.3.4.2.7.2.1.2.3.1.2 "><p id="dli_09_0150__p7788172313199">Whether to report job exceptions, for example, abnormal job running or exceptions due to an insufficient balance, to users via SMS or email.</p>
|
|
<p id="dli_09_0150__p7145192645819">If this option is selected, you need to set the following parameters:</p>
|
|
<p id="dli_09_0150__p4751959195511"><span class="parmname" id="dli_09_0150__parmname78311440378"><b>SMN Topic</b></span></p>
|
|
<p id="dli_09_0150__p2038424114597">Select a user-defined SMN topic. For details about how to customize SMN topics, see "Creating a Topic" in the <em id="dli_09_0150__i21327530717">Simple Message Notification User Guide</em>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_09_0150__row51981436589"><td class="cellrowborder" valign="top" width="16.24%" headers="mcps1.3.4.2.7.2.1.2.3.1.1 "><p id="dli_09_0150__p61982043145813">Auto Restart upon Exception</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.76%" headers="mcps1.3.4.2.7.2.1.2.3.1.2 "><p id="dli_09_0150__p6198174375810">Whether to enable automatic restart. If this function is enabled, jobs will be automatically restarted and restored when exceptions occur.</p>
|
|
<p id="dli_09_0150__p182651820924">If this option is selected, you need to set the following parameters:</p>
|
|
<ul id="dli_09_0150__ul12114713615"><li id="dli_09_0150__li182111871362"><span class="parmname" id="dli_09_0150__parmname1783351596"><b>Max. Retry Attempts</b></span>: maximum number of retry times upon an exception. The unit is <strong id="dli_09_0150__b99829405919">Times/hour</strong>.<ul id="dli_09_0150__ul1442116221967"><li id="dli_09_0150__li18351183619"><strong id="dli_09_0150__b3592194211916">Unlimited</strong>: The number of retries is unlimited.</li><li id="dli_09_0150__li132308401613"><strong id="dli_09_0150__b1146213472910">Limited</strong>: The number of retries is user-defined.</li></ul>
|
|
</li><li id="dli_09_0150__li564894516447"><span class="parmname" id="dli_09_0150__parmname2166850696"><b>Restore Job from Checkpoint</b></span>: Restore the job from the latest checkpoint.<p id="dli_09_0150__p9285367272">If you select this parameter, you also need to set <strong id="dli_09_0150__b78783541915">Checkpoint Path</strong>.</p>
|
|
<p id="dli_09_0150__p32853614273"><span class="parmname" id="dli_09_0150__parmname1719819981012"><b>Checkpoint Path</b></span>: Select the checkpoint saving path. The value must be the same as the checkpoint path you set in the application package. Note that the checkpoint path for each job must be unique. Otherwise, the checkpoint cannot be obtained. </p>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="dli_09_0150__li71021417181214"><span>Click <span class="uicontrol" id="dli_09_0150__uicontrol37991324151413"><b>Save</b></span> on the upper right of the page.</span></li><li id="dli_09_0150__li10835362220"><span>Click <span class="uicontrol" id="dli_09_0150__uicontrol414341431214"><b>Start</b></span> on the upper right side of the page. On the displayed <span class="wintitle" id="dli_09_0150__wintitle18149151412127"><b>Start Flink Job</b></span> page, confirm the job specifications, and click <span class="uicontrol" id="dli_09_0150__uicontrol191491614201212"><b>Start Now</b></span> to start the job.</span><p><p id="dli_09_0150__p923161020209">After the job is started, the system automatically switches to the <span class="menucascade" id="dli_09_0150__menucascade699012441145"><b><span class="uicontrol" id="dli_09_0150__uicontrol7990194441416">Flink Jobs</span></b></span> page, and the created job is displayed in the job list. You can view the job status in the <span class="menucascade" id="dli_09_0150__menucascade3990154421411"><b><span class="uicontrol" id="dli_09_0150__uicontrol14990644191410">Status</span></b></span> column. After a job is successfully submitted, the job status will change from <span class="menucascade" id="dli_09_0150__menucascade12454147201417"><b><span class="uicontrol" id="dli_09_0150__uicontrol1745434718142">Submitting</span></b></span> to <span class="menucascade" id="dli_09_0150__menucascade18455347171415"><b><span class="uicontrol" id="dli_09_0150__uicontrol144544473140">Running</span></b></span>. After the execution is complete, the message <strong id="dli_09_0150__b14110155010146">Completed</strong> is displayed.</p>
|
|
<p id="dli_09_0150__p62866773318">If the job status is <span class="menucascade" id="dli_09_0150__menucascade7660165281410"><b><span class="uicontrol" id="dli_09_0150__uicontrol865445216149">Submission failed</span></b></span> or <span class="menucascade" id="dli_09_0150__menucascade16660195211412"><b><span class="uicontrol" id="dli_09_0150__uicontrol13660165216141">Running exception</span></b></span>, the job submission failed or the job did not execute successfully. In this case, you can move the cursor over the status icon in the <span class="menucascade" id="dli_09_0150__menucascade137018269215"><b><span class="uicontrol" id="dli_09_0150__uicontrol2700826162117">Status</span></b></span> column of the job list to view the error details. You can click <span><img id="dli_09_0150__image127011526182111" src="en-us_image_0000001102485176.png"></span> to copy these details. After handling the fault based on the provided information, resubmit the job.</p>
|
|
<div class="note" id="dli_09_0150__note9631123617199"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0150__p063214367199">Other buttons are as follows:</p>
|
|
<p id="dli_09_0150__p94016231311"><strong id="dli_09_0150__b2318141417158">Save As</strong>: Save the created job as a new job.</p>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0150__section853864712718"><h4 class="sectiontitle">Related Operations</h4><ul id="dli_09_0150__ul165841538389"><li id="dli_09_0150__li8310754141019"><strong id="dli_09_0150__b033518228164">How Do I Configure Job Parameters?</strong><ol id="dli_09_0150__ol11336141713115"><li id="dli_09_0150__li1633613179111">In the Flink job list, select your desired job.</li><li id="dli_09_0150__li615520141149">Click <strong id="dli_09_0150__b135719120577">Edit</strong> in the <strong id="dli_09_0150__b10666114165713">Operation</strong> column.</li><li id="dli_09_0150__li1633210411614">Configure the parameters as needed.<p id="dli_09_0150__p1673101312162"><a name="dli_09_0150__li1633210411614"></a><a name="li1633210411614"></a>List of parameters of a specified class. The parameters are separated by spaces.</p>
|
|
<p id="dli_09_0150__p19593161242913">Parameter input format: --Key 1 Value 1 --Key 2 Value 2</p>
|
|
<p id="dli_09_0150__p14186175892914">For example, if you enter the following parameters on the console:</p>
|
|
<p id="dli_09_0150__p20943157143112">--bootstrap.server 192.168.168.xxx:9092</p>
|
|
<p id="dli_09_0150__p10463146163718">The parameters are parsed by ParameterTool as follows:</p>
|
|
<div class="fignone" id="dli_09_0150__fig14158230447"><span class="figcap"><b>Figure 1 </b>Parsed parameters</span><br><span><img id="dli_09_0150__image10402256113010" src="en-us_image_0000001618446021.png"></span></div>
|
|
</li></ol>
|
|
</li><li id="dli_09_0150__li1458418381818"><strong id="dli_09_0150__b10159937987">How Do I View Job Logs?</strong><ol id="dli_09_0150__ol322316321099"><li id="dli_09_0150__li17223173215919">In the Flink job list, click a job name to access its details page.</li><li id="dli_09_0150__li2735035291">Click the <strong id="dli_09_0150__b6241919216">Run Log</strong> tab and view job logs on the console.<p id="dli_09_0150__p881272015109">Only the latest run logs are displayed. For more information, see the OBS bucket that stores logs.</p>
|
|
</li></ol>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0202.html">Flink Jar Jobs</a></div>
|
|
</div>
|
|
</div>
|
|
|