forked from docs/doc-exports
chenxiaoxiong
f9e2808b7c
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com> Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
361 lines
59 KiB
HTML
361 lines
59 KiB
HTML
<a name="dataartsstudio_01_0470"></a><a name="dataartsstudio_01_0470"></a>
|
|
|
|
<h1 class="topictitle1">Setting Up Scheduling for a Job</h1>
|
|
<div id="body0000001162497603"><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p16460513273">This section describes how to set up scheduling for an orchestrated <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text8460016271">job</span>.</p>
|
|
<ul id="dataartsstudio_01_0470__ul84181338191916"><li id="dataartsstudio_01_0470__en-us_topic_0181491852_li6856113713227">If the processing mode of a job is batch processing, configure scheduling types for jobs. Three scheduling types are supported: run once, run periodically, and event-based. For details, see <a href="dataartsstudio_01_0470.html#dataartsstudio_01_0470__en-us_topic_0099797007_section1590152794714">Setting Up Scheduling for a Job Using the Batch Processing Mode</a>.</li><li id="dataartsstudio_01_0470__en-us_topic_0181491852_li887220416222">If the processing mode of a job is real-time processing, configure scheduling types for nodes. Three scheduling types are supported: run once, run periodically, and event-based. For details, see <a href="dataartsstudio_01_0470.html#dataartsstudio_01_0470__en-us_topic_0099797007_section644754422910">Setting Up Scheduling for Nodes of a Job Using the Real-Time Processing Mode</a>.</li></ul>
|
|
<div class="section" id="dataartsstudio_01_0470__en-us_topic_0099797007_section536330414847"><h4 class="sectiontitle">Prerequisites</h4><ul id="dataartsstudio_01_0470__ul1128155351514"><li id="dataartsstudio_01_0470__li10284539151">You have performed the operations in <a href="dataartsstudio_01_0435.html">Developing a Pipeline Job</a> or <a href="dataartsstudio_01_1436.html">Developing a Batch Processing Single-Task SQL Job</a>.</li><li id="dataartsstudio_01_0470__li18435714153">You have locked the job. Otherwise, you must click <strong id="dataartsstudio_01_0470__en-us_topic_0181491854_b14769125724211">Lock</strong> so that you can develop the job. A job you create or import is locked by you by default. For details, see the <a href="dataartsstudio_01_0913.html">lock function</a>.</li></ul>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0470__section1756174771410"><h4 class="sectiontitle">Constraints</h4><ul id="dataartsstudio_01_0470__ul163210671616"><li id="dataartsstudio_01_0470__li13632176191611">Set an appropriate value for this parameter. A maximum of five instances can be concurrently executed in a job. If the start time of a job instance is later than the configured job execution time, the job instances in the subsequent batch will be queued. As a result, the job execution costs a longer time than expected. For CDM and ETL jobs, the recurrence must be at least 5 minutes. In addition, the recurrence should be adjusted based on the data volume of the job table and the update frequency of the source table.</li><li id="dataartsstudio_01_0470__li0632863164">If you use <span id="dataartsstudio_01_0470__text11215183004713">DataArts Studio</span> DataArts Factory to schedule a CDM migration job and configure a scheduled task for the job in DataArts Migration, both configurations take effect. To ensure unified service logic and avoid scheduling conflicts, enable job scheduling in DataArts Factory and do not configure a scheduled task for the job in DataArts Migration.</li></ul>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0470__en-us_topic_0099797007_section1590152794714"><a name="dataartsstudio_01_0470__en-us_topic_0099797007_section1590152794714"></a><a name="en-us_topic_0099797007_section1590152794714"></a><h4 class="sectiontitle">Setting Up Scheduling for a Job Using the Batch Processing Mode</h4><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1931184412473">Three scheduling types are available: <strong id="dataartsstudio_01_0470__b2049511018509">Run once</strong>, <strong id="dataartsstudio_01_0470__b16575918135017">Run periodically</strong>, and <strong id="dataartsstudio_01_0470__b17639047115012">Event-based</strong>. The procedure is as follows:</p>
|
|
<p id="dataartsstudio_01_0470__en-us_topic_0099797007_p182410065011">Click the <strong id="dataartsstudio_01_0470__b13562103633317">Scheduling Setup</strong> tab on the right of the canvas to expand the configuration page and configure the scheduling parameters listed in <a href="#dataartsstudio_01_0470__table81266218469">Table 1</a>.</p>
|
|
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0470__table81266218469"></a><a name="table81266218469"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0470__table81266218469" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Job scheduling parameters</caption><thead align="left"><tr id="dataartsstudio_01_0470__row131261724466"><th align="left" class="cellrowborder" valign="top" width="16.669999999999998%" id="mcps1.3.5.4.2.3.1.1"><p id="dataartsstudio_01_0470__p1112632164616">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="83.33%" id="mcps1.3.5.4.2.3.1.2"><p id="dataartsstudio_01_0470__p1912612211462">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0470__row1412613244617"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.4.2.3.1.1 "><p id="dataartsstudio_01_0470__p13127123466">Scheduling Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.4.2.3.1.2 "><p id="dataartsstudio_01_0470__p412752174612">Scheduling type of the <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text759312595506">job</span>. Available options include:</p>
|
|
<ul id="dataartsstudio_01_0470__ul21279244618"><li id="dataartsstudio_01_0470__li20127226466"><strong id="dataartsstudio_01_0470__b343775318335">Run once</strong>: You need to manually execute the <span id="dataartsstudio_01_0470__text16437105363313">job</span>.</li><li id="dataartsstudio_01_0470__li10127132154612"><strong id="dataartsstudio_01_0470__b138717245349">Run periodically</strong>: The <span id="dataartsstudio_01_0470__text512713219464">job</span> is executed periodically. For details about the parameters, see <a href="#dataartsstudio_01_0470__en-us_topic_0099797007_table1055375919506">Table 2</a>.<ul id="dataartsstudio_01_0470__ul398462765413"><li id="dataartsstudio_01_0470__li1298418276548"><strong id="dataartsstudio_01_0470__b4804358542813">Manual confirmation</strong>: If this option is selected, the job instance can be executed only after manual confirmation. If manual confirmation is not performed, the job instance cannot be executed.<div class="note" id="dataartsstudio_01_0470__note51201342587"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p187781545105813"><strong id="dataartsstudio_01_0470__b14452880442813">In job instance execution scenarios</strong>, job instances are in waiting confirmation state on the <strong id="dataartsstudio_01_0470__b15625722992813">Monitor Instance</strong> page. When you click <strong id="dataartsstudio_01_0470__b11065297372813">Execute</strong>, the job instances are in waiting execution state.</p>
|
|
<p id="dataartsstudio_01_0470__p108001058125810">When you <strong id="dataartsstudio_01_0470__b9166865742813">rerun instances</strong>, they are in waiting confirmation state. When you click <strong id="dataartsstudio_01_0470__b1419865272813">Execute</strong>, the instances are in waiting execution state.</p>
|
|
<p id="dataartsstudio_01_0470__p102661819417">In <strong id="dataartsstudio_01_0470__b1953915682813">PatchData scenarios</strong>, PatchData job instances are in waiting confirmation state on the <strong id="dataartsstudio_01_0470__b10158623962813">Monitor PatchData</strong> page. When you click <strong id="dataartsstudio_01_0470__b18300491052813">Execute</strong> on the <strong id="dataartsstudio_01_0470__b7563632282813">Monitor Instance</strong> page, PatchData job instances are in waiting execution state.</p>
|
|
<p id="dataartsstudio_01_0470__p196701322105920"><strong id="dataartsstudio_01_0470__b15287837272813">In batch job monitoring scenarios</strong>, job instances are in waiting confirmation state on the <strong id="dataartsstudio_01_0470__b17098791872813">Batch Jobs</strong> page. When you click <strong id="dataartsstudio_01_0470__b3236204102813">Execute</strong>, the job instances are in waiting execution state.</p>
|
|
</div></div>
|
|
</li></ul>
|
|
</li><li id="dataartsstudio_01_0470__li17127132164618"><strong id="dataartsstudio_01_0470__b113141820365">Event-based</strong>: The job will be executed when certain external conditions are met. For details about the parameters, see <a href="#dataartsstudio_01_0470__table1862330204520">Table 3</a>. </li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row20147194914277"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.4.2.3.1.1 "><p id="dataartsstudio_01_0470__p4148849182718">Enable Dry Run</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.4.2.3.1.2 "><p id="dataartsstudio_01_0470__p3148124942713">If you select this option, the job will not be executed, and a success message will be returned.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row167992342413"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.4.2.3.1.1 "><p id="dataartsstudio_01_0470__p779202313242">Task Groups</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.4.2.3.1.2 "><p id="dataartsstudio_01_0470__p27912314241">Select a configured task group. For details, see <a href="dataartsstudio_01_1284.html">Configuring Task Groups</a>.</p>
|
|
<p id="dataartsstudio_01_0470__p73391955172416"><strong id="dataartsstudio_01_0470__b13862155151016">Do not select</strong> is selected by default.</p>
|
|
<p id="dataartsstudio_01_0470__p77372048132611">If you select a task group, you can control the maximum number of concurrent nodes in the task group in a fine-grained manner in scenarios where a job contains multiple nodes, a data patching task is ongoing, or a job is rerunning.</p>
|
|
<p id="dataartsstudio_01_0470__p12488123711345">Example 1: The maximum number of concurrent tasks in the task group is set to 2, and a job has five nodes. When the job runs, only two nodes are running and the other nodes are waiting.</p>
|
|
<p id="dataartsstudio_01_0470__p5812181716375">Example 2: The maximum number of concurrent tasks in the task group is set to 2, and the number of concurrent periods for a PatchData job is set to 5. When the PatchData job runs, two PatchData job instances are running, and the other job instances are waiting to run. The waiting instances can be delivered normally after a period of time.</p>
|
|
<p id="dataartsstudio_01_0470__p1410315435429">Example 3: If the same task group is configured for multiple jobs, and the maximum number of concurrent tasks in the task group is set to 2, only two jobs are running and the other jobs are waiting. If the same task group is configured for multiple job nodes, the maximum number of concurrent tasks in the task group is set to 2, and there are five job nodes in total, two nodes are running and the other nodes are waiting.</p>
|
|
<div class="note" id="dataartsstudio_01_0470__note21811452505"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p1118104516507">For a pipeline job, you can configure a task group for each node or for the job. A task group configured for a node is prior to one configured for the job.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0470__en-us_topic_0099797007_table1055375919506"></a><a name="en-us_topic_0099797007_table1055375919506"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0470__en-us_topic_0099797007_table1055375919506" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameters for jobs that are executed periodically</caption><thead align="left"><tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row10566359195019"><th align="left" class="cellrowborder" valign="top" width="16.669999999999998%" id="mcps1.3.5.5.2.3.1.1"><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p155721594509">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="83.33%" id="mcps1.3.5.5.2.3.1.2"><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1557655945011">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row3627759205017"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1633359185016">From and to</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p16637135914504">The period during which a scheduling task takes effect.</p>
|
|
<p id="dataartsstudio_01_0470__p1540152415120">You can set it to today or tomorrow by clicking the time box and then <strong id="dataartsstudio_01_0470__b113701028153319">Today</strong> or <strong id="dataartsstudio_01_0470__b7982430113310">Tomorrow</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row106399592503"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p86431659115014">Recurrence</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p10647175918503">The frequency at which the scheduling task is executed, which can be:</p>
|
|
<p id="dataartsstudio_01_0470__p188043311616">Set an appropriate value for this parameter. A maximum of five instances can be concurrently executed in a job. If the start time of a job instance is later than the configured job execution time, the job instances in the subsequent batch will be queued. As a result, the job execution costs a longer time than expected. For CDM and ETL jobs, the recurrence must be at least 5 minutes. In addition, the recurrence should be adjusted based on the data volume of the job table and the update frequency of the source table.</p>
|
|
<p id="dataartsstudio_01_0470__p1286122771613">You can modify the scheduling period of a running job.</p>
|
|
<ul id="dataartsstudio_01_0470__en-us_topic_0099797007_ul765025965014"><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li18650115935011"><strong id="dataartsstudio_01_0470__b1410616227525">Minutes</strong>: The job starts at the top of the hour. The interval is accurate to minute. After the scheduling ends at the end time of the current day, the scheduling automatically starts on the next day.<div class="note" id="dataartsstudio_01_0470__note1069693119347"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p14967172812712">If you select <strong id="dataartsstudio_01_0470__b28120842813">Minutes</strong> for <strong id="dataartsstudio_01_0470__b2149273912813">Scheduling Frequency</strong>, the job cannot be scheduled based on the configured interval, that is, the job cannot be executed at a fixed frequency across hours. For example:</p>
|
|
<ul id="dataartsstudio_01_0470__ul125233412717"><li id="dataartsstudio_01_0470__li162527345717">A scheduling policy is configured at 14:20 on June 19, 2024. According to the policy, the scheduling starts at 00:30 and ends at 23:59, at an interval of 30 minutes. The job is actually scheduled at 14:30:00, 15:30:00, 16:30:00, 17:30:00, 18:30:00, and more on June 19, 2024.</li><li id="dataartsstudio_01_0470__li1382352112118">A scheduling policy is configured at 14:20 on June 19, 2024. According to the policy, the scheduling starts at 00:00 and ends at 23:59, at an interval of 50 minutes. The job is actually scheduled at 14:50:00, 15:00:00, 15:50:00, 16:00:00, 16:50:00, 17:00:00, 17:50:00, and more on June 19, 2024.</li></ul>
|
|
</div></div>
|
|
</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li146551259105012"><strong id="dataartsstudio_01_0470__b1920516408114113">Hours</strong>: You can select <strong id="dataartsstudio_01_0470__b048215241476">Interval Hour</strong>, indicating that the job starts at a specified time point and that the interval is accurate to hour. After the scheduling ends at the end time of the current day, the scheduling automatically starts on the next day. You can also select <strong id="dataartsstudio_01_0470__b07631808481">Discrete Hour</strong> and specify any hour in a day to schedule the job.</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li865925975016"><strong id="dataartsstudio_01_0470__b364516551946">Every day</strong>: The job starts at a specified time on a day. The scheduling period is one day.</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li17664135915501"><strong id="dataartsstudio_01_0470__b1148919105712">Every week</strong>: You can select a specified time point of one or more days in a week.</li><li id="dataartsstudio_01_0470__li3839129411"><strong id="dataartsstudio_01_0470__b12572571187">Every month</strong>: You can select a specified time point of one or more days in a month. In addition, you can select <span class="parmvalue" id="dataartsstudio_01_0470__parmvalue2211202354415"><b>Last day of each month</b></span>.</li></ul>
|
|
<div class="note" id="dataartsstudio_01_0470__note13032311433"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p112311271521">DataArts Studio does not support concurrent running of PatchData instances and periodic job instances of underlying services (such as CDM and DLI). To prevent PatchData instances from affecting periodic job instances and avoid exceptions, ensure that they do not run at the same time.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row5626205718182"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__p862645751813">Scheduling Calendar</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__p462635751815">Select a scheduling calendar. The default value is <strong id="dataartsstudio_01_0470__b208368562323">Do not use</strong>. For details about how to configure a scheduling calendar, see <a href="dataartsstudio_01_1283.html">Configuring a Scheduling Calendar</a>.</p>
|
|
<ul id="dataartsstudio_01_0470__ul102131159482"><li id="dataartsstudio_01_0470__li18213259485">The job is scheduled on the custom working days in the calendar. On non-working days, a dry run occurs. Examples: periodic job scheduling and PatchData tasks.</li><li id="dataartsstudio_01_0470__li2143101994">Changes to the working days of the scheduling calendar do not take effect for the job instances that are being executed, but can take effect immediately for those that have not been generated.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row1646548102010"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__p114669822014">OBS Listening</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__p134661483200">If you enable this function, the system automatically listens to the OBS path for new job files. If you disable this function, the system no longer listens to the OBS path.</p>
|
|
<p id="dataartsstudio_01_0470__p18359245154213">Configure the following parameters:</p>
|
|
<ul id="dataartsstudio_01_0470__ul1919710263429"><li id="dataartsstudio_01_0470__li171970263428"><strong id="dataartsstudio_01_0470__b188813258173">OBS File</strong>: An EL expression is supported.</li><li id="dataartsstudio_01_0470__li16205143124217"><strong id="dataartsstudio_01_0470__b954675515179">Listening Interval</strong>: Set a value ranging from 1 to 60, in minutes.</li><li id="dataartsstudio_01_0470__li49583424310"><strong id="dataartsstudio_01_0470__b15358154851817">Timeout</strong>: Set a value ranging from 1 to 1440, in minutes.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row4673959145011"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p3677125917502">Dependency <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text176812059155019">job</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__p54617147526">You can select jobs that are executed periodically in different workspaces as dependency jobs. The current job starts only after the dependency jobs are executed. You can click <strong id="dataartsstudio_01_0470__b2999410165">Parse Dependency</strong> to automatically identify job dependencies.</p>
|
|
<div class="note" id="dataartsstudio_01_0470__note923119459820"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p10231144514818">For details about job dependency rules across workspaces, see <a href="dataartsstudio_01_04501.html#dataartsstudio_01_04501__section45021415123915">Job Dependency Rule</a>.</p>
|
|
</div></div>
|
|
<p id="dataartsstudio_01_0470__p20433162211614">Currently, DataArts Factory supports two types of job dependency policies, that is, dependency between jobs whose scheduling periods are traditional periods and dependency between jobs whose scheduling periods are natural periods. You can select either of them. The scheduling periods for new DataArts Studio instances are natural periods.</p>
|
|
<div class="fignone" id="dataartsstudio_01_0470__fig10661104425010"><span class="figcap"><b>Figure 1 </b>Dependency between jobs whose scheduling periods are traditional periods</span><br><span><img id="dataartsstudio_01_0470__image1056212210577" src="en-us_image_0000002269196733.png" title="Click to enlarge" class="imgResize"></span></div>
|
|
<div class="fignone" id="dataartsstudio_01_0470__fig581633918513"><span class="figcap"><b>Figure 2 </b>Dependency between jobs whose scheduling periods are natural periods</span><br><span><img id="dataartsstudio_01_0470__image617963965712" src="en-us_image_0000002234077464.png" title="Click to enlarge" class="imgResize"></span></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row157261759125010"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1873116595500">Policy for Current <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text37343593507">job</span> If Dependency <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text1573375955015">job</span> Fails</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p373913599507">Policy for processing the current <span id="dataartsstudio_01_0470__text18496246394">job</span> when one or more instances of its dependency <span id="dataartsstudio_01_0470__text6850182410396">job</span> fail to be executed in its period.</p>
|
|
<ul id="dataartsstudio_01_0470__en-us_topic_0099797007_ul774525916504"><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li1674735915504">Pending<p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1475255975010"><a name="dataartsstudio_01_0470__en-us_topic_0099797007_li1674735915504"></a><a name="en-us_topic_0099797007_li1674735915504"></a>Waits to execute the current <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text775475914505">job</span>, which affects the execution of subsequent <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text5760145915017">job</span>s. You can force the dependency <span id="dataartsstudio_01_0470__text1493219201722">job</span> to be executed successfully.</p>
|
|
</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li57656592507">Continue<p id="dataartsstudio_01_0470__en-us_topic_0099797007_p15770145910501"><a name="dataartsstudio_01_0470__en-us_topic_0099797007_li57656592507"></a><a name="en-us_topic_0099797007_li57656592507"></a>Continues to execute the current <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text777265945014">job</span>.</p>
|
|
</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li2077418592508">Cancel<p id="dataartsstudio_01_0470__en-us_topic_0099797007_p207781459165016"><a name="dataartsstudio_01_0470__en-us_topic_0099797007_li2077418592508"></a><a name="en-us_topic_0099797007_li2077418592508"></a>Cancels the current <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text9781659165020">job</span>. Its status becomes <span class="parmvalue" id="dataartsstudio_01_0470__en-us_topic_0099797007_parmvalue10786959165014"><b>Canceled</b></span>.</p>
|
|
</li></ul>
|
|
<div class="p" id="dataartsstudio_01_0470__p142493717518">For example, the recurrence of the current <span id="dataartsstudio_01_0470__text122493719514">job</span> is 1 hour and that of its dependency <span id="dataartsstudio_01_0470__text98071228877">job</span>s is 5 minutes.<ul id="dataartsstudio_01_0470__ul95594569514"><li id="dataartsstudio_01_0470__li1550884185114">If the value of this parameter is set to <strong id="dataartsstudio_01_0470__b162824391496">Cancel</strong>, the current job will be canceled as long as one of the 12 instances of its dependency job fails.</li><li id="dataartsstudio_01_0470__li11781539155113">If the value of this parameter is set to <strong id="dataartsstudio_01_0470__b1772315483">Continue</strong>, the current job will be executed after the 12 instances of its dependency job are executed.<div class="note" id="dataartsstudio_01_0470__note1951414818535"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p1514781531">You can set this parameter for multiple jobs in a batch. For details, see <a href="dataartsstudio_01_04501.html">Configuring a Default Item</a>. This parameter takes effect only for new jobs.</p>
|
|
</div></div>
|
|
</li></ul>
|
|
</div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row13789145912504"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p4793115915508">Run After Dependency <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text1579645916504">job</span> Ends</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__p981004111451">If a <span id="dataartsstudio_01_0470__text354613514496">job</span> depends on other <span id="dataartsstudio_01_0470__text125464514910">job</span>s, the job is executed only after its dependency job instances are executed within a specified time range. If the dependency job instances are not successfully executed, the current job is in waiting state.</p>
|
|
<p id="dataartsstudio_01_0470__p0349194512453">If you select this option, the system checks whether all job instances in the previous cycle have been executed before executing the current <span id="dataartsstudio_01_0470__text1026112915450">job</span>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row9769173710512"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__p20769537859">Dependency Job</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__p3769173719519">When configuring job dependencies, you can filter dependent jobs based on whether they are being scheduled. This prevents downstream job failures caused by upstream dependent jobs not being scheduled.</p>
|
|
<ul id="dataartsstudio_01_0470__ul225916118144"><li id="dataartsstudio_01_0470__li142592110146">All jobs</li><li id="dataartsstudio_01_0470__li1530710415148">Running jobs</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row268403316528"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__p76844336527">Dependency Cycle</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><ul id="dataartsstudio_01_0470__ul452962113552"><li id="dataartsstudio_01_0470__li5529121115517">Same Cycle</li><li id="dataartsstudio_01_0470__li19465327155519">Previous N Cycle. N range is from 1 to 30.<div class="note" id="dataartsstudio_01_0470__note8472162661217"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dataartsstudio_01_0470__ul1316602231316"><li id="dataartsstudio_01_0470__li1616617229136">If a daily job depends on a job scheduled by interval hour, the dependency rule can be <strong id="dataartsstudio_01_0470__b1487323623113">All instances</strong>, <strong id="dataartsstudio_01_0470__b2483154611319">Any instance</strong>, <strong id="dataartsstudio_01_0470__b891665316318">Last instance</strong>, or <strong id="dataartsstudio_01_0470__b5850111133213">Default</strong>. <strong id="dataartsstudio_01_0470__b1084314743213">Default</strong> indicates that the previous scheduling dependency rule remains unchanged.</li><li id="dataartsstudio_01_0470__li17881192512135">If a daily job depends on a job scheduled by discrete hour, the dependency rule can be <strong id="dataartsstudio_01_0470__b88192373214">All instances</strong>, <strong id="dataartsstudio_01_0470__b148122311323">Any instance</strong>, <strong id="dataartsstudio_01_0470__b108152323211">Last instance</strong>, or <strong id="dataartsstudio_01_0470__b108523173211">Default</strong>. <strong id="dataartsstudio_01_0470__b35225546329">Default</strong> indicates that the previous scheduling dependency rule remains unchanged.</li><li id="dataartsstudio_01_0470__li13385167141412">The dependencies between jobs using other scheduling periods remain unchanged, and the dependency rule is <strong id="dataartsstudio_01_0470__b1694381912335">Default</strong>.</li></ul>
|
|
</div></div>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row13813145915506"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p8817359175014">Cross-Cycle Dependency</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p148243597502">Dependency between <span id="dataartsstudio_01_0470__en-us_topic_0099797007_text382735935016">job</span> instances</p>
|
|
<ul id="dataartsstudio_01_0470__en-us_topic_0099797007_ul5831559185018"><li id="dataartsstudio_01_0470__li1766715330349"><strong id="dataartsstudio_01_0470__b94471412194613">Independent on the previous schedule cycle</strong>: You can set <strong id="dataartsstudio_01_0470__b269345133711">Concurrency</strong> to set the number of job instances that are concurrently executed. If you set it to <strong id="dataartsstudio_01_0470__b4390839376">1</strong>, a batch is executed only after the previous batch is executed (the execution is successful, cancelled, or failed).</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li583917597505"><strong id="dataartsstudio_01_0470__b1241644781618">Self-dependent</strong>: The job can be rescheduled only after it is executed in the current schedule cycle. Before that, the job is in <strong id="dataartsstudio_01_0470__b1726963015176">Waiting</strong> state.</li><li id="dataartsstudio_01_0470__li1923900162813"><strong id="dataartsstudio_01_0470__b10483150171712">Skip waiting instances and run the latest instance</strong>: Skipped job instances will be canceled and not executed. If the execution of a job instance takes a long time, multiple subsequent job instances may be skipped. However, if these job instances need to be executed, skipping them may cause service logic errors. For example, if partitioned tables are required but redundant job instances are skipped, some partitioned tables may go missing. Exercise caution when selecting this option.<div class="note" id="dataartsstudio_01_0470__note36153268287"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dataartsstudio_01_0470__ul153801350340"><li id="dataartsstudio_01_0470__li193801935143419"><strong id="dataartsstudio_01_0470__b13543164610357">Skip waiting instances and run the latest instance</strong> is only supported for jobs scheduled by minute or hour.</li><li id="dataartsstudio_01_0470__li18382123533419">If the number of concurrent jobs is small and no instance has been generated, blocked instances will not be skipped.</li><li id="dataartsstudio_01_0470__li34887313311">If a job with a shorter period depends on a job with a longer period, some instances may not be skipped and still be executed.</li></ul>
|
|
</div></div>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row10956134222413"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__p5957124202420">Clear Waiting Instances</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><ul id="dataartsstudio_01_0470__ul19600191652519"><li id="dataartsstudio_01_0470__li166011161255">No</li><li id="dataartsstudio_01_0470__li8819235252">Yes<p id="dataartsstudio_01_0470__p101291722182611"><a name="dataartsstudio_01_0470__li8819235252"></a><a name="li8819235252"></a>If this parameter is not set, expired waiting job instances will be cleared based on the workspace-level configuration by default. You can set whether to clear waiting job instances based on the site requirements.</p>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row32581825618"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__p7695165115110">Enable Dry Run</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__p146950511214">If you select this option, the job will not be executed, and a success message will be returned.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row1621252812119"><td class="cellrowborder" valign="top" width="16.669999999999998%" headers="mcps1.3.5.5.2.3.1.1 "><p id="dataartsstudio_01_0470__p1069545116113">Task Groups</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.33%" headers="mcps1.3.5.5.2.3.1.2 "><p id="dataartsstudio_01_0470__p1069617511814">Select a configured task group. For details, see <a href="dataartsstudio_01_1284.html">Configuring Task Groups</a>.</p>
|
|
<p id="dataartsstudio_01_0470__p469616514116"><strong id="dataartsstudio_01_0470__b1335597238">Do not select</strong> is selected by default.</p>
|
|
<p id="dataartsstudio_01_0470__p8696251516">If you select a task group, you can control the maximum number of concurrent nodes in the task group in a fine-grained manner in scenarios where a job contains multiple nodes, a data patching task is ongoing, or a job is rerunning.</p>
|
|
<div class="note" id="dataartsstudio_01_0470__note166963515116"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p1769615112118">For a pipeline job, you can configure a task group for each node or for the job. A task group configured for a node is prior to one configured for the job.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0470__table1862330204520"></a><a name="table1862330204520"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0470__table1862330204520" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Parameters for event-based jobs</caption><thead align="left"><tr id="dataartsstudio_01_0470__row26233014455"><th align="left" class="cellrowborder" valign="top" width="16.650000000000002%" id="mcps1.3.5.6.2.3.1.1"><p id="dataartsstudio_01_0470__p1611114452522">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="83.35000000000001%" id="mcps1.3.5.6.2.3.1.2"><p id="dataartsstudio_01_0470__p9111114516529">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0470__row1762330144510"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p56233018458">Event Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p1362313044516">Type of the event that triggers job running</p>
|
|
<ul id="dataartsstudio_01_0470__ul26231054516"><li id="dataartsstudio_01_0470__li16236064514"><strong id="dataartsstudio_01_0470__b1062320074510"><span class="parmvalue" id="dataartsstudio_01_0470__parmvalue1262390114512"><b>KAFKA</b></span></strong></li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row197828444494"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p19782184474916">Enable Dry Run</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p9543145295017">If you select this option, the job will not be executed, and a success message will be returned.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row167295814498"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p157255811499">Task Groups</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p35431152195018">Select a configured task group. For details, see <a href="dataartsstudio_01_1284.html">Configuring Task Groups</a>.</p>
|
|
<p id="dataartsstudio_01_0470__p854335214503"><strong id="dataartsstudio_01_0470__b1508079953">Do not select</strong> is selected by default.</p>
|
|
<p id="dataartsstudio_01_0470__p165439526508">If you select a task group, you can control the maximum number of concurrent nodes in the task group in a fine-grained manner in scenarios where a job contains multiple nodes, a data patching task is ongoing, or a job is rerunning.</p>
|
|
<div class="note" id="dataartsstudio_01_0470__note17543195225019"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p19543135210509">For a pipeline job, you can configure a task group for each node or for the job. A task group configured for a node is prior to one configured for the job.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row162411013454"><td class="cellrowborder" colspan="2" valign="top" headers="mcps1.3.5.6.2.3.1.1 mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p12624105457">Parameters for KAFKA event-triggered jobs</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row8624704454"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p2062412034517">Connection Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p1362440124512">Before selecting a data connection, ensure that a Kafka data connection has been created in the <span class="uicontrol" id="dataartsstudio_01_0470__uicontrol6740151241918"><b>Management Center</b></span>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row106243012452"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p20624120194513">Topic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p762410064516">Topic of the message to be sent to the Kafka.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row18624803452"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p66241607450">Concurrent Events</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p762417017455">Number of jobs that can be concurrently processed. The maximum number of concurrent events is 128.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row962416054514"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p562415024515">Event Detection Interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p6624705454">Interval at which the system detects the stream for new messages. The unit of the interval can be <strong id="dataartsstudio_01_0470__b147914242013">Seconds</strong> or <strong id="dataartsstudio_01_0470__b147912202018">Minutes</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row1062470194516"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p1362410154516">Access Policy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p86243013452">Select the location where data is to be accessed:</p>
|
|
<ul id="dataartsstudio_01_0470__ul76241909453"><li id="dataartsstudio_01_0470__li06248084519"><strong id="dataartsstudio_01_0470__b842820427">Access from the last location</strong>: For the first access, data is accessed from the most recently recorded location. For the subsequent access, data is accessed from the previously recoded location.</li><li id="dataartsstudio_01_0470__li86241400456"><strong id="dataartsstudio_01_0470__b569121308">Access from a new location</strong>: Data is accessed from the most recently recorded location each time.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row76241405453"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p262415084511">Failure Policy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p1462417024512">Select a policy to be performed after scheduling fails.</p>
|
|
<ul id="dataartsstudio_01_0470__ul76244016454"><li id="dataartsstudio_01_0470__li12624200104514">Suspend</li><li id="dataartsstudio_01_0470__li1662419010455">Ignore the failure and proceed with the next event</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row172321233165113"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p3694154565119">Enable Dry Run</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p1669474510516">If you select this option, the job will not be executed, and a success message will be returned.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row1933320372514"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p196941845125119">Task Groups</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p869416452517">Select a configured task group. For details, see <a href="dataartsstudio_01_1284.html">Configuring Task Groups</a>.</p>
|
|
<p id="dataartsstudio_01_0470__p569518453513"><strong id="dataartsstudio_01_0470__b1648336192">Do not select</strong> is selected by default.</p>
|
|
<p id="dataartsstudio_01_0470__p6695845105115">If you select a task group, you can control the maximum number of concurrent nodes in the task group in a fine-grained manner in scenarios where a job contains multiple nodes, a data patching task is ongoing, or a job is rerunning.</p>
|
|
<div class="note" id="dataartsstudio_01_0470__note9695194514518"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p2069504515116">For a pipeline job, you can configure a task group for each node or for the job. A task group configured for a node is prior to one configured for the job.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row680215316515"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p6540125765118">Enable Dry Run</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p55401057145110">If you select this option, the job will not be executed, and a success message will be returned.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row198985615513"><td class="cellrowborder" valign="top" width="16.650000000000002%" headers="mcps1.3.5.6.2.3.1.1 "><p id="dataartsstudio_01_0470__p1454115572519">Task Groups</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.35000000000001%" headers="mcps1.3.5.6.2.3.1.2 "><p id="dataartsstudio_01_0470__p1254125714510">Select a configured task group. For details, see <a href="dataartsstudio_01_1284.html">Configuring Task Groups</a>.</p>
|
|
<p id="dataartsstudio_01_0470__p10541557145117"><strong id="dataartsstudio_01_0470__b2068589254">Do not select</strong> is selected by default.</p>
|
|
<p id="dataartsstudio_01_0470__p155411657185119">If you select a task group, you can control the maximum number of concurrent nodes in the task group in a fine-grained manner in scenarios where a job contains multiple nodes, a data patching task is ongoing, or a job is rerunning.</p>
|
|
<div class="note" id="dataartsstudio_01_0470__note1754165715110"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0470__p15411057185120">For a pipeline job, you can configure a task group for each node or for the job. A task group configured for a node is prior to one configured for the job.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_01_0470__en-us_topic_0099797007_section644754422910"><a name="dataartsstudio_01_0470__en-us_topic_0099797007_section644754422910"></a><a name="en-us_topic_0099797007_section644754422910"></a><h4 class="sectiontitle">Setting Up Scheduling for Nodes of a Job Using the Real-Time Processing Mode</h4><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1974713318300">Three scheduling types are available: <strong id="dataartsstudio_01_0470__b142811321270">Run once</strong>, <strong id="dataartsstudio_01_0470__b64291332102710">Run periodically</strong>, and <strong id="dataartsstudio_01_0470__b24291132142715">Event-based</strong>. The procedure is as follows:</p>
|
|
<p id="dataartsstudio_01_0470__en-us_topic_0099797007_p675218319301">Select a node. On the node development page, click the <span class="wintitle" id="dataartsstudio_01_0470__wintitle16771651103012"><b>Scheduling Parameter Setup</b></span> tab. On the displayed page, configure the parameters listed in <a href="#dataartsstudio_01_0470__en-us_topic_0099797007_table16934134912315">Table 4</a>.</p>
|
|
|
|
<div class="tablenoborder"><a name="dataartsstudio_01_0470__en-us_topic_0099797007_table16934134912315"></a><a name="en-us_topic_0099797007_table16934134912315"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0470__en-us_topic_0099797007_table16934134912315" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Parameters for setting up node scheduling</caption><thead align="left"><tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row18401150203218"><th align="left" class="cellrowborder" valign="top" width="17.169999999999998%" id="mcps1.3.6.4.2.3.1.1"><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p8400650143210">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="82.83%" id="mcps1.3.6.4.2.3.1.2"><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p13401950183215">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row18460350163219"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1045825013325">Scheduling Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p145835011323">Scheduling type of the <span id="dataartsstudio_01_0470__text123454744117">job</span>. Available options include:</p>
|
|
<ul id="dataartsstudio_01_0470__en-us_topic_0099797007_ul1646035003211"><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li3458450103216"><strong id="dataartsstudio_01_0470__b1538112184112">Run once</strong>: You need to manually run the <span id="dataartsstudio_01_0470__text838171274112">job</span>.</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li045825013321"><strong id="dataartsstudio_01_0470__b13117151494117">Run periodically</strong>: The <span id="dataartsstudio_01_0470__text311751417411">job</span> runs automatically and periodically.</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li845825013218"><strong id="dataartsstudio_01_0470__b134421674113">Event-based</strong>: The job runs when certain external conditions are met.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row34601050153216"><td class="cellrowborder" colspan="2" valign="top" headers="mcps1.3.6.4.2.3.1.1 mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p12460145063214"><strong id="dataartsstudio_01_0470__b933426114417">Parameters displayed when Scheduling Type is <span class="parmvalue" id="dataartsstudio_01_0470__parmvalue123348634417"><b>Run periodically</b></span></strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row34634507328"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1046075073215">From and to</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p14463145013211">The period during which a scheduling task takes effect.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row164631350193214"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p74631750133215">Recurrence</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p19463185019324">The frequency at which the scheduling task is executed, which can be:</p>
|
|
<ul id="dataartsstudio_01_0470__en-us_topic_0099797007_ul246315016320"><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li5463950143218">Minutes</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li16463185013214">Hours</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li15463450123218">Every day</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li13463195043216">Every week</li><li id="dataartsstudio_01_0470__li1416831922113">Every month</li></ul>
|
|
<p id="dataartsstudio_01_0470__p773052020216">For CDM and ETL jobs, the recurrence must be at least 5 minutes. In addition, the recurrence should be adjusted based on the data volume of the job table and the update frequency of the source table.</p>
|
|
<p id="dataartsstudio_01_0470__p38137548149">You can modify the scheduling period of a running job.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row204690504326"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p64696503322">Cross-Cycle Dependency</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p164691650113213">Dependency between <span id="dataartsstudio_01_0470__text1861613521508">job</span> instances</p>
|
|
<ul id="dataartsstudio_01_0470__en-us_topic_0099797007_ul15469115053211"><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li846915508329"><strong id="dataartsstudio_01_0470__b1193295244">Independent on the previous schedule cycle</strong><p id="dataartsstudio_01_0470__p91401659123111">Set <strong id="dataartsstudio_01_0470__b690719385223">Concurrency</strong>. Number of job instances that are concurrently executed. If you set it to <strong id="dataartsstudio_01_0470__b715571210233">1</strong>, a batch is executed only after the previous batch is executed (the execution is successful, cancelled, or failed).</p>
|
|
</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li15469155016322"><strong id="dataartsstudio_01_0470__b1693193462314">Self-dependent</strong>: The job can be rescheduled only after it is executed in the current schedule cycle. Before that, the job is in Waiting state.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row8472125023216"><td class="cellrowborder" colspan="2" valign="top" headers="mcps1.3.6.4.2.3.1.1 mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p747275011326"><strong id="dataartsstudio_01_0470__b96381359712">Parameters displayed when Scheduling Type is <span class="parmvalue" id="dataartsstudio_01_0470__parmvalue2063810599117"><b>Event-based</b></span></strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row8472550163212"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p2047212502320">Event Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1834413916436">Type of the event that triggers job running</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row14765189193114"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__p17982312113112">Connection Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__p19828121313">Before selecting a data connection, ensure that a Kafka data connection has been created in the <span class="uicontrol" id="dataartsstudio_01_0470__uicontrol229785410210"><b>Management Center</b></span>. This parameter is mandatory only when <strong id="dataartsstudio_01_0470__b931454592518">Event Type</strong> is set to <strong id="dataartsstudio_01_0470__b1031519455258">KAFKA</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row1051171215317"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__p1898251253117">Topic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__p139828127315">Topic of the message to be sent to the Kafka. This parameter is mandatory only when <strong id="dataartsstudio_01_0470__b37669652619">Event Type</strong> is set to <strong id="dataartsstudio_01_0470__b127661661262">KAFKA</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row1841903752015"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__p1420133711205">Consumer Group</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__p38921314144811">A scalable and fault-tolerant group of consumers in Kafka.</p>
|
|
<p id="dataartsstudio_01_0470__p1689218145480">Consumers in a group share the same ID. They collaborate with each other to consume all partitions of subscribed topics. A partition in a topic can be consumed by only one consumer.</p>
|
|
<div class="note" id="dataartsstudio_01_0470__note16101642154918"><span class="notetitle"> NOTE: </span><div class="notebody"><ol id="dataartsstudio_01_0470__ol3106944134916"><li id="dataartsstudio_01_0470__li81061144114920">A consumer group can contain multiple consumers.</li><li id="dataartsstudio_01_0470__li9106544104913">The group ID is a string that uniquely identifies a consumer group in a Kafka cluster.</li><li id="dataartsstudio_01_0470__li2010613440494">Each partition of each topic subscribed to by a consumer group can be consumed by only one consumer. Consumer groups do not affect each other.</li></ol>
|
|
</div></div>
|
|
<p id="dataartsstudio_01_0470__p11420193772014">If you select <strong id="dataartsstudio_01_0470__b19423144219305">KAFKA</strong> for <strong id="dataartsstudio_01_0470__b13424134243013">Event Type</strong>, the consumer group ID is automatically displayed. You can also manually change the consumer group ID.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row17472050163217"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p1472145017329">Concurrent Events</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p647225033214">Number of jobs that can be concurrently processed. The maximum number of concurrent events is 10.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row102006116457"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p24941620164514">Event Detection Interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p17494162016454">Interval at which the system detects the stream for new messages. The unit of the interval can be <strong id="dataartsstudio_01_0470__b1822541814420">Seconds</strong> or <strong id="dataartsstudio_01_0470__b122262181341">Minutes</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__row15202204811164"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__p620212488168">Access Policy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><ul id="dataartsstudio_01_0470__ul551411208171"><li id="dataartsstudio_01_0470__li1351442011711">Access from the last location</li><li id="dataartsstudio_01_0470__li11531635201710">Access from a new location<p id="dataartsstudio_01_0470__p18348145017251"><a name="dataartsstudio_01_0470__li11531635201710"></a><a name="li11531635201710"></a>This parameter is mandatory only when <strong id="dataartsstudio_01_0470__b1588821320286">Event Type</strong> is set to <strong id="dataartsstudio_01_0470__b17888201352812">KAFKA</strong>.</p>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_01_0470__en-us_topic_0099797007_row8473650143219"><td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.6.4.2.3.1.1 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p20472850153211">Failure Policy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="82.83%" headers="mcps1.3.6.4.2.3.1.2 "><p id="dataartsstudio_01_0470__en-us_topic_0099797007_p7472125063218">Select a policy to be performed after scheduling fails.</p>
|
|
<ul id="dataartsstudio_01_0470__en-us_topic_0099797007_ul20473150183215"><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li4472135013211">Suspend</li><li id="dataartsstudio_01_0470__en-us_topic_0099797007_li54731250173210">Ignore failure and proceed</li></ul>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0431.html">Job Development</a></div>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<script language="JavaScript">
|
|
<!--
|
|
initImageViewer('.imgResize');
|
|
var msg_imageMax = "view original image";
|
|
var msg_imageClose = "close";
|
|
//--></script> |