yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
a41a4e0331fbc0beff63bb28fa59228cc9a751df
doc-exports/docs/dataartsstudio/umn/dataartsstudio_01_0585.html
chenxiaoxiong f9e2808b7c DataArts UMN 20250810 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
Co-committed-by: chenxiaoxiong <chenxiaoxiong@huawei.com>
2025-09-02 10:44:13 +00:00

127 lines
22 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<a name="dataartsstudio_01_0585"></a><a name="dataartsstudio_01_0585"></a>
<h1 class="topictitle1">Using PatchData</h1>
<div id="body0000001482374296"><div class="section" id="dataartsstudio_01_0585__section208624416919"><h4 class="sectiontitle">Scenario</h4><p id="dataartsstudio_01_0585__p2491754597">In the migration of a project, if you want to supplement historical business data in a previous period and view details of the historical data, PatchData can meet your requirements.</p>
<p id="dataartsstudio_01_0585__p102759215205">A job executes a scheduling task to generate a series of instances in a certain period of time. This series of instances are called PatchData. PatchData can be used to fix the job instances that have data errors in the historical records or to build job records for debugging programs.</p>
<div class="note" id="dataartsstudio_01_0585__note611812506537"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dataartsstudio_01_0585__ul1224795814551"><li id="dataartsstudio_01_0585__li152471258135510">In addition to SQL scripts, PatchData supports other nodes.</li><li id="dataartsstudio_01_0585__li13247185875516">If the content of a SQL script changes, the PatchData job runs the latest script.</li><li id="dataartsstudio_01_0585__li17785141993711">When you use PatchData, if the variable in the SQL statement is <strong id="dataartsstudio_01_0585__b1572032286">DATE</strong>, enter <strong id="dataartsstudio_01_0585__b17103217810">${DATE}</strong> in the script. The script parameter <strong id="dataartsstudio_01_0585__b3783213813">DATE</strong> is then automatically added to the job parameters, and its value can be an EL expression. If the variable is a time variable, enter the expression of the <strong id="dataartsstudio_01_0585__b1032443972517">DateUtil</strong> embedded object. The platform automatically converts the expression into a historical date. </li><li id="dataartsstudio_01_0585__li202487588553">PatchData jobs support script parameters and global environment variables as well as job parameters.</li></ul>
</div></div>
</div>
<div class="section" id="dataartsstudio_01_0585__section162591929141211"><h4 class="sectiontitle">Constraints</h4><p id="dataartsstudio_01_0585__p360952917174">PatchData is available only when periodic scheduling is configured for the data development job.</p>
</div>
<div class="section" id="dataartsstudio_01_0585__section19184742617"><h4 class="sectiontitle">Example</h4><p id="dataartsstudio_01_0585__en-us_topic_0000001115624138_p13952174615596"><strong id="dataartsstudio_01_0585__b865834465812">Scenario</strong></p>
<p id="dataartsstudio_01_0585__p13358154072717">Among the product data tables of a company, there is a source data table A that records the product sales amount. To import the historical product sales amount to the destination table B, you can create a PatchData job.</p>
<div class="p" id="dataartsstudio_01_0585__p136589613465"><a href="#dataartsstudio_01_0585__table1412775294511">Table 1</a> lists the source and destination tables.
<div class="tablenoborder"><a name="dataartsstudio_01_0585__table1412775294511"></a><a name="table1412775294511"></a><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0585__table1412775294511" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Source and destination tables</caption><thead align="left"><tr id="dataartsstudio_01_0585__row11127552204520"><th align="left" class="cellrowborder" valign="top" width="48%" id="mcps1.3.3.4.2.2.3.1.1"><p id="dataartsstudio_01_0585__p11127105213456">Source Table</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="52%" id="mcps1.3.3.4.2.2.3.1.2"><p id="dataartsstudio_01_0585__p15127652164520">Destination Table</p>
</th>
</tr>
</thead>
<tbody><tr id="dataartsstudio_01_0585__row19128145254510"><td class="cellrowborder" valign="top" width="48%" headers="mcps1.3.3.4.2.2.3.1.1 "><p id="dataartsstudio_01_0585__p1612865264519">A</p>
</td>
<td class="cellrowborder" valign="top" width="52%" headers="mcps1.3.3.4.2.2.3.1.2 "><p id="dataartsstudio_01_0585__p1712845210451">B</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<p id="dataartsstudio_01_0585__p52641258519"><strong id="dataartsstudio_01_0585__b5264054519">Procedure</strong></p>
<ol id="dataartsstudio_01_0585__ol0102326115819"><li id="dataartsstudio_01_0585__li141021926125811">Prepare the source and destination tables. To facilitate subsequent job execution and verification, you need to create a source DWS table and a destination DWS table and insert data into the tables.<ol type="a" id="dataartsstudio_01_0585__ol026514595116"><li id="dataartsstudio_01_0585__li22659514519">Create a DWS table. You can create a DWS SQL script on the DataArts Factory console of DataArts Studio and run the following SQL statements:<pre class="screen" id="dataartsstudio_01_0585__screen5265135125113">/* Create tables. */
CREATE TABLE A (PRODUCT_ID INT, SALES INT, DATE DATE);
CREATE TABLE B (PRODUCT_ID INT, SALES INT, DATE DATE);</pre>
</li><li id="dataartsstudio_01_0585__li726585205116">Insert sample data into the source data table. You can create a DWS SQL script on the DataArts Factory console of DataArts Studio and run the following SQL statements:<pre class="screen" id="dataartsstudio_01_0585__screen1826516595117">/* Insert sample historical data into the source table. */
INSERT INTO A VALUES ('1','60', '2022-03-01');
INSERT INTO A VALUES ('2','80', '2022-03-01');
INSERT INTO A VALUES ('1','50', '2022-02-28');
INSERT INTO A VALUES ('2','55', '2022-02-28');
INSERT INTO A VALUES ('1','60', '2022-02-27');
INSERT INTO A VALUES ('2','45', '2022-02-27');</pre>
</li></ol>
</li><li id="dataartsstudio_01_0585__li193611632">Develop a PatchData script. Ensure that the script expression contains a time variable. (For example, if the variable in the SQL statement is <strong id="dataartsstudio_01_0585__b159717923616">DATE</strong>, enter <strong id="dataartsstudio_01_0585__b46875177362">${DATE}</strong> in the script.) You can set the expression for script parameter <strong id="dataartsstudio_01_0585__b168581911385">DATE</strong> in job parameter settings in <a href="#dataartsstudio_01_0585__li14638161911311">3</a>.<div class="p" id="dataartsstudio_01_0585__p8816953212">On the <span class="uicontrol" id="dataartsstudio_01_0585__uicontrol199410541988"><b>Develop Script</b></span> page, enter following statement in the editor:<pre class="screen" id="dataartsstudio_01_0585__screen118211953219">INSERT INTO B (SELECT * FROM A WHERE DATE = ${DATE})</pre>
<div class="fignone" id="dataartsstudio_01_0585__fig48214918321"><span class="figcap"><b>Figure 1 </b>Developing a script</span><br><span><img id="dataartsstudio_01_0585__image148216983219" src="en-us_image_0000002234076104.png" title="Click to enlarge" class="imgResize"></span></div>
</div>
<p id="dataartsstudio_01_0585__p2943141911144">After compiling the script, save it and submit the latest version.</p>
</li><li id="dataartsstudio_01_0585__li14638161911311"><a name="dataartsstudio_01_0585__li14638161911311"></a><a name="li14638161911311"></a>Develop a PatchData batch processing job. When developing the job, you need to configure the node attributes and scheduling period.<p id="dataartsstudio_01_0585__p538549123316"><a name="dataartsstudio_01_0585__li14638161911311"></a><a name="li14638161911311"></a>In the left navigation pane of the DataArts Factory console, choose <span class="menucascade" id="dataartsstudio_01_0585__en-us_topic_0181491853_en-us_topic_0099797006_menucascade12634646125010"><b><span class="uicontrol" id="dataartsstudio_01_0585__en-us_topic_0181491853_en-us_topic_0099797006_uicontrol1063494615015"><span id="dataartsstudio_01_0585__en-us_topic_0181491853_en-us_topic_0099797006_text1563414615503">Data Development</span></span></b> &gt; <b><span class="uicontrol" id="dataartsstudio_01_0585__en-us_topic_0181491853_en-us_topic_0099797006_uicontrol14634104655019"><span id="dataartsstudio_01_0585__en-us_topic_0181491853_en-us_topic_0099797006_text126341046105013">Develop Job</span></span></b></span>.</p>
<div class="fignone" id="dataartsstudio_01_0585__fig161651455153213"><span class="figcap"><b>Figure 2 </b>Node parameters</span><br><span><img id="dataartsstudio_01_0585__image1165955133219" src="en-us_image_0000002234235952.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_01_0585__p5165135563219"></p>
<div class="note" id="dataartsstudio_01_0585__note716625518325"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dataartsstudio_01_0585__ul81661355143217"><li id="dataartsstudio_01_0585__li41651855113217">If the job-associated SQL script uses a parameter, the parameter name (such as <strong id="dataartsstudio_01_0585__b319765111417">DATE</strong>) is displayed. Set the parameter value in the text box next to the parameter name. The parameter value can be an EL expression. <p id="dataartsstudio_01_0585__p151657552329">If the parameter is time, view the example expression of the DateUtil embedded object. The platform automatically replaces the parameter with the historical date of the patch data (determined by the service date of the patch data).</p>
<p id="dataartsstudio_01_0585__p5421121810134">You can also directly enter a SQL expression.</p>
</li><li id="dataartsstudio_01_0585__li1916535514324">If the parameters of the associated SQL script change, you can click <span><img id="dataartsstudio_01_0585__image516555510324" src="en-us_image_0000002269195369.png"></span> to synchronize the change or click <span><img id="dataartsstudio_01_0585__image916535517326" src="en-us_image_0000002234076120.png"></span> to edit the parameters.</li><li id="dataartsstudio_01_0585__li9165105516328">The following is an example of script parameters:<div class="p" id="dataartsstudio_01_0585__p516535520322"><a name="dataartsstudio_01_0585__li9165105516328"></a><a name="li9165105516328"></a>Example: #{DateUtil.format(DateUtil.addDays(Job.planTime,-1),'yyyy-MM-dd')}<ul id="dataartsstudio_01_0585__ul91651055123210"><li id="dataartsstudio_01_0585__li2165155511321"><strong id="dataartsstudio_01_0585__b474093451414">Job.planTime</strong> indicates the planned job time, and <em id="dataartsstudio_01_0585__i395744416155">yyyy-MM-dd</em> indicates the time format.</li><li id="dataartsstudio_01_0585__li1616525543219">If the planned job time is March 2, the previous day is March 1. The planned job time will be replaced by the configured patch data service date.</li><li id="dataartsstudio_01_0585__li14165125519323">The <strong id="dataartsstudio_01_0585__b96204517197">Job.planTime</strong> is converted into a time in the <em id="dataartsstudio_01_0585__i399917391198">yyyy-MM-dd</em> format using an expression.</li></ul>
</div>
</li></ul>
</div></div>
<p id="dataartsstudio_01_0585__p8166125593213">Configure the scheduling period of the PatchData job. Click <strong id="dataartsstudio_01_0585__b397247152118">Scheduling Setup</strong> and set <strong id="dataartsstudio_01_0585__b146208215226">Scheduling Frequency</strong> to <strong id="dataartsstudio_01_0585__b544122316227">Every day</strong>.</p>
<div class="fignone" id="dataartsstudio_01_0585__fig16166255153217"><span class="figcap"><b>Figure 3 </b>Configuring the scheduling period</span><br><span><img id="dataartsstudio_01_0585__image0166655133213" src="en-us_image_0000002269115341.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_01_0585__p216655553213"></p>
<div class="note" id="dataartsstudio_01_0585__note116625512324"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dataartsstudio_01_0585__ul1116685523210"><li id="dataartsstudio_01_0585__li6166155123219">If <strong id="dataartsstudio_01_0585__b6919448222">Scheduling Frequency</strong> is set to <strong id="dataartsstudio_01_0585__b129644132211">Every day</strong>, the job is scheduled every day, and a PatchData instance is generated. You can view the statuses of PatchData instances on the <span class="parmname" id="dataartsstudio_01_0585__parmname1216615519325"><b>Monitor Instance</b></span> page. On the <strong id="dataartsstudio_01_0585__b4577201145111">Monitor Instance</strong> page, view the instance information about the job and perform more operations on instances as required.</li><li id="dataartsstudio_01_0585__li181661255173212">The job scheduling time takes effect from March 9, 2023, and the job is scheduled at 02:00 every day.</li><li id="dataartsstudio_01_0585__li16166855183212">Run the following SQL statement to check whether destination table B contains data of source table A:<p id="dataartsstudio_01_0585__p10166175513322"><a name="dataartsstudio_01_0585__li16166855183212"></a><a name="li16166855183212"></a>SELECT * FROM B</p>
</li></ul>
</div></div>
<p id="dataartsstudio_01_0585__p5166855153218">After configuring the parameters, save and submit the latest version of the job and test the job.</p>
<p id="dataartsstudio_01_0585__p15166125514325">Click <span class="uicontrol" id="dataartsstudio_01_0585__uicontrol15166195523218"><b>Execute</b></span> to run the job.</p>
</li><li id="dataartsstudio_01_0585__li5810248420">Create a PatchData task.<div class="p" id="dataartsstudio_01_0585__p2591251543"><a name="dataartsstudio_01_0585__li5810248420"></a><a name="li5810248420"></a>After creating a periodic job, you need to configure PatchData for the job.<ol type="a" id="dataartsstudio_01_0585__ol7179148145"><li id="dataartsstudio_01_0585__li2174748246">In the left navigation pane of DataArts Factory, choose <span class="menucascade" id="dataartsstudio_01_0585__en-us_topic_0181289902_en-us_topic_0159100548_menucascade13508951284"><b><span class="uicontrol" id="dataartsstudio_01_0585__en-us_topic_0181289902_en-us_topic_0159100548_uicontrol13508115686"><span id="dataartsstudio_01_0585__en-us_topic_0181289902_en-us_topic_0159100548_text1750819517811">Monitoring</span></span></b> &gt; <b><span class="uicontrol" id="dataartsstudio_01_0585__en-us_topic_0181289902_en-us_topic_0159100548_uicontrol125081256813"><span id="dataartsstudio_01_0585__en-us_topic_0181289902_en-us_topic_0159100548_text65081258811">Job Monitoring</span></span></b></span>.</li><li id="dataartsstudio_01_0585__li317944819414">Click the <span class="wintitle" id="dataartsstudio_01_0585__wintitle5982134417272"><b>Batch Job Monitoring</b></span> tab. In the <strong id="dataartsstudio_01_0585__b989407723102524">Operation</strong> column of the job, choose <span class="menucascade" id="dataartsstudio_01_0585__menucascade1816636139102524"><b><span class="uicontrol" id="dataartsstudio_01_0585__uicontrol1280986404102524">More</span></b> &gt; <b><span class="uicontrol" id="dataartsstudio_01_0585__uicontrol940740145102524">Configure PatchData</span></b></span>. The <span class="wintitle" id="dataartsstudio_01_0585__wintitle517516485413"><b>Configure PatchData</b></span> page is displayed.<div class="p" id="dataartsstudio_01_0585__p19625206184410">If you want to supplement historical data from February 27, 2023 to March 1, 2023, set <strong id="dataartsstudio_01_0585__b1969082055719">Date</strong> to <strong id="dataartsstudio_01_0585__b186901120185715">Feb 28, 2023 00:00:00 – Mar 02, 2023 23:59:59</strong>. The system automatically transfers the configured date to the planned job time. In the expression of the script time variable <strong id="dataartsstudio_01_0585__b569132085714">DATE</strong>, the defined time is the planned job time minus one day. That is, the time of the day before the planned job time is the time range (<strong id="dataartsstudio_01_0585__b06911820175716">Feb 27, 2023 to Mar 1, 2023</strong>) for PatchData.<div class="fignone" id="dataartsstudio_01_0585__fig254545124416"><span class="figcap"><b>Figure 4 </b>Configuring PatchData</span><br><span><img id="dataartsstudio_01_0585__image165456564413" src="en-us_image_0000002269195385.png" title="Click to enlarge" class="imgResize"></span></div>
</div>
<p id="dataartsstudio_01_0585__p145451758441"></p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_01_0585__table1054819517442" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Description</caption><thead align="left"><tr id="dataartsstudio_01_0585__row1754625184414"><th align="left" class="cellrowborder" valign="top" width="35.17%" id="mcps1.3.3.6.4.1.1.2.7.2.3.1.1"><p id="dataartsstudio_01_0585__p115469518449">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="64.83%" id="mcps1.3.3.6.4.1.1.2.7.2.3.1.2"><p id="dataartsstudio_01_0585__p16546195184413">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dataartsstudio_01_0585__row165468524417"><td class="cellrowborder" valign="top" width="35.17%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.1 "><p id="dataartsstudio_01_0585__p7546105104419">PatchData Name</p>
</td>
<td class="cellrowborder" valign="top" width="64.83%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.2 "><p id="dataartsstudio_01_0585__p1154615511448">Name of the automatically generated PatchData task. The value can be modified.</p>
</td>
</tr>
<tr id="dataartsstudio_01_0585__row954614519446"><td class="cellrowborder" valign="top" width="35.17%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.1 "><p id="dataartsstudio_01_0585__p654612514444">Job Name</p>
</td>
<td class="cellrowborder" valign="top" width="64.83%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.2 "><p id="dataartsstudio_01_0585__p115461858449">Name of the job that requires PatchData, which is automatically displayed</p>
</td>
</tr>
<tr id="dataartsstudio_01_0585__row14547259443"><td class="cellrowborder" valign="top" width="35.17%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.1 "><p id="dataartsstudio_01_0585__p155461159446">Date</p>
</td>
<td class="cellrowborder" valign="top" width="64.83%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.2 "><p id="dataartsstudio_01_0585__p954610518445">Period of time when PatchData is required. This date is transferred to the planned job time. When the job is executed, the planned job time is replaced by the time in the PatchData.</p>
<div class="note" id="dataartsstudio_01_0585__note105475511447"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0585__p65476518443">PatchData can be configured for a job multiple times. However, avoid configuring PatchData multiple times on the same date to prevent data duplication or disorder.</p>
</div></div>
<p id="dataartsstudio_01_0585__p1249965316267">If you select <strong id="dataartsstudio_01_0585__b1605426522102524">Patch data in reverse order of date</strong>, the patch data of each day is in positive sequence.</p>
<div class="note" id="dataartsstudio_01_0585__note7644153963118"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dataartsstudio_01_0585__ul3954125253719"><li id="dataartsstudio_01_0585__li7954195218373">This function is applicable when the data of each day is not coupled with each other.</li><li id="dataartsstudio_01_0585__li16954052153718">The PatchData job will ignore the dependencies between the job instances created before this date.</li></ul>
</div></div>
</td>
</tr>
<tr id="dataartsstudio_01_0585__row554710594413"><td class="cellrowborder" valign="top" width="35.17%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.1 "><p id="dataartsstudio_01_0585__p185471056444">Parallel Periods</p>
</td>
<td class="cellrowborder" valign="top" width="64.83%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.2 "><p id="dataartsstudio_01_0585__p1254719510449">Number of instances to be executed at the same time. A maximum of five instances can be executed at the same time.</p>
<div class="note" id="dataartsstudio_01_0585__note754765184419"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dataartsstudio_01_0585__p854712524414">Set this parameter based on the site requirements. For example, if a CDM job instance is used, data cannot be supplemented at the same time. The value of this parameter can only be set to <strong id="dataartsstudio_01_0585__b2019752799102524">1</strong>.</p>
</div></div>
</td>
</tr>
<tr id="dataartsstudio_01_0585__row1354816554418"><td class="cellrowborder" valign="top" width="35.17%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.1 "><p id="dataartsstudio_01_0585__p9547755445">Upstream or Downstream Job</p>
</td>
<td class="cellrowborder" valign="top" width="64.83%" headers="mcps1.3.3.6.4.1.1.2.7.2.3.1.2 "><p id="dataartsstudio_01_0585__p454755134414">This parameter is optional. Select the downstream jobs (jobs that depend on the current job) that require PatchData. You can select multiple jobs.</p>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="dataartsstudio_01_0585__li1017924816413">Click <span class="uicontrol" id="dataartsstudio_01_0585__uicontrol31795481542"><b>OK</b></span>. The system starts to run the PatchData task based on the configured scheduling period.</li><li id="dataartsstudio_01_0585__li191796488410">On the <span class="parmname" id="dataartsstudio_01_0585__parmname1979618541742"><b>Monitor PatchData</b></span> page, you can view the PatchData task status, date, number of parallel periods, PatchData job name, and stopped tasks. You can also view logs of the PatchData task.<div class="fignone" id="dataartsstudio_01_0585__fig328167182417"><span class="figcap"><b>Figure 5 </b>Querying PatchData details</span><br><span><img id="dataartsstudio_01_0585__image175412174019" src="en-us_image_0000002269195381.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_01_0585__p111991535122019"></p>
</li><li id="dataartsstudio_01_0585__li151720176427">Run the following SQL statement to check whether destination table B contains historical data of source table A:<pre class="screen" id="dataartsstudio_01_0585__screen754910024312">SELECT * FROM B</pre>
</li></ol>
</div>
</li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0520.html">Usage Guidance</a></div>
</div>
</div>
<script language="JavaScript">
<!--
initImageViewer('.imgResize');
var msg_imageMax = "view original image";
var msg_imageClose = "close";
//--></script>
View Git Blame Copy Permalink