doc-exports/docs/dataartsstudio/umn/dataartsstudio_01_1584.html

<a name="dataartsstudio_01_1584"></a><a name="dataartsstudio_01_1584"></a>

<h1 class="topictitle1">Obtaining the Maximum Value and Transferring It to a CDM Job Using a Query SQL Statement</h1>
<div id="body0000001505271116"><div class="section" id="dataartsstudio_01_1584__section112992914354"><h4 class="sectiontitle">Scenario</h4><p id="dataartsstudio_01_1584__p181334173517">You can run a query SQL statement to transfer the obtained maximum time value to a CDM job. In the advanced attributes of the CDM job, the where clause is used to determine the maximum time range to obtain the data to be migrated and complete the incremental data migration.</p>
</div>
<div class="section" id="dataartsstudio_01_1584__section8890444143510"><h4 class="sectiontitle">Constraints</h4><ol id="dataartsstudio_01_1584__ol14272435151218"><li id="dataartsstudio_01_1584__li1427211350129">You have completed operations in <a href="dataartsstudio_01_0404.html">Creating a Data Connection</a>.</li><li id="dataartsstudio_01_1584__li1170755451519">You have completed operations in <a href="dataartsstudio_01_0405.html">Creating a Database</a>.</li></ol>
</div>
<div class="section" id="dataartsstudio_01_1584__section101181056113517"><h4 class="sectiontitle">Examples</h4><p id="dataartsstudio_01_1584__p147038093615"><strong id="dataartsstudio_01_1584__b72253172514">Creating an SQL Script</strong></p>
<ol id="dataartsstudio_01_1584__ol188715537297"><li id="dataartsstudio_01_1584__li6576131765420">In the left navigation pane of DataArts Factory, choose <span class="menucascade" id="dataartsstudio_01_1584__en-us_topic_0181092879_en-us_topic_0104967364_menucascade1889024114211"><b><span class="uicontrol" id="dataartsstudio_01_1584__en-us_topic_0181092879_en-us_topic_0104967364_uicontrol88905411627"><span id="dataartsstudio_01_1584__en-us_topic_0181092879_en-us_topic_0104967364_text6890114118213">Data Development</span></span></b> &gt; <b><span class="uicontrol" id="dataartsstudio_01_1584__en-us_topic_0181092879_en-us_topic_0104967364_uicontrol1189017411422"><span id="dataartsstudio_01_1584__en-us_topic_0181092879_en-us_topic_0104967364_text18901411529">Develop Script</span></span></b></span>.</li><li id="dataartsstudio_01_1584__li1750991663015">Create an SQL script. This section uses the MRS Spark SQL script as an example.</li><li id="dataartsstudio_01_1584__li512752383014">Select a created data connection and database.</li><li id="dataartsstudio_01_1584__li10785826123016">Compile the SQL script to obtain the maximum time data from table1.<pre class="screen" id="dataartsstudio_01_1584__screen1612142417231">select max(time) from table1</pre>
</li><li id="dataartsstudio_01_1584__li20356183219307">Save and submit the version. The <strong id="dataartsstudio_01_1584__b21581467315">maxtime</strong> script is created.</li></ol>
<p id="dataartsstudio_01_1584__p833124162811"><strong id="dataartsstudio_01_1584__b1538842752819">Creating a Pipeline Subjob</strong></p>
<ol id="dataartsstudio_01_1584__ol18671885299"><li id="dataartsstudio_01_1584__li82414287313">In the left navigation pane of DataArts Factory, choose <span class="menucascade" id="dataartsstudio_01_1584__en-us_topic_0181491853_en-us_topic_0099797006_menucascade12634646125010"><b><span class="uicontrol" id="dataartsstudio_01_1584__en-us_topic_0181491853_en-us_topic_0099797006_uicontrol1063494615015">Development</span></b> &gt; <b><span class="uicontrol" id="dataartsstudio_01_1584__en-us_topic_0181491853_en-us_topic_0099797006_uicontrol14634104655019"><span id="dataartsstudio_01_1584__en-us_topic_0181491853_en-us_topic_0099797006_text126341046105013">Develop Job</span></span></b></span>.</li><li id="dataartsstudio_01_1584__li82601923113112">Select a CDM Job node and configure the node properties.<div class="fignone" id="dataartsstudio_01_1584__fig1697042533519"><span class="figcap"><b>Figure 1 </b>Configuring CDM Job node properties</span><br><span><img id="dataartsstudio_01_1584__image1971325173518" src="en-us_image_0000002234079008.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_01_1584__p956401922319"></p>
<p id="dataartsstudio_01_1584__p1385744155717">Select a CDM cluster and associate the node with an existing CDM job.</p>
<p id="dataartsstudio_01_1584__p48820338368">Configure the job parameters and add job parameter <strong id="dataartsstudio_01_1584__b188181451164911">maxtime</strong>.</p>
<div class="fignone" id="dataartsstudio_01_1584__fig19890143223715"><span class="figcap"><b>Figure 2 </b>Configuring job parameters</span><br><span><img id="dataartsstudio_01_1584__image158910320373" src="en-us_image_0000002234238852.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_01_1584__p12459947182314"></p>
</li><li id="dataartsstudio_01_1584__li126701926163814">Save and submit the version. The subjob <strong id="dataartsstudio_01_1584__b551411115359">sub</strong> is created.</li></ol>
<p id="dataartsstudio_01_1584__p10123856151717"><strong id="dataartsstudio_01_1584__b1774711413510">Creating a Pipeline Job</strong></p>
<ol id="dataartsstudio_01_1584__ol2975195818399"><li id="dataartsstudio_01_1584__li27995414115">In the left navigation pane of DataArts Factory, choose <span class="menucascade" id="dataartsstudio_01_1584__en-us_topic_0181491853_en-us_topic_0099797006_menucascade12634646125010_1"><b><span class="uicontrol" id="dataartsstudio_01_1584__en-us_topic_0181491853_en-us_topic_0099797006_uicontrol1063494615015_1">Development</span></b> &gt; <b><span class="uicontrol" id="dataartsstudio_01_1584__en-us_topic_0181491853_en-us_topic_0099797006_uicontrol14634104655019_1"><span id="dataartsstudio_01_1584__en-us_topic_0181491853_en-us_topic_0099797006_text126341046105013_1">Develop Job</span></span></b></span>.</li><li id="dataartsstudio_01_1584__li590092884318">Select an MRS Spark SQL node and a For Each node to execute the CDM subjob cyclically.</li><li id="dataartsstudio_01_1584__li71360616348">Configure properties of the MRS Spark SQL node and associate the node with the created <strong id="dataartsstudio_01_1584__b762981833717">maxtime</strong> script.<div class="fignone" id="dataartsstudio_01_1584__fig248513418364"><span class="figcap"><b>Figure 3 </b>Configuring properties for the MRS Spark SQL node</span><br><span><img id="dataartsstudio_01_1584__image1548610344369" src="en-us_image_0000002234079016.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_01_1584__p396371212247"></p>
</li><li id="dataartsstudio_01_1584__li1534202312344">Configure properties of the For Each node and associate the node with the created CDM subjob.<div class="fignone" id="dataartsstudio_01_1584__fig11473115311381"><span class="figcap"><b>Figure 4 </b>Configuring properties for the For Each node</span><br><span><img id="dataartsstudio_01_1584__image7473145316384" src="en-us_image_0000002269118201.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_01_1584__p6427744102412"></p>
<p id="dataartsstudio_01_1584__p19757101114013">After associating the node with the created subjob <strong id="dataartsstudio_01_1584__b5770484345">sub</strong>, write a parameter expression.</p>
<pre class="screen" id="dataartsstudio_01_1584__screen17505942194019">#{Loop.current[0]}</pre>
<p id="dataartsstudio_01_1584__p16889621142814">Configure the data set, with an EL expression supported.</p>
<pre class="screen" id="dataartsstudio_01_1584__screen1670836104819">#{Job.getNodeOutput("maxtime")}</pre>
</li><li id="dataartsstudio_01_1584__li054565994819">Save and submit the version. The job is created.</li></ol>
<p id="dataartsstudio_01_1584__p187843321911"><strong id="dataartsstudio_01_1584__b1044619369374">Obtaining the Maximum Time Value from the CDM Job Using a Where Clause and Transferring the Value to the Destination Job</strong></p>
<ol id="dataartsstudio_01_1584__ol954813483526"><li id="dataartsstudio_01_1584__li25481848145216">Open the created subjob.</li><li id="dataartsstudio_01_1584__li2275124810590">Click <span><img id="dataartsstudio_01_1584__image26631230907" src="en-us_image_0000002269198293.png"></span> next to the job name to go to the job configuration page.<div class="fignone" id="dataartsstudio_01_1584__fig1875333012252"><span class="figcap"><b>Figure 5 </b>Editing the CDM job</span><br><span><img id="dataartsstudio_01_1584__image20753173015257" src="en-us_image_0000002269118193.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_01_1584__p123951815132515"></p>
</li><li id="dataartsstudio_01_1584__li158826491208">In the advanced attributes of the source job configuration, configure a where clause to obtain the data to be migrated. When the job is executed, the migration data obtained from the source will be replicated, exported, and imported to the destination.<div class="fignone" id="dataartsstudio_01_1584__fig8907181605213"><span class="figcap"><b>Figure 6 </b>Configuring a where clause</span><br><span><img id="dataartsstudio_01_1584__image69082164523" src="en-us_image_0000002234079000.png" title="Click to enlarge" class="imgResize"></span></div>
<p id="dataartsstudio_01_1584__p1340011192610"></p>
<p id="dataartsstudio_01_1584__p89613414316">The where clause is as follows:</p>
<pre class="screen" id="dataartsstudio_01_1584__screen092013304310">dt &gt; '${maxtime}'</pre>
</li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_01_0520.html">Usage Guidance</a></div>
</div>
</div>


<script language="JavaScript">
<!--
initImageViewer('.imgResize');
var msg_imageMax = "view original image";
var msg_imageClose = "close";
//--></script>