forked from docs/doc-exports
Hasko, Vladimir
cfc48b3aed
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-committed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
125 lines
14 KiB
HTML
125 lines
14 KiB
HTML
<a name="dli_09_0187"></a><a name="dli_09_0187"></a>
|
|
|
|
<h1 class="topictitle1">Java Example Code</h1>
|
|
<div id="body0000001129144494"><div class="section" id="dli_09_0187__section1523149131718"><h4 class="sectiontitle">Development Description</h4><ul id="dli_09_0187__ul27577387185"><li id="dli_09_0187__li695651911134">Prerequisites<p id="dli_09_0187__en-us_topic_0200509991_p1944354710257"><a name="dli_09_0187__li695651911134"></a><a name="li695651911134"></a>A datasource connection has been created and bound to a queue on the DLI management console. </p>
|
|
<div class="note" id="dli_09_0187__note1358715714155"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0187__p1858718570154">Hard-coded or plaintext passwords pose significant security risks. To ensure security, encrypt your passwords, store them in configuration files or environment variables, and decrypt them when needed.</p>
|
|
</div></div>
|
|
</li><li id="dli_09_0187__li19554172391910">Code implementation<ol id="dli_09_0187__en-us_topic_0200509994_ol12123050181818"><li id="dli_09_0187__en-us_topic_0190647826_li1822810810586">Import dependencies.<ul id="dli_09_0187__ul0528315185614"><li id="dli_09_0187__li145285157562">Maven dependency involved<div class="codecoloring" codetype="Scala" id="dli_09_0187__en-us_topic_0190647826_screen5760163172012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="o"><</span><span class="n">dependency</span><span class="o">></span>
|
|
<span class="w"> </span><span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">spark</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
|
|
<span class="w"> </span><span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">spark</span><span class="o">-</span><span class="n">sql_2</span><span class="mf">.11</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
|
|
<span class="w"> </span><span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.3.2</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
|
|
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_09_0187__en-us_topic_0200509994_li1612316509182">Import dependency packages.<div class="codecoloring" codetype="Java" id="dli_09_0187__en-us_topic_0200509994_screen68181719144911"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ul>
|
|
</li><li id="dli_09_0187__en-us_topic_0200509994_li11272141817195">Create a session.<div class="codecoloring" codetype="Java" id="dli_09_0187__en-us_topic_0200509994_screen17320101315392"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">SparkSession</span><span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">"datasource-rds"</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ol>
|
|
</li><li id="dli_09_0187__li85121163417">Connecting to data sources through SQL APIs<ul id="dli_09_0187__ul45041116545"><li id="dli_09_0187__en-us_topic_0190647826_li1216955517548">Create a table to connect to an RDS data source and set connection parameters.<div class="codecoloring" codetype="Scala" id="dli_09_0187__en-us_topic_0190647826_screen187422020161315"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span>
|
|
<span class="w"> </span><span class="s">"CREATE TABLE IF NOT EXISTS dli_to_rds USING JDBC OPTIONS (</span>
|
|
<span class="s"> 'url'='jdbc:mysql://to-rds-1174404209-cA37siB6.datasource.com:3306', // Set this parameter to the actual URL.</span>
|
|
<span class="s"> 'dbtable'='test.customer',</span>
|
|
<span class="s"> 'user'='root', // Set this parameter to the actual user.</span>
|
|
<span class="s"> 'password'='######', // Set this parameter to the actual password.</span>
|
|
<span class="s"> 'driver'='com.mysql.jdbc.Driver')"</span><span class="p">)</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
<p id="dli_09_0187__p77404151805">For details about the parameters for creating a table, see <a href="dli_09_0067.html#dli_09_0067__en-us_topic_0190647826_table127421320141311">Table 1</a>.</p>
|
|
</li><li id="dli_09_0187__en-us_topic_0190647826_li911302371611">Insert data.<div class="codecoloring" codetype="Scala" id="dli_09_0187__en-us_topic_0190647826_screen1711312301613"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">"insert into dli_to_rds values (1,'John',24)"</span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_09_0187__en-us_topic_0190647826_li513113315552">Query data.<div class="codecoloring" codetype="Scala" id="dli_09_0187__en-us_topic_0190647826_screen8335528349"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">"select * from dli_to_rd"</span><span class="p">).</span><span class="n">show</span><span class="p">();</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
<p id="dli_09_0187__en-us_topic_0190647826_p1344084843017">Response</p>
|
|
<p id="dli_09_0187__en-us_topic_0190647826_p4683160173110"><span><img id="dli_09_0187__image24153213297" src="en-us_image_0000001129442286.png"></span></p>
|
|
</li></ul>
|
|
</li><li id="dli_09_0187__li176166213412">Submitting a Spark job<ol id="dli_09_0187__ol1457716359012"><li id="dli_09_0187__li1692416144334">Generate a JAR package based on the code and upload the package to DLI.<p id="dli_09_0187__dli_09_0063_p1749619513385"><a name="dli_09_0187__li1692416144334"></a><a name="li1692416144334"></a></p>
|
|
<p id="dli_09_0187__dli_09_0063_p114961151385"></p>
|
|
</li><li id="dli_09_0187__li11341758112615">In the Spark job editor, select the corresponding dependency module and execute the Spark job.<p id="dli_09_0187__p961812492717"><a name="dli_09_0187__li11341758112615"></a><a name="li11341758112615"></a></p>
|
|
<p id="dli_09_0187__p16726180182714"></p>
|
|
</li><li id="dli_09_0187__li4531174473610">After the Spark job is created, click <strong id="dli_09_0187__b18194171416474">Execute</strong> in the upper right corner of the console to submit the job. If the message "Spark job submitted successfully." is displayed, the Spark job is successfully submitted. You can view the status and logs of the submitted job on the <strong id="dli_09_0187__b919481464715">Spark Jobs</strong> page.<div class="note" id="dli_09_0187__note653184433613"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0187__ul4531844123612"><li id="dli_09_0187__en-us_topic_0197738142_li58215295819">If the Spark version is 2.3.2 (will be offline soon) or 2.4.5, specify the <strong id="dli_09_0187__b164276476519">Module</strong> to <strong id="dli_09_0187__b14281847205117">sys.datasource.rds</strong> when you submit a job.</li><li id="dli_09_0187__li6624653171317">If the Spark version is 3.1.1, you do not need to select a module. Configure <strong id="dli_09_0187__b7397194835117">Spark parameters (--conf)</strong>.<p id="dli_09_0187__p131071216162710">spark.driver.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/rds/*</p>
|
|
<p id="dli_09_0187__p210761614275">spark.executor.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/rds/*</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</li></ol>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_09_0187__section3403192673815"><h4 class="sectiontitle">Complete Example Code</h4><p id="dli_09_0187__p63350933011">Connecting to data sources through SQL APIs</p>
|
|
<div class="codecoloring" codetype="Java" id="dli_09_0187__en-us_topic_0200509995_screen1659318340448"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span>
|
|
<span class="normal">16</span>
|
|
<span class="normal">17</span>
|
|
<span class="normal">18</span>
|
|
<span class="normal">19</span>
|
|
<span class="normal">20</span>
|
|
<span class="normal">21</span>
|
|
<span class="normal">22</span>
|
|
<span class="normal">23</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
|
|
<span class="w"> </span>
|
|
<span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">java_rds</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span>
|
|
<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">String</span><span class="o">[]</span><span class="w"> </span><span class="n">args</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="n">SparkSession</span><span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">"datasource-rds"</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
|
|
<span class="w"> </span>
|
|
<span class="w"> </span><span class="c1">// Create a data table for DLI-associated RDS</span>
|
|
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">"CREATE TABLE IF NOT EXISTS dli_to_rds USING JDBC OPTIONS ('url'='jdbc:mysql://192.168.6.150:3306','dbtable'='test.customer','user'='root','password'='**','driver'='com.mysql.jdbc.Driver')"</span><span class="p">);</span>
|
|
<span class="w"> </span>
|
|
<span class="w"> </span><span class="c1">//*****************************SQL model***********************************</span>
|
|
<span class="w"> </span><span class="c1">//Insert data into the DLI data table</span>
|
|
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">"insert into dli_to_rds values(3,'Liu',21),(4,'Joey',34)"</span><span class="p">);</span>
|
|
<span class="w"> </span>
|
|
<span class="w"> </span><span class="c1">//Read data from DLI data table</span>
|
|
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">"select * from dli_to_rds"</span><span class="p">);</span>
|
|
<span class="w"> </span>
|
|
<span class="w"> </span><span class="c1">//drop table</span>
|
|
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">"drop table dli_to_rds"</span><span class="p">);</span>
|
|
<span class="w"> </span>
|
|
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">close</span><span class="p">();</span>
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="p">}</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0083.html">Connecting to RDS</a></div>
|
|
</div>
|
|
</div>
|
|
|