forked from docs/doc-exports
Hasko, Vladimir
cfc48b3aed
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-committed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
86 lines
9.2 KiB
HTML
86 lines
9.2 KiB
HTML
<a name="dli_09_0199"></a><a name="dli_09_0199"></a>
|
|
|
|
<h1 class="topictitle1">Java Example Code</h1>
|
|
<div id="body0000001135221806"><div class="section" id="dli_09_0199__section1523149131718"><h4 class="sectiontitle">Scenario</h4><p id="dli_09_0199__p98722576293">This section provides Java example code that demonstrates how to use a Spark job to access data from the GaussDB(DWS) data source.</p>
|
|
<p id="dli_09_0199__en-us_topic_0200509991_p1944354710257">A datasource connection has been created and bound to a queue on the DLI management console. </p>
|
|
<div class="note" id="dli_09_0199__note17925192652815"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0199__p692572617287">Hard-coded or plaintext passwords pose significant security risks. To ensure security, encrypt your passwords, store them in configuration files or environment variables, and decrypt them when needed.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="dli_09_0199__section656993743018"><h4 class="sectiontitle">Preparations</h4><ol id="dli_09_0199__en-us_topic_0200509994_ol12123050181818"><li id="dli_09_0199__en-us_topic_0190647826_li1822810810586">Import dependencies.<ul id="dli_09_0199__ul0528315185614"><li id="dli_09_0199__li145285157562">Maven dependency involved<div class="codecoloring" codetype="Scala" id="dli_09_0199__en-us_topic_0190647826_screen5760163172012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="o"><</span><span class="n">dependency</span><span class="o">></span>
|
|
<span class="w"> </span><span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">spark</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
|
|
<span class="w"> </span><span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">spark</span><span class="o">-</span><span class="n">sql_2</span><span class="mf">.11</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
|
|
<span class="w"> </span><span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.3.2</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
|
|
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_09_0199__en-us_topic_0200509994_li1612316509182">Import dependency packages.<div class="codecoloring" codetype="Java" id="dli_09_0199__en-us_topic_0200509994_screen68181719144911"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ul>
|
|
</li><li id="dli_09_0199__en-us_topic_0200509994_li11272141817195">Create a session.<div class="codecoloring" codetype="Java" id="dli_09_0199__en-us_topic_0200509994_screen17320101315392"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">SparkSession</span><span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">"datasource-dws"</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0199__section4814201113110"><h4 class="sectiontitle">Accessing a Data Source Through a SQL API</h4><ol id="dli_09_0199__ol9234133825219"><li id="dli_09_0199__li19234173815528">Create a table to connect to a GaussDB(DWS) data source and set connection parameters.<div class="codecoloring" codetype="Scala" id="dli_09_0199__screen726420535011"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">"CREATE TABLE IF NOT EXISTS dli_to_dws USING JDBC OPTIONS ('url'='jdbc:postgresql://10.0.0.233:8000/postgres','dbtable'='test','user'='dbadmin','password'='**')"</span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_09_0199__li1723433845217">Insert data.<div class="codecoloring" codetype="Scala" id="dli_09_0199__screen61301123961"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">"insert into dli_to_dws values(3,'L'),(4,'X')"</span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_09_0199__li1623483818528">Query data.<div class="codecoloring" codetype="Scala" id="dli_09_0199__screen17264451508"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">"select * from dli_to_dws"</span><span class="p">).</span><span class="n">show</span><span class="p">();</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0199__section19206113373114"><h4 class="sectiontitle">Submitting a Spark Job</h4><ol id="dli_09_0199__ol4869123010374"><li id="dli_09_0199__li8259161223111">Generate a JAR package based on the code file and upload the package to DLI.<p id="dli_09_0199__p187271030123116"><a name="dli_09_0199__li8259161223111"></a><a name="li8259161223111"></a></p>
|
|
<p id="dli_09_0199__p6956718123118"></p>
|
|
</li><li id="dli_09_0199__li17164236163115">In the Spark job editor, select the corresponding dependency module and execute the Spark job.<p id="dli_09_0199__p563042113112"><a name="dli_09_0199__li17164236163115"></a><a name="li17164236163115"></a></p>
|
|
<div class="p" id="dli_09_0199__p18532183733111"><div class="note" id="dli_09_0199__note886918308376"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0199__ul386983014371"><li id="dli_09_0199__en-us_topic_0197738142_li58215295819">If the Spark version is 2.3.2 (will be offline soon) or 2.4.5, specify the <strong id="dli_09_0199__b12935521164111">Module</strong> to <strong id="dli_09_0199__b1393514218413">sys.datasource.dws</strong> when you submit a job.</li><li id="dli_09_0199__li6624653171317">If the Spark version is 3.1.1, you do not need to select a module. Configure <strong id="dli_09_0199__b28504225412">Spark parameters (--conf)</strong>.<p id="dli_09_0199__p158960912231">spark.driver.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/dws/*</p>
|
|
<p id="dli_09_0199__p1572150112011">spark.executor.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/dws/*</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</div>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_09_0199__section613922012488"><h4 class="sectiontitle">Complete Example Code</h4><p id="dli_09_0199__p1552532954819">Accessing GaussDB(DWS) tables through SQL APIs</p>
|
|
<pre class="screen" id="dli_09_0199__screen11506205924817">import org.apache.spark.sql.SparkSession;
|
|
|
|
public class java_dws {
|
|
public static void main(String[] args) {
|
|
SparkSession sparkSession = SparkSession.builder().appName("datasource-dws").getOrCreate();
|
|
|
|
sparkSession.sql("CREATE TABLE IF NOT EXISTS dli_to_dws USING JDBC OPTIONS ('url'='jdbc:postgresql://10.0.0.233:8000/postgres','dbtable'='test','user'='dbadmin','password'='**')");
|
|
|
|
//*****************************SQL model***********************************
|
|
//Insert data into the DLI data table
|
|
sparkSession.sql("insert into dli_to_dws values(3,'Liu'),(4,'Xie')");
|
|
|
|
//Read data from DLI data table
|
|
sparkSession.sql("select * from dli_to_dws").show();
|
|
|
|
//drop table
|
|
sparkSession.sql("drop table dli_to_dws");
|
|
|
|
sparkSession.close();
|
|
}
|
|
}</pre>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0086.html">Connecting to GaussDB(DWS)</a></div>
|
|
</div>
|
|
</div>
|
|
|