forked from docs/doc-exports
Hasko, Vladimir
cfc48b3aed
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-committed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
158 lines
22 KiB
HTML
158 lines
22 KiB
HTML
<a name="dli_09_0100"></a><a name="dli_09_0100"></a>
|
|
|
|
<h1 class="topictitle1">Java Example Code</h1>
|
|
<div id="body8662426"><div class="section" id="dli_09_0100__section1523149131718"><h4 class="sectiontitle">Development Description</h4><p id="dli_09_0100__en-us_topic_0200509991_p492312464537">Redis supports only enhanced datasource connections. </p>
|
|
<ul id="dli_09_0100__ul27577387185"><li id="dli_09_0100__li695651911134">Prerequisites<p id="dli_09_0100__en-us_topic_0200509991_p1944354710257"><a name="dli_09_0100__li695651911134"></a><a name="li695651911134"></a>An enhanced datasource connection has been created on the DLI management console and bound to a queue in packages. </p>
|
|
<div class="note" id="dli_09_0100__note1358715714155"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0100__p1858718570154">Hard-coded or plaintext passwords pose significant security risks. To ensure security, encrypt your passwords, store them in configuration files or environment variables, and decrypt them when needed.</p>
|
|
</div></div>
|
|
</li><li id="dli_09_0100__li19554172391910">Code implementation<ol id="dli_09_0100__en-us_topic_0200509994_ol12123050181818"><li id="dli_09_0100__li511923413525">Import dependencies.<ul id="dli_09_0100__ul5801745145216"><li id="dli_09_0100__li145285157562">Maven dependency involved<div class="codecoloring" codetype="Scala" id="dli_09_0100__en-us_topic_0190647826_screen5760163172012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="o"><</span><span class="n">dependency</span><span class="o">></span>
|
|
<span class="w"> </span><span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">spark</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
|
|
<span class="w"> </span><span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">spark</span><span class="o">-</span><span class="n">sql_2</span><span class="mf">.11</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
|
|
<span class="w"> </span><span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.3.2</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
|
|
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_09_0100__li202542175313">Import dependency packages.<div class="codecoloring" codetype="Java" id="dli_09_0100__en-us_topic_0200509994_screen68181719144911"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span>
|
|
<span class="normal">8</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.SparkConf</span><span class="p">;</span>
|
|
<span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.api.java.JavaRDD</span><span class="p">;</span>
|
|
<span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.api.java.JavaSparkContext</span><span class="p">;</span>
|
|
<span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.*</span><span class="p">;</span>
|
|
<span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.types.DataTypes</span><span class="p">;</span>
|
|
<span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.types.StructField</span><span class="p">;</span>
|
|
<span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.types.StructType</span><span class="p">;</span>
|
|
<span class="kn">import</span><span class="w"> </span><span class="nn">java.util.*</span><span class="p">;</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ul>
|
|
</li><li id="dli_09_0100__en-us_topic_0200509994_li11272141817195">Create a session.<div class="codecoloring" codetype="Java" id="dli_09_0100__en-us_topic_0200509994_screen17320101315392"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span>
|
|
<span class="normal">8</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">SparkConf</span><span class="w"> </span><span class="n">sparkConf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">SparkConf</span><span class="p">();</span>
|
|
<span class="n">sparkConf</span><span class="p">.</span><span class="na">setAppName</span><span class="p">(</span><span class="s">"datasource-redis"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">.</span><span class="na">set</span><span class="p">(</span><span class="s">"spark.redis.host"</span><span class="p">,</span><span class="w"> </span><span class="s">"192.168.4.199"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">.</span><span class="na">set</span><span class="p">(</span><span class="s">"spark.redis.port"</span><span class="p">,</span><span class="w"> </span><span class="s">"6379"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">.</span><span class="na">set</span><span class="p">(</span><span class="s">"spark.redis.auth"</span><span class="p">,</span><span class="w"> </span><span class="s">"******"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">.</span><span class="na">set</span><span class="p">(</span><span class="s">"spark.driver.allowMultipleContexts"</span><span class="p">,</span><span class="s">"true"</span><span class="p">);</span>
|
|
<span class="n">JavaSparkContext</span><span class="w"> </span><span class="n">javaSparkContext</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">JavaSparkContext</span><span class="p">(</span><span class="n">sparkConf</span><span class="p">);</span>
|
|
<span class="n">SQLContext</span><span class="w"> </span><span class="n">sqlContext</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">SQLContext</span><span class="p">(</span><span class="n">javaSparkContext</span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ol>
|
|
</li><li id="dli_09_0100__li165931754161916">Connecting to data sources through DataFrame APIs<ol id="dli_09_0100__en-us_topic_0200509994_ol62934313101"><li id="dli_09_0100__en-us_topic_0200509994_li4293143141018">Read JSON data as DataFrames.<div class="codecoloring" codetype="Java" id="dli_09_0100__en-us_topic_0200509994_screen179101743194419"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">JavaRDD</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="n">javaRDD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">javaSparkContext</span><span class="p">.</span><span class="na">parallelize</span><span class="p">(</span><span class="n">Arrays</span><span class="p">.</span><span class="na">asList</span><span class="p">(</span>
|
|
<span class="w"> </span><span class="s">"{\"id\":\"1\",\"name\":\"Ann\",\"age\":\"18\"}"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s">"{\"id\":\"2\",\"name\":\"lisi\",\"age\":\"21\"}"</span><span class="p">));</span>
|
|
<span class="n">Dataset</span><span class="w"> </span><span class="n">dataFrame</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sqlContext</span><span class="p">.</span><span class="na">read</span><span class="p">().</span><span class="na">json</span><span class="p">(</span><span class="n">javaRDD</span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_09_0100__en-us_topic_0200509994_li10107173841110">Construct the Redis connection parameters.<div class="codecoloring" codetype="Java" id="dli_09_0100__en-us_topic_0200509994_screen1971113612489"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">Map</span><span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">HashMap</span><span class="o"><</span><span class="n">String</span><span class="p">,</span><span class="w"> </span><span class="n">String</span><span class="o">></span><span class="p">();</span>
|
|
<span class="n">map</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="s">"table"</span><span class="p">,</span><span class="s">"person"</span><span class="p">);</span>
|
|
<span class="n">map</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="s">"key.column"</span><span class="p">,</span><span class="s">"id"</span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_09_0100__en-us_topic_0200509994_li1956013474119">Save data to Redis.<div class="codecoloring" codetype="Java" id="dli_09_0100__en-us_topic_0200509994_screen7259610133515"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">dataFrame</span><span class="p">.</span><span class="na">write</span><span class="p">().</span><span class="na">format</span><span class="p">(</span><span class="s">"redis"</span><span class="p">).</span><span class="na">options</span><span class="p">(</span><span class="n">map</span><span class="p">).</span><span class="na">mode</span><span class="p">(</span><span class="n">SaveMode</span><span class="p">.</span><span class="na">Overwrite</span><span class="p">).</span><span class="na">save</span><span class="p">();</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_09_0100__en-us_topic_0200509994_li106413175281">Read data from Redis.<div class="codecoloring" codetype="Java" id="dli_09_0100__en-us_topic_0200509994_screen19601195712503"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sqlContext</span><span class="p">.</span><span class="na">read</span><span class="p">().</span><span class="na">format</span><span class="p">(</span><span class="s">"redis"</span><span class="p">).</span><span class="na">options</span><span class="p">(</span><span class="n">map</span><span class="p">).</span><span class="na">load</span><span class="p">().</span><span class="na">show</span><span class="p">();</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ol>
|
|
</li><li id="dli_09_0100__li139917304208">Submitting a Spark job<ol id="dli_09_0100__en-us_topic_0200509994_ol612481914610"><li id="dli_09_0100__li8833102503017">Upload the Java code file to DLI.<p id="dli_09_0100__p1062183393010"><a name="dli_09_0100__li8833102503017"></a><a name="li8833102503017"></a></p>
|
|
<p id="dli_09_0100__p179917261302"></p>
|
|
</li><li id="dli_09_0100__li559673623017">In the Spark job editor, select the corresponding dependency module and execute the Spark job.<p id="dli_09_0100__p06412041193011"><a name="dli_09_0100__li559673623017"></a><a name="li559673623017"></a></p>
|
|
<div class="p" id="dli_09_0100__p2278103819305"><div class="note" id="dli_09_0100__en-us_topic_0200509994_note1435543551919"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0100__en-us_topic_0200509994_ul17825285811"><li id="dli_09_0100__en-us_topic_0197738142_li58215295819">If the Spark version is 2.3.2 (will be offline soon) or 2.4.5, specify the <strong id="dli_09_0100__b18502654105245">Module</strong> to <strong id="dli_09_0100__b654202197105245">sys.datasource.redis</strong> when you submit a job.</li><li id="dli_09_0100__li6624653171317">If the Spark version is 3.1.1, you do not need to select the Module module. You need to configure the 'Spark parameter (--conf) '.<p id="dli_09_0100__p292175763011"><a name="dli_09_0100__li6624653171317"></a><a name="li6624653171317"></a>spark.driver.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/redis/*</p>
|
|
<p id="dli_09_0100__p392165713308">spark.executor.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/redis/*</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</div>
|
|
</li></ol>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_09_0100__section3403192673815"><h4 class="sectiontitle">Complete Example Code</h4><div class="codecoloring" codetype="Java" id="dli_09_0100__en-us_topic_0200509995_screen1659318340448"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span>
|
|
<span class="normal">16</span>
|
|
<span class="normal">17</span>
|
|
<span class="normal">18</span>
|
|
<span class="normal">19</span>
|
|
<span class="normal">20</span>
|
|
<span class="normal">21</span>
|
|
<span class="normal">22</span>
|
|
<span class="normal">23</span>
|
|
<span class="normal">24</span>
|
|
<span class="normal">25</span>
|
|
<span class="normal">26</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">Test_Redis_DaraFrame</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">String</span><span class="o">[]</span><span class="w"> </span><span class="n">args</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
|
|
<span class="w"> </span><span class="c1">//create a SparkSession session </span>
|
|
<span class="w"> </span><span class="n">SparkConf</span><span class="w"> </span><span class="n">sparkConf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">SparkConf</span><span class="p">();</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="n">sparkConf</span><span class="p">.</span><span class="na">setAppName</span><span class="p">(</span><span class="s">"datasource-redis"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">.</span><span class="na">set</span><span class="p">(</span><span class="s">"spark.redis.host"</span><span class="p">,</span><span class="w"> </span><span class="s">"192.168.4.199"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">.</span><span class="na">set</span><span class="p">(</span><span class="s">"spark.redis.port"</span><span class="p">,</span><span class="w"> </span><span class="s">"6379"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">.</span><span class="na">set</span><span class="p">(</span><span class="s">"spark.redis.auth"</span><span class="p">,</span><span class="w"> </span><span class="s">"******"</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="p">.</span><span class="na">set</span><span class="p">(</span><span class="s">"spark.driver.allowMultipleContexts"</span><span class="p">,</span><span class="s">"true"</span><span class="p">);</span>
|
|
<span class="w"> </span><span class="n">JavaSparkContext</span><span class="w"> </span><span class="n">javaSparkContext</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">JavaSparkContext</span><span class="p">(</span><span class="n">sparkConf</span><span class="p">);</span>
|
|
<span class="w"> </span><span class="n">SQLContext</span><span class="w"> </span><span class="n">sqlContext</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">SQLContext</span><span class="p">(</span><span class="n">javaSparkContext</span><span class="p">);</span>
|
|
|
|
<span class="w"> </span><span class="c1">//Read RDD in JSON format to create DataFrame</span>
|
|
<span class="w"> </span><span class="n">JavaRDD</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="n">javaRDD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">javaSparkContext</span><span class="p">.</span><span class="na">parallelize</span><span class="p">(</span><span class="n">Arrays</span><span class="p">.</span><span class="na">asList</span><span class="p">(</span>
|
|
<span class="w"> </span><span class="s">"{\"id\":\"1\",\"name\":\"Ann\",\"age\":\"18\"}"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s">"{\"id\":\"2\",\"name\":\"lisi\",\"age\":\"21\"}"</span><span class="p">));</span>
|
|
<span class="w"> </span><span class="n">Dataset</span><span class="w"> </span><span class="n">dataFrame</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sqlContext</span><span class="p">.</span><span class="na">read</span><span class="p">().</span><span class="na">json</span><span class="p">(</span><span class="n">javaRDD</span><span class="p">);</span>
|
|
|
|
<span class="w"> </span><span class="n">Map</span><span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">HashMap</span><span class="o"><</span><span class="n">String</span><span class="p">,</span><span class="w"> </span><span class="n">String</span><span class="o">></span><span class="p">();</span>
|
|
<span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="s">"table"</span><span class="p">,</span><span class="s">"person"</span><span class="p">);</span>
|
|
<span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="s">"key.column"</span><span class="p">,</span><span class="s">"id"</span><span class="p">);</span>
|
|
<span class="w"> </span><span class="n">dataFrame</span><span class="p">.</span><span class="na">write</span><span class="p">().</span><span class="na">format</span><span class="p">(</span><span class="s">"redis"</span><span class="p">).</span><span class="na">options</span><span class="p">(</span><span class="n">map</span><span class="p">).</span><span class="na">mode</span><span class="p">(</span><span class="n">SaveMode</span><span class="p">.</span><span class="na">Overwrite</span><span class="p">).</span><span class="na">save</span><span class="p">();</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="n">sqlContext</span><span class="p">.</span><span class="na">read</span><span class="p">().</span><span class="na">format</span><span class="p">(</span><span class="s">"redis"</span><span class="p">).</span><span class="na">options</span><span class="p">(</span><span class="n">map</span><span class="p">).</span><span class="na">load</span><span class="p">().</span><span class="na">show</span><span class="p">();</span>
|
|
|
|
<span class="w"> </span><span class="p">}</span>
|
|
<span class="p">}</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0093.html">Connecting to Redis</a></div>
|
|
</div>
|
|
</div>
|
|
|