yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
89b6bedc33dca71881cfa4d2b5675443cad6c039
doc-exports/docs/dli/dev/dli_09_0190.html
Su, Xiaomeng 89b6bedc33 dli_dev_0104_version 
Reviewed-by: Rechenburg, Matthias <matthias.rechenburg@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-01-08 15:25:35 +00:00

264 lines
29 KiB
HTML
Raw Blame History

<a name="dli_09_0190"></a><a name="dli_09_0190"></a>
<h1 class="topictitle1">Java Example Code</h1>
<div id="body0000001131042126"><div class="section" id="dli_09_0190__section16813101121217"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_09_0190__p882246201220">A datasource connection has been created on the DLI management console. </p>
</div>
<div class="section" id="dli_09_0190__section516483461213"><h4 class="sectiontitle">CSS Non-Security Cluster</h4><ul id="dli_09_0190__ul1469475820121"><li id="dli_09_0190__li19694558151211">Development description<ul id="dli_09_0190__ul185891014161317"><li id="dli_09_0190__li10365118121416">Code implementation<ul id="dli_09_0190__ul19201102691518"><li id="dli_09_0190__li1611152101516">Constructing dependency information and creating a Spark session<ol id="dli_09_0190__ol691905719169"><li id="dli_09_0190__li14919135714163">Import dependencies.<p id="dli_09_0190__p13973939191618"><a name="dli_09_0190__li14919135714163"></a><a name="li14919135714163"></a>Maven dependency</p>
<pre class="screen" id="dli_09_0190__screen58091527201719">&lt;dependency&gt;
&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
&lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
&lt;version&gt;2.3.2&lt;/version&gt;
&lt;/dependency&gt;</pre>
<div class="p" id="dli_09_0190__p69773601612">Import dependency packages.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen199717361167"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
</pre></div>
</td></tr></table></div>
</div>
</li><li id="dli_09_0190__li129192575164">Create a session.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen5110621161511"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">SparkSession</span> <span class="n">sparkSession</span> <span class="o">=</span> <span class="n">SparkSession</span><span class="o">.</span><span class="n">builder</span><span class="p">()</span><span class="o">.</span><span class="n">appName</span><span class="p">(</span><span class="s2">&quot;datasource-css&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getOrCreate</span><span class="p">();</span>
</pre></div>
</td></tr></table></div>
</li></ol>
</li></ul>
</li><li id="dli_09_0190__li202501156184313">Connecting to data sources through SQL APIs<ol id="dli_09_0190__ol92651247184419"><li id="dli_09_0190__li14265174754412">Create a table to connect to a CSS data source.<pre class="screen" id="dli_09_0190__screen1067172012467">sparkSession.sql("create table css_table(id long, name string) using css options( 'es.nodes' = '192.168.9.213:9200', 'es.nodes.wan.only' = 'true','resource' ='/mytest')");</pre>
</li><li id="dli_09_0190__li112217142469">Insert data.<pre class="screen" id="dli_09_0190__screen838044764618">sparkSession.sql("insert into css_table values(18, 'John'),(28, 'Bob')");</pre>
</li><li id="dli_09_0190__li56711020144616">Query data.<pre class="screen" id="dli_09_0190__screen1348152374714">sparkSession.sql("select * from css_table").show();</pre>
</li><li id="dli_09_0190__li12348923134712">Delete the datasource connection table.<pre class="screen" id="dli_09_0190__screen1515711394476">sparkSession.sql("drop table css_table");</pre>
</li></ol>
</li><li id="dli_09_0190__li64601159181511">Submitting a Spark job<ol id="dli_09_0190__en-us_topic_0197738142_ol612481914610"><li id="dli_09_0190__li1985310310235">Generate a JAR package based on the code file and upload the package to DLI.<p id="dli_09_0190__p19673599235"><a name="dli_09_0190__li1985310310235"></a><a name="li1985310310235"></a></p>
<p id="dli_09_0190__p1118735192311"></p>
</li><li id="dli_09_0190__li765713162314">In the Spark job editor, select the corresponding dependency module and execute the Spark job.<p id="dli_09_0190__p394492182310"><a name="dli_09_0190__li765713162314"></a><a name="li765713162314"></a></p>
<div class="p" id="dli_09_0190__p1383561492314"><div class="note" id="dli_09_0190__en-us_topic_0197738142_note1435543551919"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0190__en-us_topic_0197738142_ul17825285811"><li id="dli_09_0190__en-us_topic_0197738142_li58215295819">If the Spark version is 2.3.2 (will be offline soon) or 2.4.5, specify the <strong id="dli_09_0190__b1028017549391">Module</strong> to <strong id="dli_09_0190__b1280135411399">sys.datasource.css</strong> when you submit a job.</li><li id="dli_09_0190__li6624653171317">If the Spark version is 3.1.1, you do not need to select a module. Configure <strong id="dli_09_0190__b154421055123918">Spark parameters (--conf)</strong>.<p id="dli_09_0190__p13361102416273">spark.driver.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/css/*</p>
<p id="dli_09_0190__p123611724162718">spark.executor.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/css/*</p>
</li></ul>
</div></div>
</div>
</li></ol>
</li></ul>
</li><li id="dli_09_0190__li13741133941619">Complete example code<ul id="dli_09_0190__ul1890464916173"><li id="dli_09_0190__li14144959161411">Maven dependency<pre class="screen" id="dli_09_0190__screen1758131291513">&lt;dependency&gt;
&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
&lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
&lt;version&gt;2.3.2&lt;/version&gt;
&lt;/dependency&gt;</pre>
</li><li id="dli_09_0190__li844951121818">Connecting to data sources through SQL APIs<div class="codecoloring" codetype="Java" id="dli_09_0190__en-us_topic_0197738143_screen12862344162714"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">org.apache.spark.sql.*</span><span class="p">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">java_css_unsecurity</span> <span class="p">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="p">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="p">)</span> <span class="p">{</span>
<span class="n">SparkSession</span> <span class="n">sparkSession</span> <span class="o">=</span> <span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">&quot;datasource-css-unsecurity&quot;</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
<span class="c1">// Create a DLI data table for DLI-associated CSS</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;create table css_table(id long, name string) using css options( 'es.nodes' = '192.168.15.34:9200', 'es.nodes.wan.only' = 'true', 'resource' = '/mytest')&quot;</span><span class="p">);</span>
<span class="c1">//*****************************SQL model***********************************</span>
<span class="c1">// Insert data into the DLI data table</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;insert into css_table values(18, 'John'),(28, 'Bob')&quot;</span><span class="p">);</span>
<span class="c1">// Read data from DLI data table</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;select * from css_table&quot;</span><span class="p">).</span><span class="na">show</span><span class="p">();</span>
<span class="c1">// drop table</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;drop table css_table&quot;</span><span class="p">);</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">close</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</td></tr></table></div>
</li></ul>
</li></ul>
</div>
<div class="section" id="dli_09_0190__section10591738181013"><h4 class="sectiontitle">CSS Security Cluster</h4><ul id="dli_09_0190__ul6600143891015"><li id="dli_09_0190__li2717109112117">Preparations<p id="dli_09_0190__p109301030142119"><a name="dli_09_0190__li2717109112117"></a><a name="li2717109112117"></a>Generate the <strong id="dli_09_0190__b6506133721717">keystore.jks</strong> and <strong id="dli_09_0190__b175871441201720">truststore.jks</strong> files and upload them to the OBS bucket. For details, see <a href="dli_09_0189.html">CSS Security Cluster Configuration</a>.</p>
</li><li id="dli_09_0190__li13132103118206">Description of the development with HTTPS disabled<div class="p" id="dli_09_0190__p122151988258"><a name="dli_09_0190__li13132103118206"></a><a name="li13132103118206"></a>If HTTPS is disabled, <strong id="dli_09_0190__b1694619010199">keystore.jks</strong> and <strong id="dli_09_0190__b989711316196">truststore.jks</strong> files are not required. You only need to set SSL access parameters and credentials.<ul id="dli_09_0190__ul8589145192518"><li id="dli_09_0190__li1658455112514">Constructing dependency information and creating a Spark session<ol id="dli_09_0190__ol195843522513"><li id="dli_09_0190__li4230177192812">Import dependencies.<p id="dli_09_0190__p202004200289"><a name="dli_09_0190__li4230177192812"></a><a name="li4230177192812"></a>Maven dependency</p>
<pre class="screen" id="dli_09_0190__screen8239144692819">&lt;dependency&gt;
&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
&lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
&lt;version&gt;2.3.2&lt;/version&gt;
&lt;/dependency&gt;</pre>
<div class="p" id="dli_09_0190__p132261189284">Import dependency packages.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen12226191882811"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
</pre></div>
</td></tr></table></div>
</div>
</li><li id="dli_09_0190__li858418572511">Create a session.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen558412516257"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">SparkSession</span> <span class="n">sparkSession</span> <span class="o">=</span> <span class="n">SparkSession</span><span class="o">.</span><span class="n">builder</span><span class="p">()</span><span class="o">.</span><span class="n">appName</span><span class="p">(</span><span class="s2">&quot;datasource-css&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getOrCreate</span><span class="p">();</span>
</pre></div>
</td></tr></table></div>
</li></ol>
</li><li id="dli_09_0190__li458885122517">Connecting to data sources through SQL APIs<ol id="dli_09_0190__ol11588125112516"><li id="dli_09_0190__li1858818552518">Create a table to connect to a CSS data source.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen11587145132518"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">sparkSession</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="s2">&quot;create table css_table(id long, name string) using css options( 'es.nodes' = '192.168.9.213:9200', 'es.nodes.wan.only' = 'true', 'resource' = '/mytest','es.net.ssl'='false','es.net.http.auth.user'='admin','es.net.http.auth.pass'='*******')&quot;</span><span class="p">);</span>
</pre></div>
</td></tr></table></div>
<div class="note" id="dli_09_0190__note1758865122510"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0190__ul1747857101118"><li id="dli_09_0190__li20478187141115">For details about the parameters for creating a CSS datasource connection table, see <a href="dli_09_0061.html#dli_09_0061__en-us_topic_0190067468_table569314388144">Table 1</a>.</li><li id="dli_09_0190__li1503151021118">In the preceding example, HTTPS access is disabled for the CSS security cluster. Therefore, you need to set <strong id="dli_09_0190__b1621752012214">es.net.ssl</strong> to <strong id="dli_09_0190__b19857152319215">false</strong>. <strong id="dli_09_0190__b16360332132115">es.net.http.auth.user</strong> and <strong id="dli_09_0190__b15555372219">es.net.http.auth.pass</strong> are the username and password set during cluster creation, respectively.</li></ul>
</div></div>
</li><li id="dli_09_0190__li17588115132512">Insert data.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen95881520253"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">sparkSession</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="s2">&quot;insert into css_table values(18, 'John'),(28, 'Bob')&quot;</span><span class="p">);</span>
</pre></div>
</td></tr></table></div>
</li><li id="dli_09_0190__li1758815516254">Query data.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen195881572515"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">sparkSession</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="s2">&quot;select * from css_table&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">();</span>
</pre></div>
</td></tr></table></div>
</li><li id="dli_09_0190__li116441835102217">Delete the datasource connection table.<pre class="screen" id="dli_09_0190__screen0681195513223">sparkSession.sql("drop table css_table");</pre>
</li></ol>
</li><li id="dli_09_0190__li25892055254">Submitting a Spark job<ol id="dli_09_0190__ol258905102512"><li id="dli_09_0190__li6578337366">Generate a JAR package based on the code file and upload the package to DLI.<p id="dli_09_0190__p82334817366"><a name="dli_09_0190__li6578337366"></a><a name="li6578337366"></a></p>
<p id="dli_09_0190__p16445143618"></p>
</li><li id="dli_09_0190__li12724131015366">In the Spark job editor, select the corresponding dependency module and execute the Spark job.<p id="dli_09_0190__p244691523612"><a name="dli_09_0190__li12724131015366"></a><a name="li12724131015366"></a></p>
<div class="p" id="dli_09_0190__p8999151119363"><div class="note" id="dli_09_0190__note11589551255"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0190__ul6589554252"><li id="dli_09_0190__li95891552259">When submitting a job, you need to specify a dependency module named <strong id="dli_09_0190__b53344213238">sys.datasource.css</strong>.</li><li id="dli_09_0190__li1358995192516">For details about how to submit a job on the console, see </li><li id="dli_09_0190__li9589758251">For details about how to submit a job through an API, see the <strong id="dli_09_0190__b95291732112320">modules</strong> parameter in </li></ul>
</div></div>
</div>
</li></ol>
</li><li id="dli_09_0190__li35131334124911">Complete example code<ul id="dli_09_0190__ul09131254184920"><li id="dli_09_0190__li1581127143215">Maven dependency<pre class="screen" id="dli_09_0190__screen92512131344">&lt;dependency&gt;
&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
&lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
&lt;version&gt;2.3.2&lt;/version&gt;
&lt;/dependency&gt;</pre>
</li></ul>
</li></ul>
</div>
</li><li id="dli_09_0190__li11861030163713">Description of development with HTTPS enabled<ul id="dli_09_0190__ul38625302374"><li id="dli_09_0190__li198621830173711">Constructing dependency information and creating a Spark session<ol id="dli_09_0190__ol8862123011373"><li id="dli_09_0190__li118621930183714">Import dependencies.<p id="dli_09_0190__p10862930143713"><a name="dli_09_0190__li118621930183714"></a><a name="li118621930183714"></a>Maven dependency</p>
<pre class="screen" id="dli_09_0190__screen186243011375">&lt;dependency&gt;
&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
&lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
&lt;version&gt;2.3.2&lt;/version&gt;
&lt;/dependency&gt;</pre>
<p id="dli_09_0190__p986293023719">Import dependency packages.</p>
<div class="codecoloring" codetype="Python" id="dli_09_0190__screen15862143012374"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
</pre></div>
</td></tr></table></div>
</li><li id="dli_09_0190__li15862830173717">Create a session.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen28621530123717"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">SparkSession</span> <span class="n">sparkSession</span> <span class="o">=</span> <span class="n">SparkSession</span><span class="o">.</span><span class="n">builder</span><span class="p">()</span><span class="o">.</span><span class="n">appName</span><span class="p">(</span><span class="s2">&quot;datasource-css&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getOrCreate</span><span class="p">();</span>
</pre></div>
</td></tr></table></div>
</li></ol>
</li><li id="dli_09_0190__li1886710307374">Connecting to data sources through SQL APIs<ol id="dli_09_0190__ol19867103023716"><li id="dli_09_0190__li886743012376">Create a table to connect to a CSS data source.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen1886733003714"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">sparkSession</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="s2">&quot;create table css_table(id long, name string) using css options( 'es.nodes' = '192.168.13.189:9200', 'es.nodes.wan.only' = 'true', 'resource' = '/mytest','es.net.ssl'='true','es.net.ssl.keystore.location' = 'obs://Bucket name/Address/transport-keystore.jks','es.net.ssl.keystore.pass' = '**',</span>
<span class="s1">'es.net.ssl.truststore.location'</span><span class="o">=</span><span class="s1">'obs://Bucket name/Address/truststore.jks,</span>
<span class="s1">'es.net.ssl.truststore.pass'</span><span class="o">=</span><span class="s1">'***'</span><span class="p">,</span><span class="s1">'es.net.http.auth.user'</span><span class="o">=</span><span class="s1">'admin'</span><span class="p">,</span><span class="s1">'es.net.http.auth.pass'</span><span class="o">=</span><span class="s1">'**'</span><span class="p">)</span><span class="s2">&quot;);</span>
</pre></div>
</td></tr></table></div>
<div class="note" id="dli_09_0190__note486793015373"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0190__p2906143212568">For details about the parameters for creating a CSS datasource connection table, see <a href="dli_09_0061.html#dli_09_0061__en-us_topic_0190067468_table569314388144">Table 1</a>.</p>
</div></div>
</li><li id="dli_09_0190__li48681030183713">Insert data.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen2868193013710"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">sparkSession</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="s2">&quot;insert into css_table values(18, 'John'),(28, 'Bob')&quot;</span><span class="p">);</span>
</pre></div>
</td></tr></table></div>
</li><li id="dli_09_0190__li3868330133716">Query data.<div class="codecoloring" codetype="Python" id="dli_09_0190__screen9868203015371"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">sparkSession</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="s2">&quot;select * from css_table&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">();</span>
</pre></div>
</td></tr></table></div>
</li><li id="dli_09_0190__li1868030203710">Delete the datasource connection table.<pre class="screen" id="dli_09_0190__screen28681730163718">sparkSession.sql("drop table css_table");</pre>
</li></ol>
</li><li id="dli_09_0190__li38691430113719">Submitting a Spark job<ol id="dli_09_0190__ol4869123010374"><li id="dli_09_0190__li1618744813369">Generate a JAR package based on the code file and upload the package to DLI.<p id="dli_09_0190__p1468816557362"><a name="dli_09_0190__li1618744813369"></a><a name="li1618744813369"></a></p>
<p id="dli_09_0190__p1753934953613"></p>
</li><li id="dli_09_0190__li11789843155515">If HTTPS access is enabled, you need to upload the dependency file <strong id="dli_09_0190__b193836232810">hadoop-site.xml</strong> when creating a Spark job. The content of the <strong id="dli_09_0190__b1595782618329">hadoop-site.xml</strong> file is as follows:<pre class="screen" id="dli_09_0190__screen36251841203310">&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
&lt;!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
--&gt;
&lt;!-- Put site-specific property overrides in this file. --&gt;
<strong id="dli_09_0190__b14872145073319">&lt;configuration&gt;</strong>
<strong id="dli_09_0190__b78756506339">&lt;property&gt;</strong>
<strong id="dli_09_0190__b1914591303612"> &lt;name&gt;fs.obs.bucket.</strong><em id="dli_09_0190__i6147181319368">Bucket name</em><strong id="dli_09_0190__b5146121317368">.access.key&lt;/name&gt;</strong>
<strong id="dli_09_0190__b387885093320"> &lt;value&gt;AK&lt;/value&gt;</strong>
<strong id="dli_09_0190__b20879145023317"> &lt;/property&gt;</strong>
<strong id="dli_09_0190__b1287915504334">&lt;property&gt;</strong>
<strong id="dli_09_0190__b1166794053515"> &lt;name&gt;fs.obs.bucket.</strong><strong id="dli_09_0190__b666734063512">Bucket name</strong><strong id="dli_09_0190__b0667640123516">.secret.key &lt;/name&gt;</strong>
<strong id="dli_09_0190__b1387913502335"> &lt;value&gt;SK&lt;/value&gt;</strong>
<strong id="dli_09_0190__b178801550123312"> &lt;/property&gt;</strong>
<strong id="dli_09_0190__b11880155023312">&lt;/configuration&gt;</strong></pre>
<div class="note" id="dli_09_0190__note3765162103515"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0190__p2076819219359"><strong id="dli_09_0190__b13997201115409"> &lt;name&gt;fs.obs.bucket.<em id="dli_09_0190__i1495373872715">Bucket name</em>.access.key&lt;/name&gt;</strong> is used to better locate the bucket address. The bucket name is the name of the bucket where the<strong id="dli_09_0190__b119158241406"> keystore.jks</strong> and <strong id="dli_09_0190__b94212812401">truststore.jks</strong> files are stored.</p>
</div></div>
</li><li id="dli_09_0190__li144058323713">In the Spark job editor, select the corresponding dependency module and execute the Spark job.<p id="dli_09_0190__p12582178203712"><a name="dli_09_0190__li144058323713"></a><a name="li144058323713"></a></p>
<div class="p" id="dli_09_0190__p157871845370"><div class="note" id="dli_09_0190__note886918308376"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0190__ul386983014371"><li id="dli_09_0190__li198691730113719">When submitting a job, you need to specify a dependency module named <strong id="dli_09_0190__b10314145113432">sys.datasource.css</strong>.</li><li id="dli_09_0190__li1786983019376">For details about how to submit a job on the console, see </li><li id="dli_09_0190__li14870123010378">For details about how to submit a job through an API, see the <strong id="dli_09_0190__b11910162244511">modules</strong> parameter in </li></ul>
</div></div>
</div>
</li></ol>
</li><li id="dli_09_0190__li1487083018372">Complete example code<ul id="dli_09_0190__ul18870123014370"><li id="dli_09_0190__li987073063717">Maven dependency<pre class="screen" id="dli_09_0190__screen1787003043719">&lt;dependency&gt;
&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
&lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
&lt;version&gt;2.3.2&lt;/version&gt;
&lt;/dependency&gt;</pre>
</li><li id="dli_09_0190__li387363014373">Connecting to data sources through SQL APIs<div class="codecoloring" codetype="Java" id="dli_09_0190__screen787303013378"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">java_css_security_httpson</span> <span class="p">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="p">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="p">)</span> <span class="p">{</span>
<span class="n">SparkSession</span> <span class="n">sparkSession</span> <span class="o">=</span> <span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">&quot;datasource-css&quot;</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
<span class="c1">// Create a DLI data table for DLI-associated CSS</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;create table css_table(id long, name string) using css options( 'es.nodes' = '192.168.13.189:9200', 'es.nodes.wan.only' = 'true', 'resource' = '/mytest','es.net.ssl'='true','es.net.ssl.keystore.location' = 'obs://Bucket name/Address/transport-keystore.jks','es.net.ssl.keystore.pass' = '**','es.net.ssl.truststore.location'='obs://Bucket name/Address/truststore.jks','es.net.ssl.truststore.pass'='**','es.net.http.auth.user'='admin','es.net.http.auth.pass'='**')&quot;</span><span class="p">);</span>
<span class="c1">//*****************************SQL model***********************************</span>
<span class="c1">// Insert data into the DLI data table</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;insert into css_table values(34, 'Yuan'),(28, 'Kids')&quot;</span><span class="p">);</span>
<span class="c1">// Read data from DLI data table</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;select * from css_table&quot;</span><span class="p">).</span><span class="na">show</span><span class="p">();</span>
<span class="c1">// drop table</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;drop table css_table&quot;</span><span class="p">);</span>
<span class="n">sparkSession</span><span class="p">.</span><span class="na">close</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</td></tr></table></div>
</li></ul>
</li></ul>
<p id="dli_09_0190__p886104943714"></p>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0089.html">Connecting to CSS</a></div>
</div>
</div>
View Git Blame Copy Permalink