yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
0b3a43371a4bcc99577594de0a685249d676a567
doc-exports/docs/dli/dev/dli_09_0193.html
Hasko, Vladimir cfc48b3aed dli_dev_0104_version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-committed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
2024-05-06 09:14:57 +00:00

114 lines
13 KiB
HTML
Raw Blame History

<a name="dli_09_0193"></a><a name="dli_09_0193"></a>
<h1 class="topictitle1">Java Example Code</h1>
<div id="body0000001135191102"><div class="section" id="dli_09_0193__section1523149131718"><h4 class="sectiontitle">Development Description</h4><p id="dli_09_0193__p161043221524">This example applies only to MRS OpenTSDB.</p>
<ul id="dli_09_0193__ul27577387185"><li id="dli_09_0193__li695651911134">Prerequisites<p id="dli_09_0193__en-us_topic_0200509991_p1944354710257"><a name="dli_09_0193__li695651911134"></a><a name="li695651911134"></a>A datasource connection has been created and bound to a queue on the DLI management console. </p>
<div class="note" id="dli_09_0193__note1358715714155"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0193__p692572617287">Hard-coded or plaintext passwords pose significant security risks. To ensure security, encrypt your passwords, store them in configuration files or environment variables, and decrypt them when needed.</p>
</div></div>
</li><li id="dli_09_0193__li19554172391910">Code implementation<ol id="dli_09_0193__en-us_topic_0200509994_ol12123050181818"><li id="dli_09_0193__en-us_topic_0190647826_li1822810810586">Import dependencies.<ul id="dli_09_0193__ul0528315185614"><li id="dli_09_0193__li145285157562">Maven dependency involved<div class="codecoloring" codetype="Scala" id="dli_09_0193__en-us_topic_0190647826_screen5760163172012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">spark</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span class="n">sql_2</span><span class="mf">.11</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="mf">2.3.2</span><span class="o">&lt;/</span><span class="n">version</span><span class="o">&gt;</span>
<span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0193__en-us_topic_0200509994_li1612316509182">Import dependency packages.<div class="codecoloring" codetype="Java" id="dli_09_0193__en-us_topic_0200509994_screen68181719144911"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</li><li id="dli_09_0193__en-us_topic_0200509994_li11272141817195">Create a session.<div class="codecoloring" codetype="Java" id="dli_09_0193__en-us_topic_0200509994_screen17320101315392"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">&quot;datasource-opentsdb&quot;</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</li><li id="dli_09_0193__li85121163417">Connecting to data sources through SQL APIs<ul id="dli_09_0193__ul45041116545"><li id="dli_09_0193__en-us_topic_0190647826_li1216955517548">Create a table to connect to an MRS OpenTSDB data source and set connection parameters.<div class="codecoloring" codetype="Scala" id="dli_09_0193__en-us_topic_0190647826_screen187422020161315"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;create table opentsdb_new_test using opentsdb options('Host'='10.0.0.171:4242','metric'='ctopentsdb','tags'='city,location')&quot;</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<div class="note" id="dli_09_0193__note11208175172514"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0193__p7208951254">For details about the <strong id="dli_09_0193__b13936153811214">Host</strong>, <strong id="dli_09_0193__b1593783820126">metric</strong>, and <strong id="dli_09_0193__b1937113831214">tags</strong> parameters, see <a href="dli_09_0065.html#dli_09_0065__en-us_topic_0190597601_table463015581831">Table 1</a>.</p>
</div></div>
</li><li id="dli_09_0193__en-us_topic_0190647826_li911302371611">Insert data.<div class="codecoloring" codetype="Scala" id="dli_09_0193__en-us_topic_0190647826_screen1711312301613"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;insert into opentsdb_new_test values('Penglai', 'abc', '2021-06-30 18:00:00', 30.0)&quot;</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0193__en-us_topic_0190647826_li513113315552">Query data.<div class="codecoloring" codetype="Scala" id="dli_09_0193__en-us_topic_0190647826_screen8335528349"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;select * from opentsdb_new_test&quot;</span><span class="p">).</span><span class="n">show</span><span class="p">();</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</li><li id="dli_09_0193__li38691430113719">Submitting a Spark job<ol id="dli_09_0193__ol4869123010374"><li id="dli_09_0193__li66600534391">Generate a JAR package based on the code file and upload the package to DLI.<p id="dli_09_0193__p1196365714396"><a name="dli_09_0193__li66600534391"></a><a name="li66600534391"></a></p>
<p id="dli_09_0193__p112451855173915"></p>
</li><li id="dli_09_0193__li116994212405">In the Spark job editor, select the corresponding dependency module and execute the Spark job.<p id="dli_09_0193__p8285128134015"><a name="dli_09_0193__li116994212405"></a><a name="li116994212405"></a></p>
<div class="p" id="dli_09_0193__p78513413403"><div class="note" id="dli_09_0193__note886918308376"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0193__ul386983014371"><li id="dli_09_0193__en-us_topic_0197738142_li58215295819">If the Spark version is 2.3.2 (will be offline soon) or 2.4.5, specify the <strong id="dli_09_0193__b20965649194917">Module</strong> to <strong id="dli_09_0193__b9965249154911">sys.datasource.opentsdb</strong> when you submit a job.</li><li id="dli_09_0193__li6624653171317">If the Spark version is 3.1.1, you do not need to select a module. Configure <strong id="dli_09_0193__b6580135094910">Spark parameters (--conf)</strong>.<p id="dli_09_0193__p1723617371259">spark.driver.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/opentsdb/*</p>
<p id="dli_09_0193__p6236153714259">spark.executor.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/opentsdb/*</p>
</li></ul>
</div></div>
</div>
</li></ol>
</li></ul>
</div>
<div class="section" id="dli_09_0193__section3403192673815"><h4 class="sectiontitle">Complete Example Code</h4><ul id="dli_09_0193__ul102491411585"><li id="dli_09_0193__li11846312145819">Maven dependency involved<pre class="screen" id="dli_09_0193__screen102128338585">&lt;dependency&gt;
&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
&lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
&lt;version&gt;2.3.2&lt;/version&gt;
&lt;/dependency&gt;</pre>
</li><li id="dli_09_0193__li22491742581">Connecting to data sources through SQL APIs<div class="codecoloring" codetype="Java" id="dli_09_0193__en-us_topic_0200509995_screen1659318340448"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
<span class="w"> </span>
<span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">java_mrs_opentsdb</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span>
<span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">SparkSession</span><span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">null</span><span class="p">;</span>
<span class="w"> </span>
<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">String</span><span class="o">[]</span><span class="w"> </span><span class="n">args</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">//create a SparkSession session</span>
<span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">&quot;datasource-opentsdb&quot;</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
<span class="w"> </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;create table opentsdb_new_test using opentsdb options('Host'='10.0.0.171:4242','metric'='ctopentsdb','tags'='city,location')&quot;</span><span class="p">);</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">//*****************************SQL module***********************************</span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;insert into opentsdb_new_test values('Penglai', 'abc', '2021-06-30 18:00:00', 30.0)&quot;</span><span class="p">);</span>
<span class="w"> </span><span class="n">System</span><span class="p">.</span><span class="na">out</span><span class="p">.</span><span class="na">println</span><span class="p">(</span><span class="s">&quot;Penglai new timestamp&quot;</span><span class="p">);</span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;select * from opentsdb_new_test&quot;</span><span class="p">).</span><span class="na">show</span><span class="p">();</span>
<span class="w"> </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">close</span><span class="p">();</span>
<span class="w"> </span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0080.html">Connecting to OpenTSDB</a></div>
</div>
</div>
View Git Blame Copy Permalink