yexinru/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
Files
c7dfd27ae10e70ece83ddca97dfaaccd70fc01c9
doc-exports/docs/dli/dev/dli_09_0197.html
Hasko, Vladimir cfc48b3aed dli_dev_0104_version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-committed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
2024-05-06 09:14:57 +00:00

271 lines
33 KiB
HTML
Raw Blame History

<a name="dli_09_0197"></a><a name="dli_09_0197"></a>
<h1 class="topictitle1">Java Example Code</h1>
<div id="body0000001135198020"><div class="section" id="dli_09_0197__section1523149131718"><h4 class="sectiontitle">Development Description</h4><p id="dli_09_0197__p161043221524">This example applies only to MRS HBase.</p>
<ul id="dli_09_0197__ul27577387185"><li id="dli_09_0197__li695651911134">Prerequisites<p id="dli_09_0197__en-us_topic_0200509991_p1944354710257"><a name="dli_09_0197__li695651911134"></a><a name="li695651911134"></a>A datasource connection has been created and bound to a queue on the DLI management console. </p>
<div class="note" id="dli_09_0197__note1358715714155"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0197__p692572617287">Hard-coded or plaintext passwords pose significant security risks. To ensure security, encrypt your passwords, store them in configuration files or environment variables, and decrypt them when needed.</p>
</div></div>
</li><li id="dli_09_0197__li19554172391910">Code implementation<ol id="dli_09_0197__en-us_topic_0200509994_ol12123050181818"><li id="dli_09_0197__en-us_topic_0190647826_li1822810810586">Import dependencies.<ul id="dli_09_0197__ul0528315185614"><li id="dli_09_0197__li145285157562">Maven dependency involved<div class="codecoloring" codetype="Scala" id="dli_09_0197__en-us_topic_0190647826_screen5760163172012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">spark</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span class="n">sql_2</span><span class="mf">.11</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="mf">2.3.2</span><span class="o">&lt;/</span><span class="n">version</span><span class="o">&gt;</span>
<span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0197__en-us_topic_0200509994_li1612316509182">Import dependency packages.<div class="codecoloring" codetype="Java" id="dli_09_0197__en-us_topic_0200509994_screen68181719144911"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</li><li id="dli_09_0197__en-us_topic_0200509994_li11272141817195">Create a session.<div class="codecoloring" codetype="Java" id="dli_09_0197__en-us_topic_0200509994_screen17320101315392"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">parkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">&quot;datasource-HBase-MRS&quot;</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</li><li id="dli_09_0197__li85121163417">Connecting to data sources through SQL APIs<ul id="dli_09_0197__ul3184151465019"><li id="dli_09_0197__li72645585016">For clusters with Kerberos authentication disabled<ol id="dli_09_0197__ol9234133825219"><li id="dli_09_0197__li19234173815528">Create a table to connect to an MRS HBase data source and set connection parameters.<div class="codecoloring" codetype="Scala" id="dli_09_0197__screen726420535011"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;CREATE TABLE testhbase(id STRING, location STRING, city STRING) using hbase OPTIONS('ZKHost'='10.0.0.63:2181','TableName'='hbtest','RowKey'='id:5','Cols'='location:info.location,city:detail.city') &quot;</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0197__li1723433845217">Insert data.<div class="codecoloring" codetype="Scala" id="dli_09_0197__screen52646512506"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;insert into testhbase values('12345','abc','xxx')&quot;</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0197__li1623483818528">Query data.<div class="codecoloring" codetype="Scala" id="dli_09_0197__screen17264451508"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;select * from testhbase&quot;</span><span class="p">).</span><span class="n">show</span><span class="p">();</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</li><li id="dli_09_0197__li10951153419617">For clusters with Kerberos authentication enabled<ol id="dli_09_0197__ol32092491853"><li id="dli_09_0197__li1820915491257">Create a table to connect to an MRS HBase data source and set connection parameters.<div class="codecoloring" codetype="Scala" id="dli_09_0197__screen13146172612617"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;CREATE TABLE testhbase(id STRING, location STRING, city STRING) using hbase OPTIONS('ZKHost'='10.0.0.63:2181','TableName'='hbtest','RowKey'='id:5','Cols'='location:info.location,city:detail.city,'krb5conf'='./krb5.conf','keytab'='./user.keytab','principal'='krbtest') &quot;</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<div class="p" id="dli_09_0197__p76730271564">If Kerberos authentication is enabled, you need to set three more parameters, as listed in <a href="#dli_09_0197__table19673142717615">Table 1</a>.
<div class="tablenoborder"><a name="dli_09_0197__table19673142717615"></a><a name="table19673142717615"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_09_0197__table19673142717615" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_09_0197__row1767302717615"><th align="left" class="cellrowborder" valign="top" width="30.349999999999998%" id="mcps1.3.1.3.3.1.2.1.1.2.2.2.3.1.1"><p id="dli_09_0197__p4673227362">Parameter and Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="69.65%" id="mcps1.3.1.3.3.1.2.1.1.2.2.2.3.1.2"><p id="dli_09_0197__p167310272612">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_09_0197__row16731271366"><td class="cellrowborder" valign="top" width="30.349999999999998%" headers="mcps1.3.1.3.3.1.2.1.1.2.2.2.3.1.1 "><p id="dli_09_0197__p667311271764">'krb5conf' = './krb5.conf'</p>
</td>
<td class="cellrowborder" valign="top" width="69.65%" headers="mcps1.3.1.3.3.1.2.1.1.2.2.2.3.1.2 "><p id="dli_09_0197__p186731727969">Path of the <strong id="dli_09_0197__b11476627174515">krb5.conf</strong> file.</p>
</td>
</tr>
<tr id="dli_09_0197__row76735278613"><td class="cellrowborder" valign="top" width="30.349999999999998%" headers="mcps1.3.1.3.3.1.2.1.1.2.2.2.3.1.1 "><p id="dli_09_0197__p7673152720618">'keytab'='./user.keytab'</p>
</td>
<td class="cellrowborder" valign="top" width="69.65%" headers="mcps1.3.1.3.3.1.2.1.1.2.2.2.3.1.2 "><p id="dli_09_0197__p16734271265">Path of the <strong id="dli_09_0197__b980562920453">keytab</strong> file.</p>
</td>
</tr>
<tr id="dli_09_0197__row66736271466"><td class="cellrowborder" valign="top" width="30.349999999999998%" headers="mcps1.3.1.3.3.1.2.1.1.2.2.2.3.1.1 "><p id="dli_09_0197__p8673027965">'principal' ='krbtest'</p>
</td>
<td class="cellrowborder" valign="top" width="69.65%" headers="mcps1.3.1.3.3.1.2.1.1.2.2.2.3.1.2 "><p id="dli_09_0197__p4673827962">Authentication username.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<p id="dli_09_0197__p11714134815813">For details about how to obtain the <strong id="dli_09_0197__b9335103312453">krb5.conf</strong> and <strong id="dli_09_0197__b933623313456">keytab</strong> files, see <a href="dli_09_0196.html#dli_09_0196__section12676527182715">Completing Configurations for Enabling Kerberos Authentication</a>.</p>
</li><li id="dli_09_0197__li820910491353">Insert data.<div class="codecoloring" codetype="Scala" id="dli_09_0197__screen11209449553"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;insert into testhbase values('95274','abc','Hongkong')&quot;</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0197__li1120994911518">Query data.<div class="codecoloring" codetype="Scala" id="dli_09_0197__screen172094498510"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;select * from testhbase&quot;</span><span class="p">).</span><span class="n">show</span><span class="p">();</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</li></ul>
</li><li id="dli_09_0197__li38691430113719">Submitting a Spark job<ol id="dli_09_0197__ol4869123010374"><li id="dli_09_0197__li178719413398">Generate a JAR package based on the code file and upload the package to DLI.<p id="dli_09_0197__p2350201012399"><a name="dli_09_0197__li178719413398"></a><a name="li178719413398"></a></p>
<p id="dli_09_0197__p1838216612397"></p>
</li><li id="dli_09_0197__li20873105417215">(Optional) Add the <strong id="dli_09_0197__b13501953114619">krb5.conf</strong> and <strong id="dli_09_0197__b85018531468">user.keytab</strong> files to the dependency files of the job when creating a Spark job in an MRS cluster with Kerberos authentication enabled. Skip this step if Kerberos authentication is not enabled for the cluster. </li><li id="dli_09_0197__li745118384224">In the Spark job editor, select the corresponding dependency module and execute the Spark job.<p id="dli_09_0197__p1960174432216"><a name="dli_09_0197__li745118384224"></a><a name="li745118384224"></a></p>
<div class="p" id="dli_09_0197__p158731539202217"><div class="note" id="dli_09_0197__note886918308376"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0197__ul386983014371"><li id="dli_09_0197__en-us_topic_0197738142_li58215295819">If the Spark version is 2.3.2 (will be offline soon) or 2.4.5, specify the <strong id="dli_09_0197__b16980133214315">Module</strong> to <strong id="dli_09_0197__b5980932174317">sys.datasource.hbase</strong> when you submit a job.</li><li id="dli_09_0197__li6624653171317">If the Spark version is 3.1.1, you do not need to select a module. Configure <strong id="dli_09_0197__b9721038164313">Spark parameters (--conf)</strong>.<p id="dli_09_0197__p1765215102311">spark.driver.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/hbase/*</p>
<p id="dli_09_0197__p1865215532311">spark.executor.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/hbase/*</p>
</li></ul>
</div></div>
</div>
</li></ol>
</li></ul>
</div>
<div class="section" id="dli_09_0197__section3403192673815"><h4 class="sectiontitle">Complete Example Code</h4><ul id="dli_09_0197__ul102491411585"><li id="dli_09_0197__li22491742581">Connecting to data sources through SQL APIs<ul id="dli_09_0197__ul8540124110419"><li id="dli_09_0197__li1149233711411">Complete example code for the cluster with Kerberos authentication <strong id="dli_09_0197__b2693146174119">disabled</strong><div class="codecoloring" codetype="Java" id="dli_09_0197__screen1049214373415"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
<span class="w"> </span>
<span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">java_mrs_hbase</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span>
<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">String</span><span class="o">[]</span><span class="w"> </span><span class="n">args</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">//create a SparkSession session</span>
<span class="w"> </span><span class="n">SparkSession</span><span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">&quot;datasource-HBase-MRS&quot;</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
<span class="w"> </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;CREATE TABLE testhbase(id STRING, location STRING, city STRING) using hbase OPTIONS('ZKHost'='10.0.0.63:2181','TableName'='hbtest','RowKey'='id:5','Cols'='location:info.location,city:detail.city') &quot;</span><span class="p">);</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">//*****************************SQL model***********************************</span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;insert into testhbase values('95274','abc','Hongkong')&quot;</span><span class="p">);</span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;select * from testhbase&quot;</span><span class="p">).</span><span class="na">show</span><span class="p">();</span>
<span class="w"> </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">close</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0197__li2415145841814">Complete example code for the cluster with Kerberos authentication <strong id="dli_09_0197__b1702348134115">enabled</strong><div class="codecoloring" codetype="Java" id="dli_09_0197__screen5415358131814"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span>
<span class="normal">41</span>
<span class="normal">42</span>
<span class="normal">43</span>
<span class="normal">44</span>
<span class="normal">45</span>
<span class="normal">46</span>
<span class="normal">47</span>
<span class="normal">48</span>
<span class="normal">49</span>
<span class="normal">50</span>
<span class="normal">51</span>
<span class="normal">52</span>
<span class="normal">53</span>
<span class="normal">54</span>
<span class="normal">55</span>
<span class="normal">56</span>
<span class="normal">57</span>
<span class="normal">58</span>
<span class="normal">59</span>
<span class="normal">60</span>
<span class="normal">61</span>
<span class="normal">62</span>
<span class="normal">63</span>
<span class="normal">64</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.SparkContext</span><span class="p">;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.SparkFiles</span><span class="p">;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">org.apache.spark.sql.SparkSession</span><span class="p">;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">java.io.File</span><span class="p">;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">java.io.FileInputStream</span><span class="p">;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">java.io.FileOutputStream</span><span class="p">;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">java.io.IOException</span><span class="p">;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">java.io.InputStream</span><span class="p">;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">java.io.OutputStream</span><span class="p">;</span>
<span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">Test_HBase_SparkSql_Kerberos</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">private</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">copyFile</span><span class="p">(</span><span class="n">File</span><span class="w"> </span><span class="n">src</span><span class="p">,</span><span class="n">File</span><span class="w"> </span><span class="n">dst</span><span class="p">)</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">IOException</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">InputStream</span><span class="w"> </span><span class="n">input</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">null</span><span class="p">;</span>
<span class="w"> </span><span class="n">OutputStream</span><span class="w"> </span><span class="n">output</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">null</span><span class="p">;</span>
<span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">input</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">FileInputStream</span><span class="p">(</span><span class="n">src</span><span class="p">);</span>
<span class="w"> </span><span class="n">output</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">FileOutputStream</span><span class="p">(</span><span class="n">dst</span><span class="p">);</span>
<span class="w"> </span><span class="kt">byte</span><span class="o">[]</span><span class="w"> </span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="kt">byte</span><span class="o">[</span><span class="mi">1024</span><span class="o">]</span><span class="p">;</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">bytesRead</span><span class="p">;</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">((</span><span class="n">bytesRead</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">input</span><span class="p">.</span><span class="na">read</span><span class="p">(</span><span class="n">buf</span><span class="p">))</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">output</span><span class="p">.</span><span class="na">write</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">bytesRead</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">finally</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="n">input</span><span class="p">.</span><span class="na">close</span><span class="p">();</span>
<span class="w"> </span><span class="n">output</span><span class="p">.</span><span class="na">close</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">String</span><span class="o">[]</span><span class="w"> </span><span class="n">args</span><span class="p">)</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">InterruptedException</span><span class="p">,</span><span class="w"> </span><span class="n">IOException</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">SparkSession</span><span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SparkSession</span><span class="p">.</span><span class="na">builder</span><span class="p">().</span><span class="na">appName</span><span class="p">(</span><span class="s">&quot;Test_HBase_SparkSql_Kerberos&quot;</span><span class="p">).</span><span class="na">getOrCreate</span><span class="p">();</span>
<span class="w"> </span><span class="n">SparkContext</span><span class="w"> </span><span class="n">sc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sparkContext</span><span class="p">();</span>
<span class="w"> </span><span class="n">sc</span><span class="p">.</span><span class="na">addFile</span><span class="p">(</span><span class="s">&quot;obs://xietest1/lzq/krb5.conf&quot;</span><span class="p">);</span>
<span class="w"> </span><span class="n">sc</span><span class="p">.</span><span class="na">addFile</span><span class="p">(</span><span class="s">&quot;obs://xietest1/lzq/user.keytab&quot;</span><span class="p">);</span>
<span class="w"> </span><span class="n">Thread</span><span class="p">.</span><span class="na">sleep</span><span class="p">(</span><span class="mi">20</span><span class="p">);</span>
<span class="w"> </span><span class="n">File</span><span class="w"> </span><span class="n">krb5_startfile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">File</span><span class="p">(</span><span class="n">SparkFiles</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="s">&quot;krb5.conf&quot;</span><span class="p">));</span>
<span class="w"> </span><span class="n">File</span><span class="w"> </span><span class="n">keytab_startfile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">File</span><span class="p">(</span><span class="n">SparkFiles</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="s">&quot;user.keytab&quot;</span><span class="p">));</span>
<span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">path_user</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">System</span><span class="p">.</span><span class="na">getProperty</span><span class="p">(</span><span class="s">&quot;user.dir&quot;</span><span class="p">);</span>
<span class="w"> </span><span class="n">File</span><span class="w"> </span><span class="n">keytab_endfile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">File</span><span class="p">(</span><span class="n">path_user</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s">&quot;/&quot;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">keytab_startfile</span><span class="p">.</span><span class="na">getName</span><span class="p">());</span>
<span class="w"> </span><span class="n">File</span><span class="w"> </span><span class="n">krb5_endfile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">File</span><span class="p">(</span><span class="n">path_user</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s">&quot;/&quot;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">krb5_startfile</span><span class="p">.</span><span class="na">getName</span><span class="p">());</span>
<span class="w"> </span><span class="n">copyFile</span><span class="p">(</span><span class="n">krb5_startfile</span><span class="p">,</span><span class="n">krb5_endfile</span><span class="p">);</span>
<span class="w"> </span><span class="n">copyFile</span><span class="p">(</span><span class="n">keytab_startfile</span><span class="p">,</span><span class="n">keytab_endfile</span><span class="p">);</span>
<span class="w"> </span><span class="n">Thread</span><span class="p">.</span><span class="na">sleep</span><span class="p">(</span><span class="mi">20</span><span class="p">);</span>
<span class="w"> </span><span class="cm">/**</span>
<span class="cm"> * Create an association table for the DLI association Hbase table</span>
<span class="cm"> */</span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;CREATE TABLE testhbase(id string,booleanf boolean,shortf short,intf int,longf long,floatf float,doublef double) &quot;</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="s">&quot;using hbase OPTIONS(&quot;</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="s">&quot;'ZKHost'='10.0.0.146:2181',&quot;</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="s">&quot;'TableName'='hbtest',&quot;</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="s">&quot;'RowKey'='id:100',&quot;</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="s">&quot;'Cols'='booleanf:CF1.booleanf,shortf:CF1.shortf,intf:CF1.intf,longf:CF2.longf,floatf:CF1.floatf,doublef:CF2.doublef',&quot;</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="s">&quot;'krb5conf'='&quot;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">path_user</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s">&quot;/krb5.conf',&quot;</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="s">&quot;'keytab'='&quot;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">path_user</span><span class="o">+</span><span class="w"> </span><span class="s">&quot;/user.keytab',&quot;</span><span class="w"> </span><span class="o">+</span>
<span class="w"> </span><span class="s">&quot;'principal'='krbtest') &quot;</span><span class="p">);</span>
<span class="w"> </span><span class="c1">//*****************************SQL model***********************************</span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;insert into testhbase values('newtest',true,1,2,3,4,5)&quot;</span><span class="p">);</span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">sql</span><span class="p">(</span><span class="s">&quot;select * from testhbase&quot;</span><span class="p">).</span><span class="na">show</span><span class="p">();</span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="na">close</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0077.html">Connecting to HBase</a></div>
</div>
</div>
View Git Blame Copy Permalink