forked from docs/doc-exports
yangtong
c285e88a17
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: yangtong <yangtong2@huawei.com> Co-committed-by: yangtong <yangtong2@huawei.com>
43 lines
9.5 KiB
HTML
43 lines
9.5 KiB
HTML
<a name="admin_guide_000412"></a><a name="admin_guide_000412"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Hive SQL Inspection</h1>
|
|
<div id="body0000001971235590"><div class="section" id="admin_guide_000412__en-us_topic_0000001662482833_section393324811105"><h4 class="sectiontitle">Scenario</h4><p id="admin_guide_000412__en-us_topic_0000001662482833_p674295265614">You can configure rules for Hive SQL inspection on MRS Manager and configure rule parameters as you need.</p>
|
|
</div>
|
|
<div class="section" id="admin_guide_000412__en-us_topic_0000001662482833_section4454185435615"><h4 class="sectiontitle">Prerequisites</h4><ul id="admin_guide_000412__en-us_topic_0000001662482833_ul529883419233"><li id="admin_guide_000412__en-us_topic_0000001662482833_li192982034172313">The cluster client that contains the Hive service has been installed in the <strong id="admin_guide_000412__en-us_topic_0000001662482833_b125892029104814">/opt/hadoopclient</strong> directory.</li><li id="admin_guide_000412__en-us_topic_0000001662482833_li829814346234">The Hive service of the cluster is running properly.</li><li id="admin_guide_000412__en-us_topic_0000001662482833_li83361852483">For a cluster with Kerberos authentication enabled, a user with Hive operation permissions has been created.</li></ul>
|
|
</div>
|
|
<div class="section" id="admin_guide_000412__en-us_topic_0000001662482833_section1646059115617"><h4 class="sectiontitle">Constraints</h4><ul id="admin_guide_000412__en-us_topic_0000001662482833_ul1544612292247"><li id="admin_guide_000412__en-us_topic_0000001662482833_li044611299247">By default, SQL inspection rules need 5 seconds to take effect dynamically. After the queue is modified, it takes 10 minutes for Hive inspection rules to be reloaded.</li><li id="admin_guide_000412__en-us_topic_0000001662482833_li54461029162415">Interception and blocking rules will interrupt SQL tasks, so you need to set parameters of these rules properly based on the site requirements.</li><li id="admin_guide_000412__en-us_topic_0000001662482833_li569227203611">For the rule dynamic_0001 (the number of files scanned by SQL statements exceeds the threshold), when the Spark and Tez engines reach the threshold, interception logs are printed in Yarn task logs and cannot be output on the Beeline client.</li><li id="admin_guide_000412__en-us_topic_0000001662482833_li756113524330">Blocking rules have execution latency. For example, if the running_0004 rule is used and the threshold of the scanned data volume is 10 GB, the statement may be blocked when the data volume is 15 GB or higher due to the determination period and task concurrency.</li></ul>
|
|
</div>
|
|
<div class="section" id="admin_guide_000412__en-us_topic_0000001662482833_section18371052507"><h4 class="sectiontitle">Procedure</h4><ol id="admin_guide_000412__en-us_topic_0000001662482833_ol18912336114318"><li id="admin_guide_000412__en-us_topic_0000001662482833_li16912123616434"><span>Log in to MRS Manager, click <strong id="admin_guide_000412__en-us_topic_0000001662482833_b162641448155017">Cluster</strong>, and choose <strong id="admin_guide_000412__en-us_topic_0000001662482833_b826444814503">SQL Inspector</strong>. The <strong id="admin_guide_000412__en-us_topic_0000001662482833_b72642488506">SQL Inspector</strong> page is displayed.</span></li><li id="admin_guide_000412__en-us_topic_0000001662482833_li7650134142117"><a name="admin_guide_000412__en-us_topic_0000001662482833_li7650134142117"></a><a name="en-us_topic_0000001662482833_li7650134142117"></a><span>Add rules for Hive by referring to <a href="admin_guide_000409.html">Adding an SQL Inspection</a>.</span><p><p id="admin_guide_000412__en-us_topic_0000001662482833_p2791339182117">For details about the rules supported by the Hive SQL engine, see <a href="admin_guide_000409.html#admin_guide_000409__en-us_topic_0000001662442869_section19510043143814">MRS SQL Inspection Rules</a>.</p>
|
|
<p id="admin_guide_000412__en-us_topic_0000001662482833_p121087251228">For example, add a rule whose ID is <strong id="admin_guide_000412__en-us_topic_0000001662482833_b15291192211015">static_0001</strong> to check whether count distinct appears more than two times in the SQL statement. If so, the system displays a hint.</p>
|
|
<div class="fignone" id="admin_guide_000412__en-us_topic_0000001662482833_fig152464013312"><span class="figcap"><b>Figure 1 </b>Adding a Hive SQL inspection rule</span><br><span><img id="admin_guide_000412__en-us_topic_0000001662482833_image13960749183218" src="en-us_image_0000001971237698.png"></span></div>
|
|
</p></li><li id="admin_guide_000412__en-us_topic_0000001662482833_li174811636328"><span>Log in to the node where the Hive client is installed and run the following command to switch to the client installation directory.</span><p><p id="admin_guide_000412__en-us_topic_0000001662482833_p74811338327"><strong id="admin_guide_000412__en-us_topic_0000001662482833_b24818312325">cd /opt/hadoopclient</strong></p>
|
|
<p id="admin_guide_000412__en-us_topic_0000001662482833_p5224132317279">Run the following command to set environment variables:</p>
|
|
<p id="admin_guide_000412__en-us_topic_0000001662482833_p5481113133213"><strong id="admin_guide_000412__en-us_topic_0000001662482833_b1648183183214">source bigdata_env</strong></p>
|
|
<p id="admin_guide_000412__en-us_topic_0000001662482833_p14886112193519">Run the following command to authenticate the current user. Skip this step if <span id="admin_guide_000412__en-us_topic_0000001662482833_en-us_topic_0000001394395286_ph1781712161310">Kerberos authentication is disabled for the cluster (the cluster is in normal mode)</span>.</p>
|
|
<p id="admin_guide_000412__en-us_topic_0000001662482833_p14730527203517"><strong id="admin_guide_000412__en-us_topic_0000001662482833_b118451936181717">kinit </strong><em id="admin_guide_000412__en-us_topic_0000001662482833_i1561123912179">Component service user who has the Hive operation permission</em></p>
|
|
</p></li><li id="admin_guide_000412__en-us_topic_0000001662482833_li67703111917"><span>Run the following command to log in to the Hive client:</span><p><p id="admin_guide_000412__en-us_topic_0000001662482833_p372118449914"><strong id="admin_guide_000412__en-us_topic_0000001662482833_b187264718914">beeline</strong></p>
|
|
</p></li><li id="admin_guide_000412__en-us_topic_0000001662482833_li119112253154"><span>Run the following commands to create a table and import data to the table.</span><p><p id="admin_guide_000412__en-us_topic_0000001662482833_p1865012218289"><strong id="admin_guide_000412__en-us_topic_0000001662482833_b8297171913211">drop table if exists hivetb;</strong></p>
|
|
<p id="admin_guide_000412__en-us_topic_0000001662482833_p76514272810"><strong id="admin_guide_000412__en-us_topic_0000001662482833_b14299919420">create table hivetb(a int,b int);</strong></p>
|
|
<p id="admin_guide_000412__en-us_topic_0000001662482833_p13651112132813"><strong id="admin_guide_000412__en-us_topic_0000001662482833_b1030031918218">insert into hivetb select 1,11;</strong></p>
|
|
<p id="admin_guide_000412__en-us_topic_0000001662482833_p1965111242814"><strong id="admin_guide_000412__en-us_topic_0000001662482833_b130291919210">insert into hivetb select 2,22;</strong></p>
|
|
</p></li><li id="admin_guide_000412__en-us_topic_0000001662482833_li25386310113"><span>Run the following SQL statement to check whether the current rule takes effect:</span><p><p id="admin_guide_000412__en-us_topic_0000001662482833_p115463311115"><strong id="admin_guide_000412__en-us_topic_0000001662482833_b95461935113">select count(distinct a),count(distinct b) from hivetb;</strong></p>
|
|
<p id="admin_guide_000412__en-us_topic_0000001662482833_p1754733101118">If the number of times count distinct appears in the statement exceeds the threshold configured in <a href="#admin_guide_000412__en-us_topic_0000001662482833_li7650134142117">2</a>, the following information is displayed:</p>
|
|
<pre class="screen" id="admin_guide_000412__en-us_topic_0000001662482833_screen1054713181113">...
|
|
WARN : STATIC_0001 The count(distinct X) times exceeds the limit : 2, current count distinct times : 2
|
|
...</pre>
|
|
<div class="p" id="admin_guide_000412__en-us_topic_0000001662482833_p195021831195619">If the operation set in the rule is <strong id="admin_guide_000412__en-us_topic_0000001662482833_b1893192716242">Block</strong>, the statement fails to be executed and the following information is displayed:<pre class="screen" id="admin_guide_000412__en-us_topic_0000001662482833_screen167019518156">...
|
|
Error: Error while compiling statement: FAILED: RuleException STATIC_0001 The count(distinct X) times exceeds the limit : 2, current count distinct times : 2 (state=42000,code=40000)
|
|
...</pre>
|
|
</div>
|
|
<div class="note" id="admin_guide_000412__en-us_topic_0000001662482833_note155477311118"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="admin_guide_000412__en-us_topic_0000001662482833_ul16654192512216"><li id="admin_guide_000412__en-us_topic_0000001662482833_li86546258212">For more Hive SQL inspection rules, see <a href="admin_guide_000409.html#admin_guide_000409__en-us_topic_0000001662442869_section19510043143814">MRS SQL Inspection Rules</a>.</li><li id="admin_guide_000412__en-us_topic_0000001662482833_li631643918219">You can also obtain the SQL inspection rules via logs which are stored in <strong id="admin_guide_000412__en-us_topic_0000001662482833_b12652147171811">/var/log/Bigdata/audit/hive/hiveserver/queryinfo.log</strong>.</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="admin_guide_000407.html">SQL Inspector</a></div>
|
|
</div>
|
|
</div>
|
|
|