forked from docs/doc-exports
yangtong
c285e88a17
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: yangtong <yangtong2@huawei.com> Co-committed-by: yangtong <yangtong2@huawei.com>
44 lines
11 KiB
HTML
44 lines
11 KiB
HTML
<a name="admin_guide_000416"></a><a name="admin_guide_000416"></a>
|
|
|
|
<h1 class="topictitle1">Configuring HetuEngine SQL Inspection</h1>
|
|
<div id="body0000002007756145"><div class="section" id="admin_guide_000416__en-us_topic_0000001625941690_section173901744185611"><h4 class="sectiontitle">Scenario</h4><p id="admin_guide_000416__en-us_topic_0000001625941690_p674295265614">You can configure rules for HetuEngine SQL inspection on MRS Manager and configure rule parameters as you need.</p>
|
|
</div>
|
|
<div class="section" id="admin_guide_000416__en-us_topic_0000001625941690_section4454185435615"><h4 class="sectiontitle">Prerequisites</h4><ul id="admin_guide_000416__en-us_topic_0000001625941690_ul529883419233"><li id="admin_guide_000416__en-us_topic_0000001625941690_li192982034172313">The cluster client that contains the HetuEngine service has been installed in the <strong id="admin_guide_000416__en-us_topic_0000001625941690_b54376401363">/opt/hadoopclient</strong> directory.</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li829814346234">The HetuEngine service and compute instances are running properly.</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li109871913171115">If Kerberos authentication has been enabled for the cluster, you need to create a HetuEngine user and grant related permissions to the user. In addition, you need to use Ranger to assign the user the permission to manage databases, tables, and columns of the data source.</li></ul>
|
|
</div>
|
|
<div class="section" id="admin_guide_000416__en-us_topic_0000001625941690_section1646059115617"><h4 class="sectiontitle">Constraints</h4><ul id="admin_guide_000416__en-us_topic_0000001625941690_ul1544612292247"><li id="admin_guide_000416__en-us_topic_0000001625941690_li044611299247">The default dynamic validity period of a rule is 5 minutes.</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li54461029162415">Interception and blocking rules will interrupt SQL queries, so you need to set parameters of these rules properly based on the site requirements.</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li028845316245">Blocking rules are controlled by session-level parameters of the system. To configure blocking rules, service users must have the set session permission.</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li1865113519536">For static rule <strong id="admin_guide_000416__en-us_topic_0000001625941690_b6548163753412">static_0003</strong>, the total number of joins in queries does not include Semi joins and Anti joins.</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li5166822163512">When prompt rules are configured for dynamic_0001 and dynamic_0002, prompt messages are recorded only in logs and are not displayed on the client.</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li1549921344617">The client and server send asynchronous requests. For blocking rule <strong id="admin_guide_000416__en-us_topic_0000001625941690_b85030181917">Running_0001</strong>, after the server blocks the requests, the message "Query is gone " may be displayed on the client. In this case, you can view logs to check whether the requests are blocked.</li></ul>
|
|
</div>
|
|
<div class="section" id="admin_guide_000416__en-us_topic_0000001625941690_section1345114115711"><h4 class="sectiontitle">Procedure</h4><ol id="admin_guide_000416__en-us_topic_0000001625941690_ol202351119572"><li id="admin_guide_000416__en-us_topic_0000001625941690_li86364538219"><span>Log in to MRS Manager, click <strong id="admin_guide_000416__en-us_topic_0000001625941690_b1717812411099">Cluster</strong>, and choose <strong id="admin_guide_000416__en-us_topic_0000001625941690_b117824118910">SQL Inspector</strong>. The <strong id="admin_guide_000416__en-us_topic_0000001625941690_b91780411194">SQL Inspector</strong> page is displayed.</span></li><li id="admin_guide_000416__en-us_topic_0000001625941690_li7650134142117"><a name="admin_guide_000416__en-us_topic_0000001625941690_li7650134142117"></a><a name="en-us_topic_0000001625941690_li7650134142117"></a><span>Add rules for HetuEngine by referring to <a href="admin_guide_000409.html">Adding an SQL Inspection</a>.</span><p><p id="admin_guide_000416__en-us_topic_0000001625941690_p2791339182117">For details about the rules supported by the HetuEngine SQL engine, see <a href="admin_guide_000409.html#admin_guide_000409__en-us_topic_0000001662442869_section19510043143814">MRS SQL Inspection Rules</a>.</p>
|
|
<p id="admin_guide_000416__en-us_topic_0000001625941690_p121087251228">For example, add a rule whose ID is <strong id="admin_guide_000416__en-us_topic_0000001625941690_b147296191372">static_0001</strong> to check whether count distinct appears more than two times in the SQL statement. If so, the system displays a hint.</p>
|
|
<div class="fignone" id="admin_guide_000416__en-us_topic_0000001625941690_fig172016218254"><span class="figcap"><b>Figure 1 </b>Adding a HetuEngine SQL inspection rule</span><br><span><img id="admin_guide_000416__en-us_topic_0000001625941690_image15988143172411" src="en-us_image_0000002007758273.png"></span></div>
|
|
</p></li><li id="admin_guide_000416__en-us_topic_0000001625941690_li203291026162813"><span>Log in to the node where the <span id="admin_guide_000416__text19939534125011">HetuEngine</span> client is installed and run the following command to switch to the client installation directory:</span><p><p id="admin_guide_000416__en-us_topic_0000001625941690_p28761056204315"><strong id="admin_guide_000416__en-us_topic_0000001625941690_b687615694311">cd /opt/hadoopclient</strong></p>
|
|
<p id="admin_guide_000416__en-us_topic_0000001625941690_p5224132317279">Run the following command to set environment variables:</p>
|
|
<p id="admin_guide_000416__en-us_topic_0000001625941690_p16876175617437"><strong id="admin_guide_000416__en-us_topic_0000001625941690_b12876195684320">source bigdata_env</strong></p>
|
|
</p></li><li id="admin_guide_000416__en-us_topic_0000001625941690_li8503113010214"><span>Log in to the HetuEngine client based on the cluster authentication mode.</span><p><ul id="admin_guide_000416__en-us_topic_0000001625941690_ul134861562374"><li id="admin_guide_000416__en-us_topic_0000001625941690_li8466135743912">In security mode, run the following command to authenticate the user and log in to the HetuEngine client:<p id="admin_guide_000416__en-us_topic_0000001625941690_p1853018313404"><a name="admin_guide_000416__en-us_topic_0000001625941690_li8466135743912"></a><a name="en-us_topic_0000001625941690_li8466135743912"></a><strong id="admin_guide_000416__en-us_topic_0000001625941690_b49141793477">kinit hetu_test</strong></p>
|
|
<p id="admin_guide_000416__en-us_topic_0000001625941690_p26951059143919"><strong id="admin_guide_000416__en-us_topic_0000001625941690_b134951512472">hetu-cli --catalog hive --tenant default --schema default</strong></p>
|
|
</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li66631185400">In normal mode, run the following command to log in to the HetuEngine client:<p id="admin_guide_000416__en-us_topic_0000001625941690_p2577121394014"><a name="admin_guide_000416__en-us_topic_0000001625941690_li66631185400"></a><a name="en-us_topic_0000001625941690_li66631185400"></a><strong id="admin_guide_000416__en-us_topic_0000001625941690_b1891509474">hetu-cli --catalog hive --tenant default --schema default --user hetu_test</strong></p>
|
|
<div class="note" id="admin_guide_000416__en-us_topic_0000001625941690_note14314192411561"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="admin_guide_000416__en-us_topic_0000001625941690_p101611347175719"><strong id="admin_guide_000416__en-us_topic_0000001625941690_b2277132253810">hetu_test</strong> is a service user who has at least the tenant role specified by <strong id="admin_guide_000416__en-us_topic_0000001625941690_b227812223815">--tenant</strong> and cannot be an OS user.</p>
|
|
</div></div>
|
|
</li></ul>
|
|
</p></li><li id="admin_guide_000416__en-us_topic_0000001625941690_li395254382813"><span>Check whether the current rule takes effect.</span><p><p id="admin_guide_000416__en-us_topic_0000001625941690_p5630713164813">Run the following statement to create a table:</p>
|
|
<p id="admin_guide_000416__en-us_topic_0000001625941690_p139918442485"><strong id="admin_guide_000416__en-us_topic_0000001625941690_b1359224812489">CREATE TABLE table1(id int, name varchar,rank int);</strong></p>
|
|
<p id="admin_guide_000416__en-us_topic_0000001625941690_p4991744164814"><strong id="admin_guide_000416__en-us_topic_0000001625941690_b16593134824819">INSERT INTO table1 VALUES(10,'sachin',1),(45,'rohit',2),(46,'rohit',3),(18,'virat',4),(25,'dhawan',5);</strong></p>
|
|
<p id="admin_guide_000416__en-us_topic_0000001625941690_p1963173620487">Run the following statement to query data:</p>
|
|
<p id="admin_guide_000416__en-us_topic_0000001625941690_p128911320527"><strong id="admin_guide_000416__en-us_topic_0000001625941690_b1237111290521">select count(distinct id),count(distinct id),count(distinct id),count(distinct id),count(distinct id),count(distinct id) from table1;</strong></p>
|
|
<p id="admin_guide_000416__en-us_topic_0000001625941690_p360055017283">If the number of times count distinct appears in the statement exceeds the threshold configured in <a href="#admin_guide_000416__en-us_topic_0000001625941690_li7650134142117">2</a>, the following information is displayed:</p>
|
|
<pre class="screen" id="admin_guide_000416__en-us_topic_0000001625941690_screen89495396524">WARNING: Occurrence number of 'COUNT(DISTINCT XX)' (6) reaches the hint limitation (2)</pre>
|
|
<div class="note" id="admin_guide_000416__en-us_topic_0000001625941690_note376651019533"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="admin_guide_000416__en-us_topic_0000001625941690_ul16687175613413"><li id="admin_guide_000416__en-us_topic_0000001625941690_li156881456184116">If the action set in the rule is <strong id="admin_guide_000416__en-us_topic_0000001625941690_b1437713317392">Intercept</strong> or <strong id="admin_guide_000416__en-us_topic_0000001625941690_b179715351395">Block</strong>, the following information may be displayed:<pre class="screen" id="admin_guide_000416__en-us_topic_0000001625941690_screen118438994613">Intercepted. Reason: Occurrence number of 'COUNT(DISTINCT XX)' (6) reaches the interception limitation (2)</pre>
|
|
</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li1268812564415">You can query HetuEngine SQL inspection details in logs stored in <strong id="admin_guide_000416__en-us_topic_0000001625941690_b1280915511403">hdfs://hacluster/hetuserverhistory/tenant/coordinator/application_ID/container_ID/yyyyMMdd/server.log</strong>.</li><li id="admin_guide_000416__en-us_topic_0000001625941690_li379323715403">If warning information is required for JDBC secondary development, add the following configuration for the JDBC application:<pre class="screen" id="admin_guide_000416__en-us_topic_0000001625941690_screen145701057144211">statement = connection.prepareStatement(sql.trim());
|
|
resultSet = statement.executeQuery();
|
|
SQLWarning sqlWarning = statement.getWarnings();</pre>
|
|
</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="admin_guide_000407.html">SQL Inspector</a></div>
|
|
</div>
|
|
</div>
|
|
|