forked from docs/doc-exports
yangtong
c285e88a17
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: yangtong <yangtong2@huawei.com> Co-committed-by: yangtong <yangtong2@huawei.com>
67 lines
19 KiB
HTML
67 lines
19 KiB
HTML
<a name="admin_guide_000427"></a><a name="admin_guide_000427"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Doris SQL Inspection</h1>
|
|
<div id="body0000002379757792"><p id="admin_guide_000427__p1541313443209">This function applies only to clusters of MRS 3.5.0 and later versions.</p>
|
|
<div class="section" id="admin_guide_000427__section24814312328"><h4 class="sectiontitle">Scenario</h4><p id="admin_guide_000427__p54811033326">You can configure inspection rules for Doris SQL statements on MRS Manager and set rule parameters based on your needs.</p>
|
|
</div>
|
|
<div class="section" id="admin_guide_000427__section1148112353213"><h4 class="sectiontitle">Prerequisites</h4><ul id="admin_guide_000427__ul529883419233"><li id="admin_guide_000427__li15482155334412">The node to be connected to the Doris database can communicate with the MRS cluster.</li><li id="admin_guide_000427__li717623524015">FE and BE instances are normal.</li><li id="admin_guide_000427__li8850102585410">The MySQL client has been installed.</li></ul>
|
|
</div>
|
|
<div class="section" id="admin_guide_000427__section1248116313328"><h4 class="sectiontitle">Constraints</h4><ul id="admin_guide_000427__ul1886525615356"><li id="admin_guide_000427__li986665616359">A SQL inspection rule is automatically applied in 5 minutes.</li><li id="admin_guide_000427__li103085503329">Interception and blocking rules will interrupt SQL queries, so you need to set parameters of these rules properly based on the site requirements.</li></ul>
|
|
</div>
|
|
<div class="section" id="admin_guide_000427__section73635164318"><h4 class="sectiontitle">Procedure</h4><ol id="admin_guide_000427__ol1748143173218"><li id="admin_guide_000427__li141114717911"><a name="admin_guide_000427__li141114717911"></a><a name="li141114717911"></a><span>Create a user with the Doris administrator permissions to connect to the Doris service.</span><p><ol type="a" id="admin_guide_000427__ol79542203918"><li id="admin_guide_000427__li562119891">Log in to MRS Manager as user <strong id="admin_guide_000427__b1011317333250">admin</strong>, choose <strong id="admin_guide_000427__b19971933175816">System</strong> > <strong id="admin_guide_000427__b521873545813">Permission</strong> > <strong id="admin_guide_000427__b1243173712584">Role</strong>, click <strong id="admin_guide_000427__b127632385914">Create Role</strong>, set the following parameters, and click <strong id="admin_guide_000427__b28201078599">OK</strong>.<ul id="admin_guide_000427__ul79465277113"><li id="admin_guide_000427__li10946132710113"><strong id="admin_guide_000427__b1275481420597">Role Name</strong>: Enter a role name, for example, <strong id="admin_guide_000427__b8755314205920">dorisrole</strong>.</li><li id="admin_guide_000427__li1713753816126"><strong id="admin_guide_000427__b19174205982617">Configure Resource Permission</strong>: In the resource permission list, choose <em id="admin_guide_000427__i1035161016296">Name of the target cluster</em> > <strong id="admin_guide_000427__b183505195296">Doris</strong> and select <strong id="admin_guide_000427__b133099413298">Doris Administrator Permission</strong> according to the permission you need. Click <strong id="admin_guide_000427__b4711115812916">Doris Read and Write Privileges</strong> and select <strong id="admin_guide_000427__b1129353473110">Select</strong>, <strong id="admin_guide_000427__b13325333123110">Drop</strong>, <strong id="admin_guide_000427__b5605113043112">Load</strong>, <strong id="admin_guide_000427__b286311290315">Alter</strong>, <strong id="admin_guide_000427__b20821102803115">Create</strong>, and <strong id="admin_guide_000427__b1721315278315">Grant</strong> permission in the <strong id="admin_guide_000427__b139659267306">internal</strong> row.</li></ul>
|
|
<div class="fignone" id="admin_guide_000427__fig13697834105019"><span class="figcap"><b>Figure 1 </b>Creating a Doris role</span><br><span><img id="admin_guide_000427__image8370824145011" src="en-us_image_0000002413536053.png"></span></div>
|
|
</li><li id="admin_guide_000427__li146513334165">Click <strong id="admin_guide_000427__b121581625919">User</strong>. If <span id="admin_guide_000427__ph1160117461812">Kerberos authentication is enabled for the cluster (the cluster is in security mode)</span>, add a human-machine user. If <span id="admin_guide_000427__ph19204321624">Kerberos authentication is disabled for the cluster (the cluster is in normal mode)</span>, add a machine-machine user. Bind the user to the created role.</li><li id="admin_guide_000427__li1293719332186">Change the initial password.</li></ol>
|
|
</p></li><li id="admin_guide_000427__li86364538219"><span>Log in to MRS Manager as a user with the <strong id="admin_guide_000427__b16430185913214">Manager_administrator</strong> and <strong id="admin_guide_000427__b92222029314">Manager_viewer</strong> permissions and choose <strong id="admin_guide_000427__b83346510318">Cluster</strong> > <strong id="admin_guide_000427__b14540285318">SQL Inspector</strong>.</span></li><li id="admin_guide_000427__li7650134142117"><a name="admin_guide_000427__li7650134142117"></a><a name="li7650134142117"></a><span>Add a rule for Doris by referring to <a href="admin_guide_000409.html">Adding an SQL Inspection</a>.</span><p><p id="admin_guide_000427__p2791339182117">For details about the rules supported by the Doris SQL engine, see <a href="admin_guide_000409.html#admin_guide_000409__en-us_topic_0000001662442869_section19510043143814">MRS SQL Inspection Rules</a>.</p>
|
|
<p id="admin_guide_000427__p121087251228">For example, add the <strong id="admin_guide_000427__b365013131718">static_0001</strong> rule while setting the threshold to <strong id="admin_guide_000427__b365083141710">1</strong> for the <strong id="admin_guide_000427__b146501133171">Hint</strong> action and <strong id="admin_guide_000427__b19651334179">6</strong> for the <strong id="admin_guide_000427__b665119320176">Intercept</strong> action. This rule detects a SQL statement that contains more than one <strong id="admin_guide_000427__b1365115391710">count distinct</strong>.</p>
|
|
<div class="fignone" id="admin_guide_000427__fig1043555220312"><span class="figcap"><b>Figure 2 </b>Adding a Doris SQL inspection rule</span><br><span><img id="admin_guide_000427__image172841440153018" src="en-us_image_0000002379936846.png"></span></div>
|
|
</p></li><li id="admin_guide_000427__li1464942395914"><span>Log in to the node where MySQL is installed and connect to the Doris database.</span><p><p id="admin_guide_000427__en-us_topic_0000001549411570_p4996142365316">If <span id="admin_guide_000427__ph972801217223">Kerberos authentication (security mode) has been enabled for the cluster</span>, run the following commands to connect to the Doris database:</p>
|
|
<p id="admin_guide_000427__en-us_topic_0000001549411570_p1297082110537"><strong id="admin_guide_000427__en-us_topic_0000001549411570_b11404152514556">export LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN=1</strong></p>
|
|
<ul id="admin_guide_000427__en-us_topic_0000001549411570_ul136531378287"><li id="admin_guide_000427__en-us_topic_0000001549411570_li66531437152812">Directly connect to the FE node to access the Doris database.<p id="admin_guide_000427__en-us_topic_0000001549411570_p1526916017259"><a name="admin_guide_000427__en-us_topic_0000001549411570_li66531437152812"></a><a name="en-us_topic_0000001549411570_li66531437152812"></a><strong id="admin_guide_000427__b317015820429">mysql -u</strong><em id="admin_guide_000427__i1817168164217">Database login username </em><strong id="admin_guide_000427__b7171382428">-p</strong> <strong id="admin_guide_000427__b10171985428">-P</strong><em id="admin_guide_000427__i61723812422">Connection port for FE queries </em><strong id="admin_guide_000427__b1517268184214">-h</strong><em id="admin_guide_000427__i317268114215">IP address of the Doris FE instance</em></p>
|
|
<p id="admin_guide_000427__p82131113191411">Enter the password for logging in to the database.</p>
|
|
</li><li id="admin_guide_000427__en-us_topic_0000001549411570_li019219018296">Connect the DBalancer service. DBalancer connects to the FE node to access the Doris database based on the configured policy.<p id="admin_guide_000427__en-us_topic_0000001549411570_p671153113291"><a name="admin_guide_000427__en-us_topic_0000001549411570_li019219018296"></a><a name="en-us_topic_0000001549411570_li019219018296"></a><strong id="admin_guide_000427__b11196250032">mysql -u</strong><em id="admin_guide_000427__i121963501319">Database login user</em><strong id="admin_guide_000427__b1619615012310"> -p</strong><strong id="admin_guide_000427__b1019618501938"> -P</strong><em id="admin_guide_000427__i18196350237">TCP access port of DBalancer</em><strong id="admin_guide_000427__b51968508319"> -h</strong><em id="admin_guide_000427__i21963505312">IP address of the Doris DBalancer instance</em></p>
|
|
<p id="admin_guide_000427__p198811918146">Enter the password for logging in to the database.</p>
|
|
</li></ul>
|
|
<div class="note" id="admin_guide_000427__en-us_topic_0000001549411570_note12349124011514"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="admin_guide_000427__en-us_topic_0000001549411570_ul2349134013157"><li id="admin_guide_000427__li19859105852914">The database login user is the user created in <a href="#admin_guide_000427__li141114717911">1</a> and has the Doris administrator permission.</li><li id="admin_guide_000427__en-us_topic_0000001549411570_li1535013409155">To obtain the query connection port of the Doris FE instance, you can log in to MRS Manager, choose <strong id="admin_guide_000427__b526675924216">Cluster</strong> > <strong id="admin_guide_000427__b1826619599425">Services</strong> > <strong id="admin_guide_000427__b1726745924217">Doris</strong> > <strong id="admin_guide_000427__b17267185913423">Configurations</strong>, and query the value of <strong id="admin_guide_000427__b16267155974214">query_port</strong> of the Doris service.</li><li id="admin_guide_000427__en-us_topic_0000001549411570_li384514306456">To obtain the query connection port of the Doris FE instance, you can log in to MRS Manager, choose <strong id="admin_guide_000427__b6464181911434">Cluster</strong> > <strong id="admin_guide_000427__b204651419184314">Services</strong> > <strong id="admin_guide_000427__b6465201914438">Doris</strong> > <strong id="admin_guide_000427__b246531919430">Configurations</strong>, and query the value of <strong id="admin_guide_000427__b146517196439">balancer_tcp_port</strong> of the Doris service.</li><li id="admin_guide_000427__en-us_topic_0000001549411570_li7350440191513">To obtain the IP address of the Doris FE or DBalancer instance, log in to MRS Manager of the MRS cluster and choose <strong id="admin_guide_000427__b10334182916435">Cluster</strong> > <strong id="admin_guide_000427__b2334429104316">Services</strong> > <strong id="admin_guide_000427__b1833516295432">Doris</strong> > <strong id="admin_guide_000427__b153351298435">Instances</strong> to view the service IP address of any FE or DBalancer instance.</li><li id="admin_guide_000427__en-us_topic_0000001549411570_li1350940161518">You can also use the MySQL connection software or Doris web UI to connect to the database.</li></ul>
|
|
</div></div>
|
|
<p id="admin_guide_000427__p0139113312817">Enter the password of the <strong id="admin_guide_000427__b11327251442">dorisuser</strong> user after running the command.</p>
|
|
</p></li><li id="admin_guide_000427__li10511104862113"><span>Check the configured SQL inspection rules.</span><p><p id="admin_guide_000427__p54639163447"><strong id="admin_guide_000427__b1952316231448">use __internal_schema;</strong></p>
|
|
<p id="admin_guide_000427__p865063811224"><strong id="admin_guide_000427__b1844994182212">select * from sqldefend_rule;</strong></p>
|
|
<pre class="screen" id="admin_guide_000427__screen139006442274">+-------------+----------+--------------------+--------+-----------------+------+------+------+
|
|
| rule_name | is_group | group_or_user_name | action | threshold_value | p_1 | p_2 | p_3 |
|
|
+-------------+----------+--------------------+--------+-----------------+------+------+------+
|
|
| static_0001 | 1 | A | 1 | 3 | NULL | NULL | NULL |
|
|
| static_0001 | 1 | A | 2 | 2 | NULL | NULL | NULL |
|
|
+-------------+----------+--------------------+--------+-----------------+------+------+------+</pre>
|
|
</p></li><li id="admin_guide_000427__li86191531529"><span>Create a database and switch to the database.</span><p><p id="admin_guide_000427__p185777548318"><strong id="admin_guide_000427__b65761654335">create database </strong><em id="admin_guide_000427__i15775541934">test</em><strong id="admin_guide_000427__b135771854635">;</strong></p>
|
|
<p id="admin_guide_000427__p107341632638"><strong id="admin_guide_000427__b986714581032">use</strong> <em id="admin_guide_000427__i184061171749">test</em><strong id="admin_guide_000427__b11959116741">;</strong></p>
|
|
</p></li><li id="admin_guide_000427__li3429310346"><span>Create tables <strong id="admin_guide_000427__b131549184713">t1</strong>, <strong id="admin_guide_000427__b1646316264717">t2</strong>, and <strong id="admin_guide_000427__b1228819444710">t3</strong>.</span><p><p id="admin_guide_000427__p11812125214411"><strong id="admin_guide_000427__b1727317431789">create table if not exists </strong><em id="admin_guide_000427__i17971843184">t1</em><strong id="admin_guide_000427__b11274204314820">(id int) engine=olap distributed by hash(id);</strong></p>
|
|
<p id="admin_guide_000427__p1798210179811"><strong id="admin_guide_000427__b1892174818812">create table if not exists </strong><em id="admin_guide_000427__i155722481488">t2</em><strong id="admin_guide_000427__b1993848283">(id int) engine=olap distributed by hash(id);</strong></p>
|
|
<p id="admin_guide_000427__p115235267818"><strong id="admin_guide_000427__b139231350185">create table if not exists </strong><em id="admin_guide_000427__i36561518810">t3</em><strong id="admin_guide_000427__b18923185010810">(id int) engine=olap distributed by hash(id);</strong></p>
|
|
</p></li><li id="admin_guide_000427__li1183213294"><span>Check whether the <strong id="admin_guide_000427__b19476741647524">static_0001</strong> rule has taken effect.</span><p><p id="admin_guide_000427__p1260723115918"><strong id="admin_guide_000427__b196445411795">select count(distinct id) from </strong><em id="admin_guide_000427__i32844424919">t1</em><strong id="admin_guide_000427__b46752461492"> except select count(distinct id) from </strong><em id="admin_guide_000427__i24281847194">t2</em><strong id="admin_guide_000427__b11119566918"> intersect select count(distinct id) from </strong><em id="admin_guide_000427__i12467356497">t3</em><strong id="admin_guide_000427__b7900378108"> union all select count(distinct id) from </strong><em id="admin_guide_000427__i16437386101">t1</em><strong id="admin_guide_000427__b20594112191012"> except select count(distinct id) from </strong><em id="admin_guide_000427__i19344161316106">t2</em><strong id="admin_guide_000427__b15464617181016"> intersect select count(distinct id) from </strong><em id="admin_guide_000427__i14974417141010">t3</em><strong id="admin_guide_000427__b114641417181010">;</strong></p>
|
|
<p id="admin_guide_000427__p1758655562">The number of <strong id="admin_guide_000427__b1933533919478">count distinct</strong> in this statement exceeds the threshold configured in <a href="#admin_guide_000427__li7650134142117">3</a>.</p>
|
|
<ul id="admin_guide_000427__ul1479118412716"><li id="admin_guide_000427__li8791142715">If the number of <strong id="admin_guide_000427__b5531944121920">count distinct</strong> is greater than the threshold for a hint but is no more than the threshold for interception, the statement will be executed successfully. Run the following command to view the detailed prompt information:<p id="admin_guide_000427__p12202650115215"><strong id="admin_guide_000427__b1198719357538">show warnings;</strong></p>
|
|
<pre class="screen" id="admin_guide_000427__screen16192432175715">+----------------------------------------------+-------------+------------------------------------------------------------------------------------+
|
|
| QueryId | RuleType | Message |
|
|
+----------------------------------------------+-------------+------------------------------------------------------------------------------------+
|
|
| <em id="admin_guide_000427__i9953141355820">stmt[306, 887f43753bb749c9-b6ed36350f9454a8]</em> | STATIC_RULE | static_0001 number of count(distinct) in the query <em id="admin_guide_000427__i81402024123210">6</em> more than the allowed limit <em id="admin_guide_000427__i059920218588">1</em> |
|
|
+----------------------------------------------+-------------+------------------------------------------------------------------------------------+</pre>
|
|
<p id="admin_guide_000427__p1995182355713"><em id="admin_guide_000427__i16706840132112">887f43753bb749c9-b6ed36350f9454a8</em> indicates the query ID. To view the SQL statement corresponding to the message, run the following command:</p>
|
|
<p id="admin_guide_000427__p796863705810"><strong id="admin_guide_000427__b1379463225910">select * from query_history where query_id='</strong><em id="admin_guide_000427__i18425123315594">887f43753bb749c9-b6ed36350f9454a8</em><strong id="admin_guide_000427__b679493210592">'</strong><strong id="admin_guide_000427__b13632162415916">;</strong></p>
|
|
</li><li id="admin_guide_000427__li98118618719">If number of <strong id="admin_guide_000427__b8140123152514">count distinct</strong> is greater than the interception threshold, the statement will be interrupted and fail to be executed and the following information is displayed:<pre class="screen" id="admin_guide_000427__screen1126110136166">ERROR ... detailMessage = static_0001 number of count(distinct) in the query <em id="admin_guide_000427__i25955318321">7</em> more than the allowed limit <em id="admin_guide_000427__i1112819377321">6</em></pre>
|
|
<p id="admin_guide_000427__p13900722191610"><strong id="admin_guide_000427__b4963143215168">show warnings;</strong></p>
|
|
<p id="admin_guide_000427__p177862321616">The statement output is empty.</p>
|
|
<pre class="screen" id="admin_guide_000427__screen2067233412171">Empty set (0.00 sec)</pre>
|
|
</li></ul>
|
|
<div class="note" id="admin_guide_000427__note381363515182"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="admin_guide_000427__ul4417204791811"><li id="admin_guide_000427__li156881456184116">If the action set in the SQL inspection rule is <strong id="admin_guide_000427__b0748195110485">Block</strong>, the following information may be displayed:<pre class="screen" id="admin_guide_000427__screen184981671934">ERROR ... detailMessage = <em id="admin_guide_000427__i145005719320">running_0001</em> Num of result rows num reaches the fuse threshold(<em id="admin_guide_000427__i25001171534">1000</em>), so cancel the output</pre>
|
|
</li><li id="admin_guide_000427__li1268812564415">You can query SQL inspection details in <strong id="admin_guide_000427__b156291520154910">/var/log/Bigdata/doris/fe/fe.log</strong> and <strong id="admin_guide_000427__b444632316491">/var/log/Bigdata/audit/doris/fe/fe.audit.log</strong>.</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="admin_guide_000407.html">SQL Inspector</a></div>
|
|
</div>
|
|
</div>
|
|
|