forked from docs/doc-exports
yangtong
c285e88a17
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: yangtong <yangtong2@huawei.com> Co-committed-by: yangtong <yangtong2@huawei.com>
49 lines
15 KiB
HTML
49 lines
15 KiB
HTML
<a name="admin_guide_000414"></a><a name="admin_guide_000414"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Spark SQL Inspection</h1>
|
|
<div id="body0000001971075814"><div class="section" id="admin_guide_000414__en-us_topic_0000001625860966_section173901744185611"><h4 class="sectiontitle">Scenario</h4><p id="admin_guide_000414__en-us_topic_0000001625860966_p674295265614">You can configure rules for Spark SQL inspection on MRS Manager and configure rule parameters as you need.</p>
|
|
</div>
|
|
<div class="section" id="admin_guide_000414__en-us_topic_0000001625860966_section4454185435615"><h4 class="sectiontitle">Prerequisites</h4><ul id="admin_guide_000414__en-us_topic_0000001625860966_ul529883419233"><li id="admin_guide_000414__en-us_topic_0000001625860966_li192982034172313">The cluster client that contains the Spark service has been installed in the <strong id="admin_guide_000414__en-us_topic_0000001625860966_b137930348420">/opt/hadoopclient</strong> directory.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li829814346234">Spark is running properly.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li380720312561">A tenant, for example, <strong id="admin_guide_000414__en-us_topic_0000001625860966_b31407903516">sparkstatic1</strong> has been added to <strong id="admin_guide_000414__en-us_topic_0000001625860966_b1870602510358">Tenant Resources</strong>. For details, see <a href="admin_guide_000100.html">Creating Tenants</a>.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li83361852483">For a cluster with Kerberos authentication enabled, a service user has been created, for example, user <strong id="admin_guide_000414__b125721753102614">sparkuser</strong>. The user belongs to groups <strong id="admin_guide_000414__b2572145312264">hive</strong>, <strong id="admin_guide_000414__b13572105311269">hadoop</strong>, and <strong id="admin_guide_000414__b357245316265">supergroup</strong>, the primary group is <strong id="admin_guide_000414__b957315310263">hive</strong>, and the bound role is set to <strong id="admin_guide_000414__b45737533268">sparkstatic1</strong>.</li></ul>
|
|
</div>
|
|
<div class="section" id="admin_guide_000414__en-us_topic_0000001625860966_section1646059115617"><h4 class="sectiontitle">Constraints</h4><ul id="admin_guide_000414__en-us_topic_0000001625860966_ul1544612292247"><li id="admin_guide_000414__en-us_topic_0000001625860966_li044611299247">The default dynamic validity period of a rule is 6 minutes.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li1673561511446">Only SQL jobs are supported.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li54461029162415">Interception and blocking rules will interrupt SQL queries, so you need to set parameters of these rules properly based on the site requirements.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li1274917379457">The static rule static_0007 is not required for Spark because it has a Cartesian product restriction (controlled by <strong id="admin_guide_000414__en-us_topic_0000001625860966_b73153912172">spark.sql.crossJoin.enabled</strong>, which is <strong id="admin_guide_000414__en-us_topic_0000001625860966_b17122318131715">true</strong> by default). If the spark parameter is <strong id="admin_guide_000414__en-us_topic_0000001625860966_b12275151751616">false</strong>, the static_0007 rule will not take effect.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li16687131637">Dynamic rules do work on carbon tables.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li78841653185">The dynamic rule dynamic_0002 supports SELECT, ALTER TABLE ADD PARTITION and ALTER TABLE DROP PARTITION. If you use a batch deletion statement that contains judgment conditions, for example, ALTER TABLE DROP PARTITION (pt < 10), the statement will be intercepted before the dynamic_0002 rule because the number of partitions is limited by <strong id="admin_guide_000414__en-us_topic_0000001625860966_b13730369205">spark.sql.dropPartitionsInBatch.limit</strong>, which is defaulted to <strong id="admin_guide_000414__en-us_topic_0000001625860966_b4711125218208">1000</strong>.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li20251112111114">Blocking rules has execution latency. For example, if the running_0004 rule is used and the threshold of the scanned data volume is 10 GB, the statement may be blocked when the data volume is 15 GB or higher due to the determination period and task concurrency.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li573020108168">If a job does not exceed the threshold until the last several tasks are complete before the block action is triggered, the job cannot be canceled.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li172391017163118">Blocking rule running_0004: The SQL execution duration includes the execution duration on the Driver side and the job running duration. When the SQL execution is blocked on the Driver side, the job cannot be canceled even if the execution duration exceeds the value of interruption threshold. This problem may occur when INSERT OVERWRITE is performed on a large number of partitions where storage and compute are decoupled.</li></ul>
|
|
</div>
|
|
<div class="section" id="admin_guide_000414__en-us_topic_0000001625860966_section73635164318"><h4 class="sectiontitle">Procedure</h4><ol id="admin_guide_000414__en-us_topic_0000001625860966_ol202351119572"><li id="admin_guide_000414__en-us_topic_0000001625860966_li86364538219"><span>Log in to MRS Manager, click <strong id="admin_guide_000414__en-us_topic_0000001625860966_b93713519125">Cluster</strong>, and choose <strong id="admin_guide_000414__en-us_topic_0000001625860966_b1837115201217">SQL Inspector</strong>. The <strong id="admin_guide_000414__en-us_topic_0000001625860966_b163711551214">SQL Inspector</strong> page is displayed.</span></li><li id="admin_guide_000414__en-us_topic_0000001625860966_li7650134142117"><a name="admin_guide_000414__en-us_topic_0000001625860966_li7650134142117"></a><a name="en-us_topic_0000001625860966_li7650134142117"></a><span>Add rules for Spark by referring to <a href="admin_guide_000409.html">Adding an SQL Inspection</a>.</span><p><p id="admin_guide_000414__en-us_topic_0000001625860966_p2791339182117">For details about the rules supported by the Spark SQL engine, see <a href="admin_guide_000409.html#admin_guide_000409__en-us_topic_0000001662442869_section19510043143814">MRS SQL Inspection Rules</a>.</p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p121087251228">For example, add a rule whose ID is <strong id="admin_guide_000414__en-us_topic_0000001625860966_b1253918125297">static_0001</strong> to check whether count distinct appears more than two times in the SQL statement. If so, the system displays a hint.</p>
|
|
<div class="fignone" id="admin_guide_000414__en-us_topic_0000001625860966_fig172016218254"><span class="figcap"><b>Figure 1 </b>Adding a Spark SQL inspection rule</span><br><span><img id="admin_guide_000414__en-us_topic_0000001625860966_image395113285504" src="en-us_image_0000001971077934.png"></span></div>
|
|
</p></li><li id="admin_guide_000414__en-us_topic_0000001625860966_li203291026162813"><span>Log in to the node where the Spark client is installed and run the following command to switch to the client installation directory.</span><p><p id="admin_guide_000414__en-us_topic_0000001625860966_p28761056204315"><strong id="admin_guide_000414__en-us_topic_0000001625860966_b687615694311">cd /opt/hadoopclient</strong></p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p5224132317279">Run the following command to set environment variables:</p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p74781826141113"><strong id="admin_guide_000414__en-us_topic_0000001625860966_b1747817265117">source bigdata_env</strong></p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p104237312111"><strong id="admin_guide_000414__en-us_topic_0000001625860966_b6423831121116">source Spark/component_env</strong></p>
|
|
</p></li><li id="admin_guide_000414__en-us_topic_0000001625860966_li20669114113218"><span>Perform user authentication for clusters in security mode (Kerberos authentication enabled). Skip this step for clusters in normal mode (Kerberos authentication disabled).</span><p><p id="admin_guide_000414__en-us_topic_0000001625860966_p5822164894720"><strong id="admin_guide_000414__en-us_topic_0000001625860966_b10190175032918">kinit</strong><em id="admin_guide_000414__en-us_topic_0000001625860966_i11903509297"> Spark operation user</em></p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p1282264810471">Example:</p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p17822174813476"><strong id="admin_guide_000414__en-us_topic_0000001625860966_b15822194874711">kinit</strong> <strong id="admin_guide_000414__en-us_topic_0000001625860966_b12129135795418">sparkuser</strong></p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p1182284824720">Enter the password as prompted and change the password upon your first login.</p>
|
|
</p></li><li id="admin_guide_000414__en-us_topic_0000001625860966_li1464011254019"><span>Run the following command to log in to the spark-sql client:</span><p><p id="admin_guide_000414__en-us_topic_0000001625860966_p33585391406"><strong id="admin_guide_000414__en-us_topic_0000001625860966_b764711568195">cd opt/client/Spark/spark/bin</strong></p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p1189619548193"><strong id="admin_guide_000414__en-us_topic_0000001625860966_en-us_topic_0085597464_b836354617160">./spark-sql</strong></p>
|
|
</p></li><li id="admin_guide_000414__en-us_topic_0000001625860966_li395254382813"><span>Run the following SQL statement on the client to check whether the current rule takes effect:</span><p><p id="admin_guide_000414__en-us_topic_0000001625860966_p5630713164813">Run the following statement to create a table:</p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p4948193912011"><strong id="admin_guide_000414__en-us_topic_0000001625860966_af7a958553d114a41baffc7f21a68fe91">create table table1(id int, name string) stored as parquet</strong></p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p1963173620487">Run the following statement to query data:</p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p128911320527"><strong id="admin_guide_000414__en-us_topic_0000001625860966_b1237111290521">select count(distinct id),count(distinct id),count(distinct id),count(distinct id),count(distinct id),count(distinct id) from table1;</strong></p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p360055017283">If the number of times count distinct appears in the statement exceeds the threshold configured in <a href="#admin_guide_000414__en-us_topic_0000001625860966_li7650134142117">2</a>, the following information is displayed:</p>
|
|
<pre class="screen" id="admin_guide_000414__en-us_topic_0000001625860966_screen89495396524">WARNING: static_0001 Occurrence num of 'COUNT(DISTINCT)'(6) reaches the hint threshold(2)</pre>
|
|
<div class="p" id="admin_guide_000414__en-us_topic_0000001625860966_p1493312312215">If the action set in the rule is <strong id="admin_guide_000414__en-us_topic_0000001625860966_b6915172211312">Intercept</strong>, the following information is displayed:<pre class="screen" id="admin_guide_000414__en-us_topic_0000001625860966_screen1946112612217">Error in query: static_0001 Occurrence num of 'COUNT(DISTINCT)'(6) reaches the intercept threshold(2)</pre>
|
|
</div>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p164081019228">In Spark Beeline, you can obtain SQL inspection details from logs.</p>
|
|
<ol type="a" id="admin_guide_000414__en-us_topic_0000001625860966_ol189714421330"><li id="admin_guide_000414__en-us_topic_0000001625860966_li178974421638">Log in to MRS Manager and choose <strong id="admin_guide_000414__en-us_topic_0000001625860966_b98401516314">Cluster</strong> > <strong id="admin_guide_000414__en-us_topic_0000001625860966_b96012564312">Services</strong> > <strong id="admin_guide_000414__en-us_topic_0000001625860966_b28190183212">Yarn</strong>. On the <strong id="admin_guide_000414__en-us_topic_0000001625860966_b123835713323">Dashboard</strong> page, click the link next to <strong id="admin_guide_000414__en-us_topic_0000001625860966_b104359196325">ResourceManager WebUI</strong> to enter the Yarn web UI.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li145971056837">Click the ID of the target application on the <strong id="admin_guide_000414__en-us_topic_0000001625860966_b18497114114320">All Applications</strong> page. The application details page is displayed.<p id="admin_guide_000414__en-us_topic_0000001625860966_p1086671796"><span><img id="admin_guide_000414__en-us_topic_0000001625860966_image208597291" src="en-us_image_0000001971237702.png"></span></p>
|
|
</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li143256391692">Click <strong id="admin_guide_000414__en-us_topic_0000001625860966_b54996107330">Logs</strong> of the application. On the displayed page, click <strong id="admin_guide_000414__en-us_topic_0000001625860966_b1967585273315">stdout</strong> logs to view SQL inspection details.<p id="admin_guide_000414__en-us_topic_0000001625860966_p585313438145"><span><img id="admin_guide_000414__en-us_topic_0000001625860966_image285384381417" src="en-us_image_0000002007717737.png"></span></p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p1225485219195"><span><img id="admin_guide_000414__en-us_topic_0000001625860966_image1925465217191" src="en-us_image_0000002007758277.png"></span></p>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p3355171872120"><span><img id="admin_guide_000414__en-us_topic_0000001625860966_image53554181213" src="en-us_image_0000001971077942.png"></span></p>
|
|
</li></ol>
|
|
<div class="note" id="admin_guide_000414__en-us_topic_0000001625860966_note14314192411561"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ol type="a" id="admin_guide_000414__en-us_topic_0000001625860966_ol938752555312"><li id="admin_guide_000414__en-us_topic_0000001625860966_li538710251537">For more Spark SQL inspection rules, see <a href="admin_guide_000409.html#admin_guide_000409__en-us_topic_0000001662442869_section19510043143814">MRS SQL Inspection Rules</a>.</li><li id="admin_guide_000414__en-us_topic_0000001625860966_li29051926195310">You can view query info in the <strong id="admin_guide_000414__en-us_topic_0000001625860966_b16338185161518">/opt/hadoopclient/Spark/spark/audit/query.log</strong> path if you are using the Spark client. The log contains detailed running information and the corresponding SQL inspection information.</li></ol>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
<p id="admin_guide_000414__en-us_topic_0000001625860966_p8060118"></p>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="admin_guide_000407.html">SQL Inspector</a></div>
|
|
</div>
|
|
</div>
|
|
|