forked from docs/doc-exports
Su, Xiaomeng
04d4597cf3
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
69 lines
5.6 KiB
HTML
69 lines
5.6 KiB
HTML
<a name="dli_08_0266"></a><a name="dli_08_0266"></a>
|
|
|
|
<h1 class="topictitle1">Common Configuration Items of Batch SQL Jobs</h1>
|
|
<div id="body1586845347201"><p id="dli_08_0266__p36393515233">This section describes the common configuration items of the SQL syntax for DLI batch jobs.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0266__table334825142314" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Common configuration items</caption><thead align="left"><tr id="dli_08_0266__row66396519230"><th align="left" class="cellrowborder" valign="top" width="28.349999999999998%" id="mcps1.3.2.2.4.1.1"><p id="dli_08_0266__p463913514234">Item</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="9.56%" id="mcps1.3.2.2.4.1.2"><p id="dli_08_0266__p2639158232">Default Value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="62.09%" id="mcps1.3.2.2.4.1.3"><p id="dli_08_0266__p1963920542313">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0266__row1363935122313"><td class="cellrowborder" valign="top" width="28.349999999999998%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0266__p56404542316">spark.sql.files.maxRecordsPerFile</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.56%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0266__p1864016519235">0</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="62.09%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0266__p186405592312">Maximum number of records to be written into a single file. If the value is zero or negative, there is no limit.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0266__row8640135142317"><td class="cellrowborder" valign="top" width="28.349999999999998%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0266__p664017510239">spark.sql.autoBroadcastJoinThreshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.56%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0266__p1664018542314">209715200</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="62.09%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0266__p15640157231">Maximum size of the table that displays all working nodes when a connection is executed. You can set this parameter to <span class="parmvalue" id="dli_08_0266__parmvalue25351449152616"><b>-1</b></span> to disable the display.</p>
|
|
<div class="note" id="dli_08_0266__note1562136162513"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_0266__p166418366254">Currently, only the configuration unit metastore table that runs the <b><span class="cmdname" id="dli_08_0266__cmdname86941634172615">ANALYZE TABLE COMPUTE statistics noscan</span></b> command and the file-based data source table that directly calculates statistics based on data files are supported.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0266__row364020522314"><td class="cellrowborder" valign="top" width="28.349999999999998%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0266__p146401254236">spark.sql.shuffle.partitions</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.56%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0266__p176409516238">200</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="62.09%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0266__p186400514233">Default number of partitions used to filter data for join or aggregation.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0266__row1664018513237"><td class="cellrowborder" valign="top" width="28.349999999999998%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0266__p464005132313">spark.sql.dynamicPartitionOverwrite.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.56%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0266__p4640185132312">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="62.09%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0266__p011513355428">Whether DLI overwrites the partitions where data will be written into during runtime. If you set this parameter to <strong id="dli_08_0266__b1764369104317">false</strong>, all partitions that meet the specified condition will be deleted before data overwrite starts. For example, if you set <strong id="dli_08_0266__b4874153035118">false</strong> and use INSERT OVERWRITE to write partition 2021-02 to a partitioned table that has the 2021-01 partition, this partition will be deleted.</p>
|
|
<p id="dli_08_0266__p1611519355427">If you set this parameter to <strong id="dli_08_0266__b462663524314">true</strong>, DLI does not delete partitions before overwrite starts.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0266__row56403510234"><td class="cellrowborder" valign="top" width="28.349999999999998%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0266__p106403522310">spark.sql.files.maxPartitionBytes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.56%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0266__p9640175112311">134217728</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="62.09%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0266__p6640759237">Maximum number of bytes to be packed into a single partition when a file is read.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0266__row1364118532316"><td class="cellrowborder" valign="top" width="28.349999999999998%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0266__p1641751235">spark.sql.badRecordsPath</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.56%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0266__p206411656232">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="62.09%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0266__p46411552315">Path of bad records.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0221.html">Spark SQL Syntax Reference</a></div>
|
|
</div>
|
|
</div>
|
|
|