forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
39 lines
4.0 KiB
HTML
39 lines
4.0 KiB
HTML
<a name="mrs_01_1962"></a><a name="mrs_01_1962"></a>
|
|
|
|
<h1 class="topictitle1">Filtering Partitions without Paths in Partitioned Tables</h1>
|
|
<div id="body1595920207449"><div class="section" id="mrs_01_1962__s2b62bb90c325486a84fd7e0b10f07dc1"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1962__a27da25ad871a40cbbf94feb9371d9b6f">When you perform the <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_1962__c65da6d27c63f40b2bb55b6862477a542">select</span></b></i> query in Hive partitioned tables, the <strong id="mrs_01_1962__aea17bee469484532ad603985c701b6a2">FileNotFoundException</strong> exception is displayed if a specified partition path does not exist in HDFS. To avoid the preceding exception, configure <span class="parmname" id="mrs_01_1962__p4890d4c11f684c72845517b1ffba6606"><b>spark.sql.hive.verifyPartitionPath</b></span> parameter to filter partitions without paths.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1962__sc495445e54174faeb746e12fd202113f"><h4 class="sectiontitle">Procedure</h4><p id="mrs_01_1962__a62880744f2ca4555941c30f0cd5529ed">Perform either of the following methods to filter partitions without paths:</p>
|
|
<ul id="mrs_01_1962__ud6beba060aa54e8d9a5fa28a49ad1e04"><li id="mrs_01_1962__lb19f47de747445f9b0605272bd1c106c">Configure the following parameter in the <span class="filepath" id="mrs_01_1962__f81c0c5358c094ee694599604b7fc6f1e"><b>spark-defaults.conf</b></span> file on Spark client.
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1962__tf162a064840b4ebb906b9fa1cf18ba20" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1962__rcaf9afdf62c64b0aa5aa13aa20709a6c"><th align="left" class="cellrowborder" valign="top" width="33.33%" id="mcps1.3.2.3.1.2.2.4.1.1"><p id="mrs_01_1962__a60a010b78f654fa0bbf73aeb3b9340e5">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="47.43%" id="mcps1.3.2.3.1.2.2.4.1.2"><p id="mrs_01_1962__a03f0f66ae9da43e29565bb82eed95f94">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="19.24%" id="mcps1.3.2.3.1.2.2.4.1.3"><p id="mrs_01_1962__a0b693d1fa3174c819e0455cbec79598f">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1962__ra36b782eb0f5428ba8bfeb717092e0e5"><td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.2.3.1.2.2.4.1.1 "><p id="mrs_01_1962__a536d8f5eb23b4d9bb56fd0d7edc1717e">spark.sql.hive.verifyPartitionPath</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="47.43%" headers="mcps1.3.2.3.1.2.2.4.1.2 "><p id="mrs_01_1962__aa491807fda47497e9319bad825dffb24">Whether to filter partitions without paths when reading Hive partitioned tables.</p>
|
|
<p id="mrs_01_1962__a6abf8f5681ad47a287417fe75a85e337"><span class="parmvalue" id="mrs_01_1962__p1cbde5b4f2424a84be5012ee3bc5e564"><b>true</b></span>: enables the filtering</p>
|
|
<p id="mrs_01_1962__a892f863c4ff0484c97aa966001cef632"><span class="parmvalue" id="mrs_01_1962__p5c4be1b03d32400d88bd6de3533b1a61"><b>false</b></span>: disables the filtering</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.24%" headers="mcps1.3.2.3.1.2.2.4.1.3 "><p id="mrs_01_1962__ac59f6f13b52541bf8e25af1c4426556e">false</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_1962__l554b31937b5b4b4c9cdae02accbd8848">When running the spark-submit command to submit an application, configure the <span class="parmname" id="mrs_01_1962__pa7806379412b48e88099bd68ed323b3a"><b>--conf</b></span> parameter to filter partitions without paths.<div class="p" id="mrs_01_1962__ab85ad508d47c49ff8a654cbe5e6ff16e">For example:<pre class="screen" id="mrs_01_1962__sb11255fc546e46959b288af8d3b4ae01">spark-submit --class org.apache.spark.examples.SparkPi --conf spark.sql.hive.verifyPartitionPath=true $SPARK_HOME/lib/spark-examples_*.jar</pre>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|