forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
18 lines
2.3 KiB
HTML
18 lines
2.3 KiB
HTML
<a name="mrs_01_2042"></a><a name="mrs_01_2042"></a>
|
|
|
|
<h1 class="topictitle1">Why Is the "Code of method ... grows beyond 64 KB" Error Message Displayed When I Run Complex SQL Statements?</h1>
|
|
<div id="body1595920222656"><div class="section" id="mrs_01_2042__s51acc59d4cf742beaae58ec96651d018"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2042__aa36858a649ad48b381e0d0c4f8e83cdb">When I run a complex SQL statement, for example, SQL statements with multiple layers of nesting statements and a single layer statement contains a large number of logic clauses such as case when, an error message indicating that the code of a certain method exceeds 64 KB is displayed. The log is as follows:</p>
|
|
<pre class="screen" id="mrs_01_2042__s112ea2a35ef6458090b4605c2039c701">java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB</pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_2042__sb05b93701bf14ceca8f68383e6a7e2c0"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_2042__a20493896040a429397ffc8f76bb24103">If Project Tungsten is enabled, Spark will use codegen method to generate Java code for part of execution plan. However, each function in Java code to be compiled by JDK must be less than 64 KB. If complex SQL statements are run, the function in the Java code generated by codegen may exceed 64 KB, causing compilation failure.</p>
|
|
<p id="mrs_01_2042__a98c9c91e040644f48e5a3a251467f215">To solve the problem, go to the <span class="filepath" id="mrs_01_2042__fb2917f14a5ef48369e99daa63772b4ec"><b>spark-defaults.conf</b></span> file on the client and set the <span class="parmname" id="mrs_01_2042__pa3b50296f9684370827596439b793b16"><b>spark.sql.codegen.wholeStage</b></span> parameter to <span class="parmvalue" id="mrs_01_2042__pa010641060fd42c7a679191c319945ef"><b>false</b></span> to disable Project Tungsten.</p>
|
|
<p id="mrs_01_2042__aed1354e5e3d64be994e49b9db0c3e6df"></p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2022.html">Spark SQL and DataFrame</a></div>
|
|
</div>
|
|
</div>
|
|
|