forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
47 lines
4.0 KiB
HTML
47 lines
4.0 KiB
HTML
<a name="mrs_01_2021"></a><a name="mrs_01_2021"></a>
|
|
|
|
<h1 class="topictitle1">Why Does the Realm Information Fail to Be Obtained When SparkBench is Run on HiBench for the Cluster in Security Mode?</h1>
|
|
<div id="body1595920219937"><div class="section" id="mrs_01_2021__sd3c8c2042f1a4c02ab3b455cc0dffefa"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2021__af39f16375c6441aa8f92d763ce93ca3e">Execution of the sparkbench task (for example, Wordcount) of HiBench6 fails. The bench.log indicates that the Yarn task fails to be executed. The failure information displayed on the Yarn UI is as follows:</p>
|
|
<pre class="screen" id="mrs_01_2021__screen1980318256371">Exception in thread "main" org.apache.spark.SparkException: Unable to load YARN support
|
|
at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:390)
|
|
at org.apache.spark.deploy.SparkHadoopUtil$.yarn$lzycompute(SparkHadoopUtil.scala:385)
|
|
at org.apache.spark.deploy.SparkHadoopUtil$.yarn(SparkHadoopUtil.scala:385)
|
|
at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:410)
|
|
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:796)
|
|
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:821)
|
|
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
|
|
Caused by: java.lang.IllegalArgumentException: Can't get Kerberos realm
|
|
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
|
|
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:288)
|
|
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:336)
|
|
at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:51)
|
|
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.<init>(YarnSparkHadoopUtil.scala:49)
|
|
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
|
|
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
|
|
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
|
|
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
|
|
at java.lang.Class.newInstance(Class.java:442)
|
|
at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:387)
|
|
... 6 more
|
|
Caused by: java.lang.reflect.InvocationTargetException
|
|
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
|
|
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
|
|
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
|
|
at java.lang.reflect.Method.invoke(Method.java:498)
|
|
at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88)
|
|
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
|
|
... 16 more
|
|
Caused by: KrbException: Cannot locate default realm
|
|
at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
|
|
... 22 more</pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_2021__s8853a53f1c3344d5ab108862179838aa"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_2021__a1a60917f0cff4f8b92784e82ebbcdd0b">In C80SPC200 and later, the file stored in the <strong id="mrs_01_2021__b1559519410104">/etc/krb5.conf</strong> directory is no longer replaced during cluster installation. Instead, the file is stored in the corresponding path on the client through parameter configurations, and HiBench does not reference the client configuration file. Solution: Use the file stored in the <strong id="mrs_01_2021__b377019621013">/opt/client/KrbClient/kerberos/var/krb5kdc/krb5.conf</strong> directory on the client to overwrite that in the <strong id="mrs_01_2021__b177010681018">/etc/krb5.conf</strong> directories of all nodes. Make a backup before the overwriting.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2003.html">Spark Core</a></div>
|
|
</div>
|
|
</div>
|
|
|