forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
104 lines
12 KiB
HTML
104 lines
12 KiB
HTML
<a name="mrs_01_1957"></a><a name="mrs_01_1957"></a>
|
|
|
|
<h1 class="topictitle1">Setting the Log Level Dynamically</h1>
|
|
<div id="body1595920207026"><div class="section" id="mrs_01_1957__sbb91d5591c9043bb9aa749094ba25ba2"><h4 class="sectiontitle">Scenarios</h4><p id="mrs_01_1957__ad7ac920451f94c479d8aa442222bde0b">In some scenarios, to locate problems or check information by changing the log level,</p>
|
|
<p id="mrs_01_1957__a7e3e55fce88345e8aa593747dbc27405">you can add the <span class="parmname" id="mrs_01_1957__pca12fa6f399e48ca98ae1a8ac91f0033"><b>-Dlog4j.configuration.watch=true</b></span> parameter to the JVM parameter of a process before the process is started. After the process is started, you can modify the log4j configuration file corresponding to the process to change the log level.</p>
|
|
<p id="mrs_01_1957__a901bc22b82ef49a8a11e85caff8c5f9a">The following processes support the dynamic setting of log levels: driver, executor, ApplicationMaster, JobHistory and JDBCServer.</p>
|
|
<p id="mrs_01_1957__af0ff46d218cd4cefac855a55b21ccba5">Allowed log levels are as follows: FATAL, ERROR, WARN, INFO, DEBUG, TRACE, and ALL.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1957__sf4a9fd772e5e40299ae4f6d6dd099633"><h4 class="sectiontitle">Configuration Description</h4><p id="mrs_01_1957__a9083a1dd6a0642e8a09cd49ddf520963">Add the following parameters to the JVM parameter corresponding to a process.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1957__t31066ab015844b68954d233c847c0c65" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1957__rc37d5080820341238973e51f5ae89a4a"><th align="left" class="cellrowborder" valign="top" width="33.226677332266775%" id="mcps1.3.2.3.2.4.1.1"><p id="mrs_01_1957__a0f8a1cde1655403286497770a4392903"><strong id="mrs_01_1957__a570750b5fa3945f898266e8b5512dfb6">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="39.13608639136087%" id="mcps1.3.2.3.2.4.1.2"><p id="mrs_01_1957__a155160dbe0ef4e409c5115bba0eb945f"><strong id="mrs_01_1957__afd8baedaa59e42cd962e3edf8d4b4f66">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="27.637236276372363%" id="mcps1.3.2.3.2.4.1.3"><p id="mrs_01_1957__a281d231dcb044cdda8c7894bd7809a25"><strong id="mrs_01_1957__a4d616c13406d4551b995e73776887f15">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1957__rc6a65ff98069460d9aae30159c431735"><td class="cellrowborder" valign="top" width="33.226677332266775%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1957__a254fcb01c3ae4c1da0dc9aa19b228614">-Dlog4j.configuration.watch</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.13608639136087%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1957__a9b6cafc8f38849e4b59361a5dde129fb">Indicates a JVM parameter of a process. If this parameter is set to <span class="parmvalue" id="mrs_01_1957__p0427e35a0750440d97422cd350eb70db"><b>true</b></span>, the dynamic configuration of log levels is enabled.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="27.637236276372363%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1957__a398ba52dafce476b91ab662248f7d577">Left blank, indicating that the dynamic configuration of log levels is disabled</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_1957__a642da311648d4482ba2eb55ab5c90ac4"><a href="#mrs_01_1957__te63dc901a739455981a4564929997088">Table 2</a> lists the JVM parameters of the driver, executor, and ApplicationMaster processes. Configure the following parameters in the <span class="parmname" id="mrs_01_1957__p8bb260b9dcf041cfab6337201cdf3f0a"><b>spark-defaults.conf</b></span> file on the Spark client. Set the log levels of the driver, executor, and ApplicationMaster processes in the log4j configuration file specified by the <span class="filepath" id="mrs_01_1957__fc8b3aaf4a89540cfbb5d6b675b06b11f"><b>-Dlog4j.configuration</b></span> parameter.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1957__te63dc901a739455981a4564929997088"></a><a name="te63dc901a739455981a4564929997088"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1957__te63dc901a739455981a4564929997088" frame="border" border="1" rules="all"><caption><b>Table 2 </b>JVM parameters of processes (1)</caption><thead align="left"><tr id="mrs_01_1957__r678e1de0d32f45538d69102069dc678b"><th align="left" class="cellrowborder" valign="top" width="33.226677332266775%" id="mcps1.3.2.5.2.4.1.1"><p id="mrs_01_1957__a5d1e2eb63fcd448c88f8bd865ae06be9"><strong id="mrs_01_1957__a3185106d01a44529b6347100505b1f42">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="39.13608639136087%" id="mcps1.3.2.5.2.4.1.2"><p id="mrs_01_1957__a022798b24c1b4bee985db0205bb344b1"><strong id="mrs_01_1957__ae093fccedde94605affa78d57e639605">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="27.637236276372363%" id="mcps1.3.2.5.2.4.1.3"><p id="mrs_01_1957__a89a1a5f7eb92442db798efdb3a6bdd17"><strong id="mrs_01_1957__adc5a2cf6a72d4c1d9958d26ff156f565">Default Log Level</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1957__r7eba02ceb38e4fa3bc060a08925992b9"><td class="cellrowborder" valign="top" width="33.226677332266775%" headers="mcps1.3.2.5.2.4.1.1 "><p id="mrs_01_1957__ae345aa9fb9c843b6a1a010dc77fbbdc5">spark.driver.extraJavaOptions</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.13608639136087%" headers="mcps1.3.2.5.2.4.1.2 "><p id="mrs_01_1957__ac3b359f0ffed4cf49132a11964ab2ea2">Indicates the JVM parameter of the driver process.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="27.637236276372363%" headers="mcps1.3.2.5.2.4.1.3 "><p id="mrs_01_1957__a6eae46ccf9ab418e86a85c154e19e681">INFO</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1957__r14fcd02d5cdc4bcea2aa243ccae70666"><td class="cellrowborder" valign="top" width="33.226677332266775%" headers="mcps1.3.2.5.2.4.1.1 "><p id="mrs_01_1957__a81e20d3a4dd344f5a2756a102d7aa5a5">spark.executor.extraJavaOptions</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.13608639136087%" headers="mcps1.3.2.5.2.4.1.2 "><p id="mrs_01_1957__ae4b4e9f9d14046449ba0781b7263cc4e">Indicates the JVM parameter of the executor process.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="27.637236276372363%" headers="mcps1.3.2.5.2.4.1.3 "><p id="mrs_01_1957__a8f6dbf0e434a4d46a0d7c03730af676d">INFO</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1957__rf330cdae63a74f489abe2a84cf160f80"><td class="cellrowborder" valign="top" width="33.226677332266775%" headers="mcps1.3.2.5.2.4.1.1 "><p id="mrs_01_1957__a8546c2e1a0e6496399ea17bf39d9b46f">spark.yarn.am.extraJavaOptions</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.13608639136087%" headers="mcps1.3.2.5.2.4.1.2 "><p id="mrs_01_1957__a5090279b97794977aae0c7191f93e31d">Indicates the JVM parameter of the ApplicationMaster process.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="27.637236276372363%" headers="mcps1.3.2.5.2.4.1.3 "><p id="mrs_01_1957__a914723231920458582a114109e9efd8a">INFO</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_1957__a1dcfd00bed8b45729766c24b946ee520"><a href="#mrs_01_1957__t9699ea38453d469a8492f356e2b212cc">Table 3</a> describes the JVM parameters of JobHistory Server and JDBCServer. Set the parameters in the <span class="filepath" id="mrs_01_1957__face8d6e3b28b4dd9bf022d1a634e0e8e"><b>ENV_VARS</b></span> configuration file. Set the log levels of JobHistory Server and JDBCServer in the <span class="filepath" id="mrs_01_1957__f500d0ad6effa48eca176dd8b55764df8"><b>log4j.properties</b></span> configuration file.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1957__t9699ea38453d469a8492f356e2b212cc"></a><a name="t9699ea38453d469a8492f356e2b212cc"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1957__t9699ea38453d469a8492f356e2b212cc" frame="border" border="1" rules="all"><caption><b>Table 3 </b>JVM parameters of processes (2)</caption><thead align="left"><tr id="mrs_01_1957__r7ab77a6dd3c74633a178bb40abc2d8db"><th align="left" class="cellrowborder" valign="top" width="33.226677332266775%" id="mcps1.3.2.7.2.4.1.1"><p id="mrs_01_1957__a5d3c8be74642463db884e8932b73aa7a"><strong id="mrs_01_1957__a563bf63eaed84ac29f01ef9c65bfb9c1">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="39.13608639136087%" id="mcps1.3.2.7.2.4.1.2"><p id="mrs_01_1957__a11d030a1dd5249e4b26c3bdf5ec5cb6d"><strong id="mrs_01_1957__a50dffb4d28424f8a88e9572cc9a63625">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="27.637236276372363%" id="mcps1.3.2.7.2.4.1.3"><p id="mrs_01_1957__aa4008d8cc0154e20bd4bda405861aa89"><strong id="mrs_01_1957__a134959fb978e42ecaa8edc718bf0ab60">Default Log Level</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1957__r1030621bf4bd436b81db6972e78dd20d"><td class="cellrowborder" valign="top" width="33.226677332266775%" headers="mcps1.3.2.7.2.4.1.1 "><p id="mrs_01_1957__a62365381b9f24307babb9dc4952b2aec">GC_OPTS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.13608639136087%" headers="mcps1.3.2.7.2.4.1.2 "><p id="mrs_01_1957__a8c09e069012b44679813c56752d02c8c">Indicates the JVM parameter of the JobHistory Server process.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="27.637236276372363%" headers="mcps1.3.2.7.2.4.1.3 "><p id="mrs_01_1957__abeb841b25d6d4df88a8fb6f98694f6ed">INFO</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1957__r9519070d6b864354b6e42b536e45daf6"><td class="cellrowborder" valign="top" width="33.226677332266775%" headers="mcps1.3.2.7.2.4.1.1 "><p id="mrs_01_1957__a06776d6f341a43339411a098a21e8cff">SPARK_SUBMIT_OPTS</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.13608639136087%" headers="mcps1.3.2.7.2.4.1.2 "><p id="mrs_01_1957__ac2b6c668a9674548b56487e8f9b634f5">Indicates the JVM parameter of JDBCServer.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="27.637236276372363%" headers="mcps1.3.2.7.2.4.1.3 "><p id="mrs_01_1957__ad2a73248ca7b4157ba239d7251e62d2e">INFO</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_1957__a009828936b974022942e40336ab3531f"><strong id="mrs_01_1957__a8102a8c0f0af4cfb895a9b77e90a33b6">Example:</strong></p>
|
|
<p id="mrs_01_1957__a0b941f5377714eb7a51739080d8151b4">To change the log level of the executor process to DEBUG dynamically, modify the <span class="parmname" id="mrs_01_1957__p04f05d5e9ab84b45b5ab63a78a552ac7"><b>spark.executor.extraJavaOptions</b></span> JVM parameter of the executor process in the <span class="filepath" id="mrs_01_1957__fe73f98d4d07742df8625d5fdc2c71928"><b>spark-defaults.conf</b></span> file and run the following command to add the following configuration before the process is started:</p>
|
|
<pre class="screen" id="mrs_01_1957__s2bdf345f7e714cb5aea50ff840343ae3">-Dlog4j.configuration.watch=true </pre>
|
|
<p id="mrs_01_1957__a96ba11f9e94f4a1e9c79978c33fb8a58">After the user application is submitted, change the log level in the log4j configuration file (for example, <strong id="mrs_01_1957__b1939754152015">-Dlog4j.configuration=file:${BIGDATA_HOME}/FusionInsight_Spark2x_<span id="mrs_01_1957__text18397154142018">8.1.0.1</span>/install/FusionInsight-Spark2x-<span id="mrs_01_1957__text5563355171417">3.1.1</span>/spark/conf/log4j-executor.properties</strong>) specified by the <span class="filepath" id="mrs_01_1957__filepath20397641172016"><b>-Dlog4j.configuration</b></span> parameter in <span class="parmname" id="mrs_01_1957__parmname63980418200"><b>spark.executor.extraJavaOptions</b></span> to DEBUG:</p>
|
|
<pre class="screen" id="mrs_01_1957__sb6a3b552150e4d88b71eb58dcc4053c6">log4j.rootCategory=DEBUG, sparklog</pre>
|
|
<p id="mrs_01_1957__a9af44a1900fd46378e1eca07b6da056a">It takes several seconds for the DEBUG level to take effect.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|