:original_name: mrs_01_0844.html

.. _mrs_01_0844:

Optimization Configuration for Multiple CPU Cores
=================================================

Scenario
--------

Optimization can be performed when the number of CPU cores is large, for example, the number of CPU cores is three times the number of disks.

Procedure
---------

You can set the following parameters in either of the following ways:

-  Configuration on the server:

   On the **All Configurations** page of the Yarn service, enter a parameter name in the search box. For details, see :ref:`Modifying Cluster Service Configuration Parameters <mrs_01_2125>`.

-  Configuration on the client:

   Modify the corresponding configuration file on the client.

   .. note::

      -  Path of configuration files on the HDFS client: *Client installation directory*\ **/HDFS/hadoop/etc/hadoop/hdfs-site.xml**

      -  Path of configuration files on the Yarn client: *Client installation directory*\ **/HDFS/hadoop/etc/hadoop/yarn-site.xml**.

      -  Path of configuration files on the MapReduce client: *Client installation directory*\ **/HDFS/hadoop/etc/hadoop/mapred-site.xml**.

.. table:: **Table 1** Settings of multiple CPU cores

    
   | Configuration                       | Description                                                                                                                                                                                                                                                    | Parameter                                                                                                                                                           | Default Value                          | Server/Client | Impact                                                                                                                                                                                                                                                                                                            | Remarks                                                                                               |
    
   | Number of slots in a node container | The combination of the following parameters determines the number of concurrent tasks (Map and Reduce tasks) of each node:                                                                                                                                     | yarn.nodemanager.resource.memory-mb                                                                                                                                 | Versions earlier than MRS 3.\ *x*:     | Server        | If data needs to be read from and written into disks for all tasks (Map/Reduce tasks), a disk may be accessed by multiple processes at the same time, which leads to poor disk I/O performance. To ensure disk I/O performance, the number of concurrent access requests from a client to a disk cannot exceed 3. | The maximum number of concurrent containers must be [2.5 x Number of disks configured in Hadoop].     |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     | -  yarn.nodemanager.resource.memory-mb                                                                                                                                                                                                                         | .. note::                                                                                                                                                           | 8192                                   |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     | -  mapreduce.map.memory.mb                                                                                                                                                                                                                                     |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     | -  mapreduce.reduce.memory.mb                                                                                                                                                                                                                                  |    For versions earlier than MRS 3.x: You need to configure this parameter on the MRS console.                                                                      | MRS 3.\ *x* or later:                  |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |    For MRS 3.\ *x* or later: You need to configure this parameter on FusionInsight Manager.                                                                         | 16384                                  |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
    
   |                                     |                                                                                                                                                                                                                                                                | mapreduce.map.memory.mb                                                                                                                                             | 4096                                   | Client        |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                | .. note::                                                                                                                                                           |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |    You need to set this parameter in the configuration file on the client in the *Client installation directory*\ **/HDFS/hadoop/etc/hadoop/mapred-site.xml** path. |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
    
   |                                     |                                                                                                                                                                                                                                                                | mapreduce.reduce.memory.mb                                                                                                                                          | 4096                                   | Client        |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                | .. note::                                                                                                                                                           |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |    You need to set this parameter in the configuration file on the client in the *Client installation directory*\ **/HDFS/hadoop/etc/hadoop/mapred-site.xml** path. |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
    
   | Map output and compression          | The Map task output before being written into disks can be compressed. This can save disk space, offer faster data write, and reduce the data traffic delivered to Reducer. You need to configure the following parameters on the client:                      | mapreduce.map.output.compress                                                                                                                                       | true                                   | Client        | The disk I/O is the bottleneck. Therefore, use a compression algorithm with a high compression rate.                                                                                                                                                                                                              | Snappy is used. The benchmark test results show that Snappy delivers high performance and efficiency. |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     | -  **mapreduce.map.output.compress**: The Map task output can be compressed before it is transmitted over the network. It is a per-job configuration.                                                                                                          | .. note::                                                                                                                                                           |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     | -  **mapreduce.map.output.compress.codec**: the codec used for data compression                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |    You need to set this parameter in the configuration file on the client in the *Client installation directory*\ **/HDFS/hadoop/etc/hadoop/mapred-site.xml** path. |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
    
   |                                     |                                                                                                                                                                                                                                                                | mapreduce.map.output.compress.codec                                                                                                                                 | org.apache.hadoop.io.compress.Lz4Codec | Client        |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                | .. note::                                                                                                                                                           |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |    You need to set this parameter in the configuration file on the client in the *Client installation directory*\ **/HDFS/hadoop/etc/hadoop/mapred-site.xml** path. |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
    
   | Spills                              | mapreduce.map.sort.spill.percent                                                                                                                                                                                                                               | mapreduce.map.sort.spill.percent                                                                                                                                    | 0.8                                    | Client        | Disk I/Os are the bottleneck. You can set the value of **mapreduce.task.io.sort.mb** to minimize the memory spilled to the disk.                                                                                                                                                                                  | ``-``                                                                                                 |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                | .. note::                                                                                                                                                           |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |    You need to set this parameter in the configuration file on the client in the *Client installation directory*\ **/HDFS/hadoop/etc/hadoop/mapred-site.xml** path. |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
    
   | Data packet size                    | When the HDFS client writes data to a data node, the data will be accumulated until a packet is generated. Then, the packet is transmitted over the network. **dfs.client-write-packet-size** specifies the data packet size. It can be specified by each job. | dfs.client-write-packet-size                                                                                                                                        | 262144                                 | Client        | The data node receives data packets from the HDFS client and writes data into disks through single threads. When disks are in the concurrent write state, increasing the data packet size can reduce the disk seek time and improve the I/O performance.                                                          | dfs.client-write-packet-size = 262144                                                                 |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                | .. note::                                                                                                                                                           |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |                                                                                                                                                                     |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
   |                                     |                                                                                                                                                                                                                                                                |    You need to set this parameter in the configuration file on the client in the *Client installation directory*\ **/HDFS/hadoop/etc/hadoop/hdfs-site.xml/** path.  |                                        |               |                                                                                                                                                                                                                                                                                                                   |                                                                                                       |
    
