:original_name: mrs_01_1961.html

.. _mrs_01_1961:

Configuring Streaming Reading of Driver Execution Results
=========================================================

Scenario
--------

When a query statement is executed, the returned result may be large (containing more than 100,000 records). In this case, JDBCServer out of memory (OOM) may occur. Therefore, the data aggregation function is provided to avoid OOM without sacrificing the performance.

Configuration
-------------

Two data aggregation function configuration parameters are provided. The two parameters are set in the **tunning** option on the Spark JDBCServer server. After the setting is complete, restart JDBCServer.

.. table:: **Table 1** Parameter description

    
   | Parameter                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                            | Default Value         |
    
   | spark.sql.bigdata.thriftServer.useHdfsCollect | Indicates whether to save result data to HDFS instead of the memory.                                                                                                                                                                                                                                                                                                                                                                                   | false                 |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | Advantages: The query result is stored in HDFS. Therefore, JDBCServer OOM does not occur.                                                                                                                                                                                                                                                                                                                                                              |                       |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | Disadvantages: The query is slow.                                                                                                                                                                                                                                                                                                                                                                                                                      |                       |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | -  **true**: Result data is saved to HDFS.                                                                                                                                                                                                                                                                                                                                                                                                             |                       |
   |                                               | -  **false**: This function is disabled.                                                                                                                                                                                                                                                                                                                                                                                                               |                       |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               |    .. important::                                                                                                                                                                                                                                                                                                                                                                                                                                      |                       |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               |       NOTICE:                                                                                                                                                                                                                                                                                                                                                                                                                                          |                       |
   |                                               |       When **spark.sql.bigdata.thriftServer.useHdfsCollect** is set to **true**, result data is saved to HDFS. However, the job description on the native JobHistory page cannot be associated with the corresponding SQL statement. In addition, the execution ID in the spark-beeline command output is null. To solve the JDBCServer OOM problem and ensure correct information display, you are advised to set **spark.sql.userlocalFileCollect**. |                       |
    
   | spark.sql.uselocalFileCollect                 | Indicates whether to save result data to the local disk instead of memory.                                                                                                                                                                                                                                                                                                                                                                             | false                 |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | Advantages: In the case of small data volume, the performance loss can be ignored compared with the data storage mode using the native memory. In the case of large data volume (hundreds of millions of data records), the performance is much better than that when data is stored in the HDFS and native memory.                                                                                                                                    |                       |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | Disadvantages: Optimization is required. In the case of large data volume, it is recommended that the JDBCServer driver memory be 10 GB and each core of the executor be allocated with 3 GB memory.                                                                                                                                                                                                                                                   |                       |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | -  **true**: This function is enabled.                                                                                                                                                                                                                                                                                                                                                                                                                 |                       |
   |                                               | -  **false**: This function is disabled.                                                                                                                                                                                                                                                                                                                                                                                                               |                       |
    
   | spark.sql.collect.Hive                        | This parameter is valid only when **spark.sql.uselocalFileCollect** is set to **true**. It indicates whether to save the result data to a disk in direct serialization mode or in indirect serialization mode.                                                                                                                                                                                                                                         | false                 |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | Advantage: For queries of tables with a large number of partitions, the aggregation performance of the query results is better than that of the storage mode that query results are directly stored on the disk.                                                                                                                                                                                                                                       |                       |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | Disadvantages: The disadvantages are the same as those when **spark.sql.uselocalFileCollect** is enabled.                                                                                                                                                                                                                                                                                                                                              |                       |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | -  **true**: This function is enabled.                                                                                                                                                                                                                                                                                                                                                                                                                 |                       |
   |                                               | -  **false**: This function is disabled.                                                                                                                                                                                                                                                                                                                                                                                                               |                       |
    
   | spark.sql.collect.serialize                   | This parameter takes effect only when both **spark.sql.uselocalFileCollect** and **spark.sql.collect.Hive** are set to **true**.                                                                                                                                                                                                                                                                                                                       | java                  |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | The function is to further improve performance.                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                       |
   |                                               | -  **java**: Data is collected in Java serialization mode.                                                                                                                                                                                                                                                                                                                                                                                             |                       |
   |                                               | -  **kryo**: Data is collected in kryo serialization mode. The performance is better than that when the Java serialization mode is used.                                                                                                                                                                                                                                                                                                               |                       |
    

.. note::

   **spark.sql.bigdata.thriftServer.useHdfsCollect** and **spark.sql.uselocalFileCollect** cannot be set to **true** at the same time.
