ALM-16001 Hive Warehouse Space Usage Exceeds the Threshold

Description

The system checks the Hive warehouse space usage every 30 seconds. The indicator Percentage of HDFS Space Used by Hive to the Available Space can be viewed on the Hive service monitoring page. This alarm is generated when the Hive warehouse space usage exceeds the specified threshold (85% by default).

This alarm is cleared when the Hive warehouse space usage is less than or equal to the threshold. You can reduce the warehouse space usage by expanding the warehouse capacity or releasing the used space.

Attribute

Parameters

Impact on the System

The system fails to write data, which causes data loss.

Possible Causes

• The upper limit of the HDFS capacity available for Hive is too small.
• The system disk space is insufficient.
• Some data nodes break down.

Procedure

1. Expand the system configuration.
   1. Analyze the cluster HDFS capacity usage and increase the upper limit of the HDFS capacity available for Hive.

      Go to the MRS cluster details page, choose Components > Hive > Service Configuration, set Type to All, search for hive.metastore.warehouse.size.percent, and increase the value of this parameter. Suppose that the value of the configuration item is A, total HDFS storage space is B, the threshold is C, and HDFS space used by Hive is D. Adjust the value of the configuration item according to A x B x C > D. The total HDFS storage space can be viewed on the HDFS monitoring page, and HDFS space used by Hive can be viewed on the Hive monitoring page.

      

      For MRS 1.7.2 or earlier, log in to MRS Manager and choose Components > Hive > Service Configuration.

   2. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.a.

2. Expand the system.
   1. Add nodes.
   2. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 3.a.

3. Check whether the data node is normal.
   1. Go to the cluster details page and choose Alarms.
   2. Check whether ALM-12006 Node Fault, ALM-12007 Process Fault, or ALM-14002 DataNode Disk Usage Exceeds the Threshold exists.
      • If yes, go to 3.c.
      • If no, go to 4.
   3. Clear the alarm by following the steps provided in ALM-12006 Node Fault, ALM-12007 Process Fault, or ALM-14002 DataNode Disk Usage Exceeds the Threshold.
   4. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 4.

4. Collect fault information.
   1. On MRS Manager, choose System > Export Log.
   2. Contact technical support engineers for help. For details, see technical support.

Reference

None