Description

The system checks the ZooKeeper service status every 30 seconds. The alarm is generated when the memory usage of a ZooKeeper instance exceeds the threshold (80% of the maximum memory).

The alarm is cleared when the memory usage is less than the threshold.

Attribute

Alarm ID	Alarm Severity	Auto Clear
13002	Major	Yes

Parameters

Parameter	Description
ServiceName	Specifies the service for which the alarm is generated.
RoleName	Specifies the role for which the alarm is generated.
HostName	Specifies the host for which the alarm is generated.
Trigger Condition	Generates an alarm when the actual indicator value exceeds the specified threshold.

Impact on the System

If the available ZooKeeper memory is insufficient, a memory overflow occurs and the service breaks down.

Possible Causes

The memory usage of the ZooKeeper instance is overused or the memory is inappropriately allocated.

Procedure

Check the memory usage.
1. On the MRS cluster details page, choose Alarms > ALM-13002 ZooKeeper Memory Usage Exceeds the Threshold > Location. Check the IP address of the instance for which the alarm is generated.
2. On the MRS cluster details page, choose Components > ZooKeeper > Instances > quorumpeer (IP address of the instance for which the alarm is generated) > Customize > ZooKeeper Heap And Direct Buffer Resource. Check the heap memory usage.
  
  For MRS 1.7.2 or earlier, log in to MRS Manager and choose Services > ZooKeeper > Instance > quorumpeer (IP address of the instance for which the alarm is generated) > Customize > ZooKeeper Heap And Direct Buffer Resource. Check the heap memory usage.
3. Check whether the used heap memory of ZooKeeper reaches 80% of the maximum heap memory specified for ZooKeeper.
  - If yes, go to 1.d.
  - If no, go to 1.f.
4. On MRS Manager, choose Services > ZooKeeper > Configuration > All > quorumpeer > System. Increase the value of -Xmx in GC_OPTS as required.
5. Check whether the alarm is cleared.
  - If yes, no further action is required.
  - If no, go to 1.f.
6. On the MRS cluster details page, choose Components > ZooKeeper > Instances > quorumpeer (IP address of the instance for which the alarm is generated) > Customize > ZooKeeper Heap And Direct Buffer Resource. Check the direct buffer memory usage.
  
  For MRS 1.7.2 or earlier, log in to MRS Manager and choose Services > ZooKeeper > Instance > quorumpeer (IP address of the instance for which the alarm is generated) > Customize > ZooKeeper Heap And Direct Buffer Resource. Check the direct buffer memory usage.
7. Check whether the used direct buffer memory of ZooKeeper reaches 80% of the maximum direct buffer memory specified for ZooKeeper.
  - If yes, go to 1.h.
  - If no, go to 2.
8. On the MRS cluster details page, choose Components > ZooKeeper > Service Configuration. Set Type to All and choose quorumpeer > System.
  Increase the value of -XX:MaxDirectMemorySize in GC_OPTS as required.
  
  For MRS 1.7.2 or earlier, log in to MRS Manager, choose Services > ZooKeeper > Service Configuration. Set Type to All and choose quorumpeer > System.
9. Check whether the alarm is cleared.
  - If yes, no further action is required.
  - If no, go to 2.
Collect fault information.
1. On MRS Manager, choose System > Export Log.
2. Contact technical support engineers for help. For details, see technical support.

Reference

None