How Do I Troubleshoot an Unhealthy Backend Server?

Symptom

If a client cannot access a backend server through a load balancer, the backend server is declared unhealthy. You can check the health check result of the backend server on the ELB console.

Background

The load balancer uses IP addresses in 100.125.0.0/16 to send heartbeats to backend servers and check their health. To ensure that health checks can be performed normally, IP addresses in 100.125.0.0/16 must be allowed to access the backend servers.

Security group rules configured for backend servers associated with dedicated load balancers are different from those configured for backend servers associated with shared load balancers.

If a backend server is considered unhealthy, ELB will not route traffic to it until it is declared healthy again.

If you change the weight of a healthy backend server to 0, the health check result of this server becomes Unhealthy.

Troubleshooting Procedure

Possible causes are sequenced based on their occurrence probability.

Check these causes one by one until the fault persists.

You may need to change the health check configuration. It takes a while for the modification to take effect. The required time depends on health check interval and timeout duration. View the health check result in the backend server list of the load balancer.

Figure 1 Troubleshooting process
Table 1 Troubleshooting process

Possible Cause

Solution

Backend server group

Checking Whether the Backend Server Group Is Associated with a Listener

Health check configuration

Checking the Health Check Configuration

Security group rules

Checking Security Group Rules

Network ACL rules

Checking Firewall Rules

Backend server listening configuration

Checking the Backend Server

Backend server firewall configuration

Checking the Firewall on the Backend Server

Backend server route configuration

Checking the Backend Server Route

Backend server load

Checking the Backend Server Load

Backend server host.deny file

Checking the host.deny File

Checking Whether the Backend Server Group Is Associated with a Listener

Check whether the backend server group that the unhealthy backend server belongs to is associated with a listener.

Checking the Health Check Configuration

Classic load balancers: In the Listeners area, locate the listener with an unhealthy backend server associated and click View in the Health Check column. The Health Check dialog box is displayed. Check the following parameters:

Shared load balancers: Click the name of the load balancer to view its details. Click Backend Server Groups and then click the name of the server group. On the Basic Information page, click Configure on the right of Health Check. Check the following parameters:

Checking Security Group Rules

Checking Firewall Rules

You can associate one or more subnets with a firewall for controlling traffic in and out of the subnets. Similar to security groups, firewalls provide access control functions, but add an additional layer of defense to your VPC. Default firewall rules reject all inbound and outbound traffic. If the subnet of a load balancer or associated backend servers has a firewall associated, the load balancer cannot receive traffic from the Internet or route traffic to backend servers, and backend servers cannot receive traffic from and respond to the load balancer.

You can configure an inbound firewall rule to permit access from 100.125.0.0/16.

  1. Log in to the management console.
  2. In the upper left corner of the page, click and select the desired region and project.
  3. Under Network, click Virtual Private Cloud.
  4. In the navigation pane on the left, choose Access Control > Firewalls.
  5. Locate the firewall, and click the firewall name to switch to the firewall details page.
  6. On the Inbound Rules or Outbound Rules tab page, click Add Rule to add a rule.
  7. Click OK.

Checking the Backend Server

If the backend server runs a Windows OS, use a browser to access https://Backend server IP address:Health check port. If a 2xx or 3xx code is returned, the backend server is running normally.

Checking the Firewall on the Backend Server

The firewall or other security software on the backend server may mask IP addresses in 100.125.0.0/16. Ensure that access from 100.125.0.0/16 is allowed in the security group containing the backend server.

Checking the Backend Server Route

Check whether the default route configured for the primary NIC has been manually modified. If the default route is changed, health check packets may fail to reach the backend server.

Run the following command on the backend server to check whether the default route points to the gateway (For Layer 3 communications, the default route must be configured to point to the gateway):
ip route

Alternatively, run the following command:

route -n

If the command output does not contain the highlighted route or the IP address to which the route points is not the gateway address of the VPC subnet, change the route to the default one.

Figure 12 Example default route pointing to the gateway
Figure 13 Example default route not pointing to the gateway

Checking the Backend Server Load

Check the load of the backend server. If the load is high, connections or requests for health checks may time out.

Checking the host.deny File

Verify that IP addresses in 100.125.0.0/16 are not written to the /etc/hosts.deny file on the backend server.

Parent topic: Health Check