Fault Locating
If the pod is in the Pending state and the event contains pod scheduling failure information, locate the cause based on the event information. For details about how to view events, see How Do I Use Events to Fix Abnormal Workloads?
Troubleshooting Process
Determine the cause based on the event information, as listed in Table 1.
Event Information |
Cause and Solution |
|---|---|
no nodes available to schedule pods. |
No node is available in the cluster. |
0/2 nodes are available: 2 Insufficient cpu. 0/2 nodes are available: 2 Insufficient memory. |
Node resources (CPU and memory) are insufficient. Check Item 2: Whether Node Resources (CPU and Memory) Are Sufficient |
0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) didn't match pod affinity rules, 1 node(s) didn't match pod affinity/anti-affinity. |
The node and pod affinity configurations are mutually exclusive. No node meets the pod requirements. Check Item 3: Affinity and Anti-Affinity Configuration of the Workload |
0/2 nodes are available: 2 node(s) had volume node affinity conflict. |
The EVS volume mounted to the pod and the node are not in the same AZ. Check Item 4: Whether the Workload's Volume and Node Reside in the Same AZ |
0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. |
Taints exist on the node, but the pod cannot tolerate these taints. |
0/7 nodes are available: 7 Insufficient ephemeral-storage. |
The ephemeral storage space of the node is insufficient. |
0/1 nodes are available: 1 everest driver not found at node |
The everest-csi-driver on the node is not in the running state. |
Failed to create pod sandbox: ... Create more free space in thin pool or use dm.min_free_space option to change behavior |
The node thin pool space is insufficient. |
0/1 nodes are available: 1 Too many pods. |
The number of pods scheduled to the node exceeded the maximum number allowed by the node. |
Check Item 1: Whether a Node Is Available in the Cluster
Log in to the CCE console and check whether the node status is Available. Alternatively, run the following command to check whether the node status is Ready:
$ kubectl get node NAME STATUS ROLES AGE VERSION 192.168.0.37 Ready <none> 21d v1.19.10-r1.0.0-source-121-gb9675686c54267 192.168.0.71 Ready <none> 21d v1.19.10-r1.0.0-source-121-gb9675686c54267
If the status of all nodes is Not Ready, no node is available in the cluster.
Solution
- Add a node. If an affinity policy is not configured for the workload, the pod will be automatically migrated to the new node to ensure that services are running properly.
- Locate the unavailable node and rectify the fault. For details, see What Should I Do If a Cluster Is Available But Some Nodes Are Unavailable?
- Reset the unavailable node.
Check Item 2: Whether Node Resources (CPU and Memory) Are Sufficient
0/2 nodes are available: 2 Insufficient cpu. This means insufficient CPUs.
0/2 nodes are available: 2 Insufficient memory. This means insufficient memory.
If the resources requested by the pod exceed the allocatable resources of the node where the pod runs, the node cannot provide the resources required to run new pods and pod scheduling onto the node will definitely fail.
If the number of resources that can be allocated to a node is less than the number of resources that a pod requests, the node does not meet the resource requirements of the pod. As a result, the scheduling fails.
Solution
Add nodes to the cluster. Scale-out is the common solution to insufficient resources.
Check Item 3: Affinity and Anti-Affinity Configuration of the Workload
Inappropriate affinity policies will cause pod scheduling to fail.
Example:
An anti-affinity relationship is established between workload 1 and workload 2. Workload 1 is deployed on node 1 while workload 2 is deployed on node 2.
When you try to deploy workload 3 on node 1 and establish an affinity relationship with workload 2, a conflict occurs, resulting in a workload deployment failure.
0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) didn't match pod affinity rules, 1 node(s) didn't match pod affinity/anti-affinity.
- node selector indicates that the node affinity is not met.
- pod affinity rules indicate that the pod affinity is not met.
- pod affinity/anti-affinity indicates that the pod affinity/anti-affinity is not met.
Solution
- When adding workload-workload affinity and workload-node affinity policies, ensure that the two types of policies do not conflict each other. Otherwise, workload deployment will fail.
- If the workload has a node affinity policy, make sure that supportContainer in the label of the affinity node is set to true. Otherwise, pods cannot be scheduled onto the affinity node and the following event is generated:
No nodes are available that match all of the following predicates: MatchNode Selector, NodeNotSupportsContainer
If the value is false, the scheduling fails.
Check Item 4: Whether the Workload's Volume and Node Reside in the Same AZ
0/2 nodes are available: 2 node(s) had volume node affinity conflict. An affinity conflict occurs between volumes and nodes. As a result, the scheduling fails.
This is because EVS disks cannot be attached to nodes across AZs. For example, if the EVS volume is located in AZ 1 and the node is located in AZ 2, scheduling fails.
The EVS volume created on CCE has affinity settings by default, as shown below.
kind: PersistentVolume
apiVersion: v1
metadata:
name: pvc-c29bfac7-efa3-40e6-b8d6-229d8a5372ac
spec:
...
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
-
Solution
In the AZ where the workload's node resides, create a volume. Alternatively, create an identical workload and select an automatically assigned cloud storage volume.
Check Item 5: Taint Toleration of Pods
0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. This means the node is tainted and the pod cannot be scheduled to the node.
Check the taints on the node. If the following information is displayed, taints exist on the node:
$ kubectl describe node 192.168.0.37 Name: 192.168.0.37 ... Taints: key1=value1:NoSchedule ...
In some cases, the system automatically adds a taint to a node. The current built-in taints include:
- node.kubernetes.io/not-ready: The node is not ready.
- node.kubernetes.io/unreachable: The node controller cannot access the node.
- node.kubernetes.io/memory-pressure: The node has memory pressure.
- node.kubernetes.io/disk-pressure: The node has disk pressure. Follow the instructions described in Check Item 4: Whether the Node Disk Space Is Insufficient to handle it.
- node.kubernetes.io/pid-pressure: The node is under PID pressure.
- node.kubernetes.io/network-unavailable: The node network is unavailable.
- node.kubernetes.io/unschedulable: The node cannot be scheduled.
- node.cloudprovider.kubernetes.io/uninitialized: If an external cloud platform driver is specified when kubelet is started, kubelet adds a taint to the current node and marks it as unavailable. After cloud-controller-manager initializes the node, kubelet deletes the taint.
Solution
To schedule the pod to the node, use either of the following methods:
- If the taint is added by a user, you can delete the taint on the node. If the taint is automatically added by the system, the taint will be automatically deleted after the fault is rectified.
- Specify a toleration for the pod containing the taint. For details, see Taints and Tolerations.
apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: nginx:alpine tolerations: - key: "key1" operator: "Equal" value: "value1" effect: "NoSchedule"
Check Item 6: Ephemeral Volume Usage
0/7 nodes are available: 7 Insufficient ephemeral-storage. This means insufficient ephemeral storage of the node.
Check whether the size of the ephemeral volume in the pod is limited. If the size of the ephemeral volume required by the application exceeds the existing capacity of the node, the application cannot be scheduled. To solve this problem, change the size of the ephemeral volume or expand the disk capacity of the node.
apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: app
image: images.my-company.example/app:v4
resources:
requests:
ephemeral-storage: "2Gi"
limits:
ephemeral-storage: "4Gi"
volumeMounts:
- name: ephemeral
mountPath: "/tmp"
volumes:
- name: ephemeral
emptyDir: {}
To obtain the total capacity (Capacity) and available capacity (Allocatable) of the temporary volume mounted to the node, run the kubectl describe node command, and view the application value and limit value of the temporary volume mounted to the node.
The following is an example of the output:
... Capacity: cpu: 4 ephemeral-storage: 61607776Ki hugepages-1Gi: 0 hugepages-2Mi: 0 localssd: 0 localvolume: 0 memory: 7614352Ki pods: 40 Allocatable: cpu: 3920m ephemeral-storage: 56777726268 hugepages-1Gi: 0 hugepages-2Mi: 0 localssd: 0 localvolume: 0 memory: 6180752Ki pods: 40 ... Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 1605m (40%) 6530m (166%) memory 2625Mi (43%) 5612Mi (92%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) localssd 0 0 localvolume 0 0 Events: <none>
Check Item 7: Whether everest Works Properly
0/1 nodes are available: 1 everest driver not found at node. This means the everest-csi-driver of everest is not started properly on the node.
Check the daemon named everest-csi-driver in the kube-system namespace and check whether the pod is started properly. If not, delete the pod. The daemon will restart the pod.
Check Item 8: Thin Pool Space
A data disk dedicated for kubelet and the container engine will be attached to a new node. If the data disk space is insufficient, the pod cannot be created.
Solution 1: Clearing images
- Nodes that use containerd
- Obtain local images on the node.
crictl images -v
- Delete the images that are not required by image ID.
crictl rmi Image ID
- Obtain local images on the node.
- Nodes that use Docker
- Obtain local images on the node.
docker images
- Delete the images that are not required by image ID.
docker rmi Image ID
- Obtain local images on the node.
Do not delete system images such as the cce-pause image. Otherwise, pods may fail to be created.
Solution 2: Expanding the disk capacity
To expand a disk capacity, perform the following steps:
- Expand the capacity of a data disk on the EVS console.
Only the storage capacity of the EVS disk is expanded. You also need to perform the following steps to expand the capacity of the logical volume and file system.
- Log in to the CCE console and click the cluster. In the navigation pane, choose Nodes. Click More > Sync Server Data in the row containing the target node.
- Log in to the target node.
- Run the lsblk command to check the block device information of the node.
A data disk is divided depending on the container storage Rootfs:
Overlayfs: No independent thin pool is allocated. Image data is stored in dockersys.
- Check the disk and partition sizes of the device.
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 8:0 0 50G 0 disk └─vda1 8:1 0 50G 0 part / vdb 8:16 0 200G 0 disk # Data disk has been expanded but not allocated ├─vgpaas-dockersys 253:0 0 90G 0 lvm /var/lib/containerd # Space used by the container engine └─vgpaas-kubernetes 253:1 0 10G 0 lvm /mnt/paas/kubernetes/kubelet # Space used by Kubernetes
- Expand the disk capacity.
Add the new disk capacity to the dockersys logical volume used by the container engine.
- Expand the PV capacity so that LVM can identify the new EVS capacity. /dev/vdb specifies the physical volume where dockersys is located.
pvresize /dev/vdbInformation similar to the following is displayed:
Physical volume "/dev/vdb" changed 1 physical volume(s) resized or updated / 0 physical volume(s) not resized
- Expand 100% of the free capacity to the logical volume. vgpaas/dockersys specifies the logical volume used by the container engine.
lvextend -l+100%FREE -n vgpaas/dockersysInformation similar to the following is displayed:
Size of logical volume vgpaas/dockersys changed from <90.00 GiB (23039 extents) to <190.00 GiB (48639 extents). Logical volume vgpaas/dockersys successfully resized.
- Adjust the size of the file system. /dev/vgpaas/dockersys specifies the file system path of the container engine.
resize2fs /dev/vgpaas/dockersysInformation similar to the following is displayed:
Filesystem at /dev/vgpaas/dockersys is mounted on /var/lib/containerd; on-line resizing required old_desc_blocks = 12, new_desc_blocks = 24 The filesystem on /dev/vgpaas/dockersys is now 49807360 (4k) blocks long.
- Expand the PV capacity so that LVM can identify the new EVS capacity. /dev/vdb specifies the physical volume where dockersys is located.
- Check the disk and partition sizes of the device.
Check Item 9: Number of Pods Scheduled onto the Node
0/1 nodes are available: 1 Too many pods. indicates excessive number of pods have been scheduled to the node.
When creating a node, configure Max. Pods in Advanced Settings to specify the maximum number of pods that can run properly on the node. The default value varies with the node flavor. You can change the value as needed.
On the Nodes page, obtain the Pods (Allocated/Total) value of the node, and check whether the number of pods scheduled onto the node has reached the upper limit. If so, add nodes or change the maximum number of pods.
To change the maximum number of pods that can run on a node, do as follows:
- For nodes in the default node pool: Change the Max. Pods value when resetting the node.
- For nodes in a customized node pool: Change the value of the node pool parameter max-pods.