# What Should I Do If the OOM Killer Is Triggered When a Container Uses Memory Resources More Than Limited?

If a node has sufficient memory resources, a container on this node can use more memory resources than requested, but no more than limited. If the memory allocated to a container exceeds the upper limit, the container is stopped first. If the container continuously uses memory resources more than limited, the container is terminated. If a stopped container is allowed to be restarted, kubelet will restart it, but other types of run errors will occur.

#### Scenario 1

The node's memory has reached the memory limit reserved for the node. As a result, OOM killer is triggered.

Solution

You can either scale up the node or migrate the pods on the node to other nodes.

#### Scenario 2

The upper limit of resources configured for the pod is too small. When the actual usage exceeds the limit, OOM killer is triggered.

Solution

Set a higher upper limit for the workload.

#### Example

A pod will be created and allocated memory that exceeds the limit. As shown in the following configuration file of the pod, the pod requests 50 MB memory and the memory limit is set to 100 MB.

Example YAML file (memory-request-limit-2.yaml):

    apiVersion: v1
    kind: Pod
    metadata:
      name: memory-demo-2
    spec:
      containers:
      - name: memory-demo-2-ctr
        image: vish/stress
        resources:
          requests:
            memory: 50Mi
          limits:
            memory: "100Mi"
        args:
        - -mem-total
        - 250Mi
        - -mem-alloc-size
        - 10Mi
        - -mem-alloc-sleep
        - 1s

The args parameters indicate that the container attempts to allocate 250 MB memory, which exceeds the pod's upper limit (100 MB).

Creating a pod:

    kubectl create -f https://k8s.io/docs/tasks/configure-pod-container/memory-request-limit-2.yaml --namespace=mem-example

Viewing the details about the pod:

    kubectl get pod memory-demo-2 --namespace=mem-example

In this stage, the container may be running or be killed. If the container is not killed, repeat the previous command until the container is killed.

    NAME            READY     STATUS      RESTARTS   AGE
    memory-demo-2   0/1       OOMKilled   1          24s

Viewing detailed information about the container:

    kubectl get pod memory-demo-2 --output=yaml --namespace=mem-example

This output indicates that the container is killed because the memory limit is exceeded.

    lastState:
       terminated:
         containerID: docker://7aae52677a4542917c23b10fb56fcb2434c2e8427bc956065183c1879cc0dbd2
         exitCode: 137
         finishedAt: 2020-02-20T17:35:12Z
         reason: OOMKilled
         startedAt: null

In this example, the container can be automatically restarted. Therefore, kubelet will start it again. You can run the following command several times to see how the container is killed and started:

    kubectl get pod memory-demo-2 --namespace=mem-example

The preceding command output indicates how the container is killed and started back and forth:

    stevepe@sperry-1:~/steveperry-53.github.io$ kubectl get pod memory-demo-2 --namespace=mem-example
    NAME            READY     STATUS      RESTARTS   AGE
    memory-demo-2   0/1       OOMKilled   1          37s
    stevepe@sperry-1:~/steveperry-53.github.io$ kubectl get pod memory-demo-2 --namespace=mem-example
    NAME            READY     STATUS    RESTARTS   AGE
    memory-demo-2   1/1       Running   2          40s

Viewing the historical information of the pod:

    kubectl describe pod memory-demo-2 --namespace=mem-example

The following command output indicates that the pod is repeatedly killed and started.

    ... Normal  Created   Created container with id 66a3a20aa7980e61be4922780bf9d24d1a1d8b7395c09861225b0eba1b1f8511
    ... Warning BackOff   Back-off restarting failed container

Parent topic: Workload Abnormalities