When a Docker container stops with exit code 137 or keeps restarting, the Linux kernel has likely killed a process inside it. Docker lets you set a memory limit for each container. If the container’s processes need more memory than that limit allows and the kernel cannot free enough, the kernel kills one of them. This is an out-of-memory kill, or OOM kill.
The kernel enforces the limit, so its records show what happened. Docker’s state fields confirm an OOM kill, while kernel logs and counters show whether the container hit its own limit or the host ran out of memory. A breakdown of the container’s memory shows whether a process keeps growing or the file cache is filling up. Once you know which case applies, you can choose the fix: raise the limit, fix a memory leak, reduce the number of concurrent workers, or fix a shutdown problem that looks like an OOM kill. The diagnostic commands inspect a Linux host running Docker. The docker run example creates a new container and sets its limit. Where the commands depend on the cgroup version, that version is stated.
How Docker sets the memory cap
Docker does not enforce the limit itself. It writes the limit into a control group, or cgroup, a Linux kernel feature that groups processes and applies resource rules to each group. Each container runs in its own cgroup, and the kernel enforces the limit on that group.
The main flag is --memory, or -m, which sets a hard limit. A container started with a limit can use no more memory than that amount in RAM. The resource constraints page also describes --memory-reservation, a soft limit that must be lower than --memory. Docker applies the soft limit when the host runs short of memory. It does not guarantee the container stays under the hard limit.
Swap is disk space the kernel uses to hold memory pages that do not fit in RAM. It has its own rules. The docker run reference lists the combinations. If neither --memory nor --memory-swap is set, the container has no memory limit. If --memory is set and --memory-swap is -1, the container cannot use more than the memory limit in RAM, but it can use as much swap as the host allows. A third-party guide describes --memory-swap as the total of RAM plus swap. Follow the official combinations instead.
To set a hard limit on a new container, pass the flag at start:
docker run -d --name <name> --memory <limit> <image>
The <limit> value is a size such as a number of megabytes with a suffix. Choose it based on measured memory use, not a guess.
What counts against the memory limit
The memory of one process is usually measured as its resident set size, or RSS. RSS is the part of the process’s memory currently held in physical RAM. A process also has memory with no file behind it. This is called anonymous memory, and it includes the heap (the memory a program allocates while it runs) and each thread’s stack. To free space, the kernel must first write anonymous memory to swap. It cannot simply drop it.
The kernel also keeps recently read file contents in RAM. This is the page cache. When another program needs memory, the kernel can drop cached file pages, because it can read them from disk again. The cgroup v2 documentation describes the limit as memory.max, and the kernel charges both anonymous memory and page cache to the container’s cgroup. In practice, reading files alone can push a container’s usage toward its limit.
The Netdata memory guide states that the OOM killer acts when charges that cannot be reclaimed exceed the limit, not when page cache is high. In practice, a container full of cached files is not in danger of an OOM kill by itself. A container whose anonymous memory keeps growing is.
Confirming an OOM kill from Docker’s state
Exit code 137 means the process received SIGKILL, the signal that stops a process immediately and cannot be caught or handled. Unix reports a process ended by a signal as 128 plus the signal number. SIGKILL is signal 9, so 128 plus 9 gives 137. The exit code alone does not show who sent the signal, since any sender of SIGKILL produces 137.
Docker records one more field: OOMKilled. Netdata’s exit 137 guide says Docker sets this field to true when the kernel’s OOM killer ends the container’s main process, which is PID 1 in the container. Other guides describe the flag more broadly, as a sign that the container exceeded its cgroup memory limit. Read OOMKilled as a strong signal of a container-limit kill, and expect a false value when a child process was the one killed.
Check the state with these commands:
docker inspect --format '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}' <container>
docker inspect --format '{{.HostConfig.Memory}}' <container>
docker events --filter event=oom --since 1h
The first command prints the exit code, the OOM flag, and any error message the daemon stored. The second prints the hard limit in bytes. A value of 0 means no per-container hard limit is set, so the container could grow until the host runs out of memory. The third command lists OOM events from the last hour. The --since window limits the search, so widen it if the crash happened earlier.
Restarts give another clue. Netdata’s diagnosis guide describes exit 137 with a rising restart count as the pattern of an OOM crash loop, where the container restarts, grows back to its limit, and gets killed again.
Telling a container limit from a host or external kill
Exit 137 with OOMKilled set to true points to the container’s own limit. Exit 137 with OOMKilled set to false points elsewhere. The table below summarizes the common combinations, based on the Netdata guide and the exit 137 article on DEV.
| Exit code | OOMKilled | Usual cause |
|---|---|---|
| 137 | true | The container exceeded its own memory limit |
| 137 | false, no container limit | The host ran out of memory, and the kernel killed the container’s process |
| 137 | false | Something else sent SIGKILL, such as a docker kill or a stop that ran past its timeout |
| 137 | false, after docker stop |
The stop command sent SIGTERM, the polite stop signal, and the process did not exit in time, so Docker sent SIGKILL |
The kernel log gives a second check. A container-limit kill writes a line that starts with "Memory cgroup out of memory: Killed process." A host-wide kill writes "Out of memory: Killed process." The DEV article says the kernel’s task_memcg= field usually holds the container’s cgroup path, such as /system.slice/docker-<id>.scope, so searching for the first 12 characters of the container ID finds the matching lines. Read the logs with these commands:
sudo dmesg -T | grep -iE 'oom|killed process'
journalctl -k --grep="oom"
The 2018 kernel patch that defined these message names, linked here as the OOM report patch, is older than many current kernels. Use the wording as a guide and check the actual lines on your host.
On Docker Desktop for macOS or Windows, containers run inside a Linux virtual machine. The kernel log you read belongs to that virtual machine. The virtual machine has its own memory limit, and that limit sits below the container limit. A host-wide kill there means the virtual machine ran out, not the laptop.
Other senders also produce SIGKILL, and from the outside they can look like an OOM kill. Running docker kill or docker rm -f produces a kill event with signal 9. A docker stop produces a kill event with signal 15, then a signal 9 if the process ignores SIGTERM. The DEV article gives 10 seconds as the default grace period before Docker escalates. Someone running kill -9 on the host produces only a die event. User-space tools such as systemd-oomd and earlyoom kill processes before the kernel does. They record their actions in the journal, not the kernel log.
The cause decides the fix. Raising a memory limit solves a container-limit OOM. It does nothing for a shutdown that runs past its timeout, and a host-wide OOM needs more host memory or lower limits on the other containers. The older Docker resource constraints page on GitHub says the Docker daemon gets a lower OOM priority than containers, so the kernel tends to pick a container before the daemon. Containers receive no such protection, so the container is the usual victim.
Reading kernel counters inside the container
Cgroup v2 keeps per-group counters in files under /sys/fs/cgroup. The most useful one for OOM work is memory.events. It has three counters that matter here. The max counter records how often usage reached the limit. The oom counter records how many OOM events happened in the group. The oom_kill counter records how many processes the kernel killed. A Netdata academy article and kernel internals notes describe these counters, and the wording differs between them. Read the file in a running container:
docker exec <container> cat /sys/fs/cgroup/memory.events
The counter file belongs to the running container’s cgroup. A container that has already exited has no counters to read, and a restarted container starts a new cgroup, so capture the file while the problem is happening.
The cgroup v2 documentation describes three limit files. memory.max holds the hard limit. memory.high is a throttle. Processes above it slow down under reclaim pressure, and the kernel does not invoke the OOM killer for going over it, though extreme conditions can exceed it. memory.min protects a minimum amount of memory from reclaim. A PandaStack explainer gives a triage reading of the counters. If max is above zero and oom_kill is zero, the kernel is still reclaiming memory successfully. If oom_kill is above zero, the kernel killed a process inside the container. That reading comes from a secondary source, so treat it as a starting point and not a rule.
Check the cgroup version first, since the file paths depend on that version:
docker info --format 'cgroup={{.CgroupVersion}} driver={{.CgroupDriver}}'
On cgroup v1 hosts, the matching values are in memory.limit_in_bytes for the limit, memory.failcnt for the number of times the limit was hit, and memory.oom_control for OOM settings. The Netdata guide gives the path pattern /sys/fs/cgroup/memory/docker/<id>/ for these files.
Why docker stats numbers mislead
The docker container stats reference shows memory usage and the memory limit for each running container. Usage near the limit looks like an OOM risk, but the number can differ from what the kernel charges against the limit. The Netdata memory guide states that docker stats subtracts inactive file cache from the usage it shows. On cgroup v1, the calculation is memory.usage_in_bytes minus total_inactive_file. On cgroup v2, it is memory.current minus inactive_file. A ctop issue comment reports that the Docker API returns total usage and a cache value, and that the Docker CLI performs the subtraction. The official metrics page was available only as an excerpt, so the exact calculation comes from these secondary sources.
The gap between the two numbers can make cache growth look like a leak. A Middleware example describes a container whose usage rose from 340 MiB to 410 MiB against a 512 MiB limit, with no change in traffic. The Alex Oglou write-up describes a test in which memory grew steadily but was reclaimed from file cache, and no OOM occurred. A single reading cannot tell these cases apart. Sample usage over time and compare it with the anonymous and file breakdown described in the next section.
The Netdata diagnosis guide suggests treating a docker stats memory percentage above a sustained 85 percent as a warning. That threshold is the guide’s own heuristic, not a Docker setting, so use it as a prompt to look closer rather than as a pass or fail test.
Separating leaks from cache growth
To tell a leak from cache, read the memory breakdown in memory.stat inside the container:
docker exec <container> grep -E '^(anon|file|slab) ' /sys/fs/cgroup/memory.stat
The anon line holds anonymous memory and the file line holds page cache. The slab line holds kernel memory for caches of kernel objects. If anon grows steadily across repeated samples, the program probably holds on to memory it no longer needs, which is the leak pattern. If file dominates and anon stays flat, the container is mostly caching files, and the kernel can reclaim that memory. The Netdata diagnosis guide lists these patterns, along with a case in which a container is killed while its Java heap is below the limit. For Java, the same guide suggests setting the container limit 20 to 30 percent above the intended maximum heap and setting -Xmx to 80 percent of the limit. These are rules of thumb from that guide, not Docker or Java requirements, and the memory outside the heap, including thread stacks and native libraries, needs room too.
Each worker process or thread holds its own working memory. More concurrent workers therefore raise the container’s peak usage. The sources do not give a per-worker figure, so measure the peak with the same counters while the load runs at the level you plan to use. Lowering the worker count is often a cheaper fix than raising the limit, but test the throughput impact before you choose it.
Summing the usage of every container does not give the host’s total memory. The Guru-host explanation notes that the daemon and kernel memory sit outside the containers, so the sum undercounts what the host uses.
Triage steps for a suspected OOM kill
Work through these steps in order, because each one narrows the cause before the next.
- Confirm the limit with
docker inspect --format '{{.HostConfig.Memory}}'. A value of 0 means no per-container hard limit is set. - Read the exit state with the
ExitCode,OOMKilled, andErrorfields. - Search the event log with
docker events --filter event=oom --since 1h, and look for kill events with signal 9 or 15 if the container was stopped. - Search the kernel log for "Memory cgroup out of memory" (container limit) or "Out of memory" (host), and check that the container’s ID appears in the
task_memcg=field. - Read
memory.eventsfrom the running container, or the cgroup v1 files on older hosts. - Read
memory.statto separate anonymous memory from file cache before you call the problem a leak. - Choose the fix from the cause. Raise the limit or reduce concurrency for a container-limit OOM. Fix the shutdown path for a stop-timeout kill. Add host capacity or lower the limits of other containers for a host-wide OOM.
Where this method falls short
The official Docker pages were available only as excerpts for this article. The swap rules and the --oom-kill-disable warning come from the official run reference, but the official excerpts do not state a default swap amount, so do not assume one. The OOMKilled field is described more narrowly in one guide and more broadly in others. Use the kernel log and the cgroup counters to confirm the cause, not the flag alone.
The OOM message names come from a 2018 kernel patch, and the counter behavior comes from kernel documentation plus several secondary guides. Newer kernels, cgroup v1 hosts, and Docker Desktop virtual machines can behave differently, so check the output on the host you run.
The method also cannot prove a leak from a single reading. A container with a high cache share and a container with a slow leak can look similar at one moment. Compare samples taken over the same workload before you change code or limits.
Finally, --oom-kill-disable removes the OOM killer for the container. The run reference says to use that flag only with -m. Without a memory limit, the container can keep growing until the host itself runs out of memory. Use it only when you have a limit in place and a plan for what the stalled container does.
Key Takeaways
- Confirm the cause with two records: Check
OOMKilledwithdocker inspectand search the kernel log for "Memory cgroup out of memory" to confirm a container-limit kill. - Treat exit 137 as a signal, not a verdict: Exit 137 with
OOMKilledset to false points to a host-wide OOM, an external kill, or a stop that ran past its timeout, so check the cause before you raise a limit. - Read
memory.statbefore calling it a leak: Compare theanonandfilelines over several samples, because file cache can push usage toward the limit without a kill. - Do not trust
docker statsas an OOM alarm: The displayed usage subtracts inactive file cache, so readmemory.eventsin the running container for kill evidence. - Match the fix to the cause: Raise the limit or cut concurrency for a container-limit OOM, and fix the shutdown path for a stop-timeout kill.
