The first run of ElasticSearch and the ELK stack, often results in an:
ERROR: elasticsearch exited unexpectedly
In our experience, the most common reason is related to RAM memory allocation, and here are some troubleshooting tips.
Troubleshooting Step 1: vm.max_map_count
ElasticSearch is using mmapfs
and does so extensively. On most systems, by default, vm.max_map_count
is set to 65530
, whereas ElasticSearch requires 262144
. You can check the current value on your system by typing:
sysctl -a | grep vm.max_map_count
To set the value to the correct one, you can type
sysctl -w vm.max_map_count=262144
This command needs to be run as root user.
Sidenote: If you are using containers, the containers will automatically use this value from the host, unless instructed otherwise.
Troubleshooting Step 2: Heap Memory
ElasticSearch is based on Java, and Java’s memory management is key.
If not set specifically, the heap size is relative to the available memory of the host. To avoid issues, it is recommended to always set the expected memory manually when starting up ElasticSearch by setting the XMX and XMS settings to at least 4GB each. More information on how to do that can be found here.
Troubleshooting 3: Container/User Limitations
As mentioned in Troubleshooting Step 2, the Java Heap memory is by default calculated with respect to the available memory on the system.
However, if you run within a container and set a memory limit on that container, Java will not be aware of the limits set on the container via the ‘limits
’ directive (-m
e.g., more details https://docs.docker.com/config/containers/resource_constraints/). If the memory limit for the Java Heap exceeds the container set limit, the container will be stopped.
The same applies to memory limits for users and processes via the limits.conf
file. If they exceed the container set limit, the container will be stopped.
Find container and application configuration intersections
CoGuard is a code scanner for configuration files. It has a Generalized Infrastructure Model that identifies intersections of configuration parameters between applications, operating systems and devices/containers like those described above. You can scan your IaC configurations, containers and cloud infrastructure for misconfigurations and security best practices.
Get started for free today »