Troubleshooting Configurations

ElasticSearch’s most common reason for “exited unexpectedly”

Optimize ElasticSearch's First Run: Troubleshoot "Exited Unexpectedly" Error Ensure proper RAM allocation. Be aware of cross container, application and user limits to prevent unexpected exits. Find configuration collisions with CoGuard.

CoGuard
Written by
CoGuard

The first run of ElasticSearch and the ELK stack, often results in an:

ERROR: elasticsearch exited unexpectedly

In our experience, the most common reason is related to RAM memory allocation, and here are some troubleshooting tips.

Troubleshooting Step 1: vm.max_map_count

ElasticSearch is using mmapfs and does so extensively. On most systems, by default, vm.max_map_count is set to 65530, whereas ElasticSearch requires 262144. You can check the current value on your system by typing:

sysctl -a | grep vm.max_map_count

To set the value to the correct one, you can type

sysctl -w vm.max_map_count=262144

This command needs to be run as root user.

Sidenote: If you are using containers, the containers will automatically use this value from the host, unless instructed otherwise.

Troubleshooting Step 2: Heap Memory

ElasticSearch is based on Java, and Java’s memory management is key. 

If not set specifically, the heap size is relative to the available memory of the host. To avoid issues, it is recommended to always set the expected memory manually when starting up ElasticSearch by setting the XMX and XMS settings to at least 4GB each. More information on how to do that can be found here.

Troubleshooting 3: Container/User Limitations

As mentioned in Troubleshooting Step 2, the Java Heap memory is by default calculated with respect to the available memory on the system.

However, if you run within a container and set a memory limit on that container, Java will not be aware of the limits set on the container via the ‘limits’ directive (-m e.g., more details https://docs.docker.com/config/containers/resource_constraints/). If the memory limit for the Java Heap exceeds the container set limit, the container will be stopped.

The same applies to memory limits for users and processes via the limits.conf file. If they exceed the container set limit, the container will be stopped.

Find container and application configuration intersections 

CoGuard is a code scanner for configuration files. It has a Generalized Infrastructure Model that identifies intersections of configuration parameters between applications, operating systems and devices/containers like those described above. You can scan your IaC configurations, containers and cloud infrastructure for misconfigurations and security best practices. 

Get started for free today »

Explore a test environment

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Check out and explore a test environment to run infra audits on sample repositories of web applications and view select reports on CoGuard's interative dashboard today.