Introduction

What is configuration drift? Configuration drift is the gradual divergence of a system’s actual configuration from the secure baseline configuration. The proper configuration of your infrastructure components is vital for security, compliance and performance. But setting changes are often made without formal approval, proper testing or clear documentation. And the configuration drift in your systems, applications and infrastructure can create security gaps that put an organization at risk. 

Infrastructure as Code (IaC) came with many promises, including being able to help manage configuration drift. Yet the reality of real world implementations look different than the intended goals. This means that the intended/coded behavior of the infrastructure/systems does not match the actual behavior as observed in the wild. 

We are going to outline some reasons for configuration drift and propose a few methods to prevent them.

Reason #1: Issues Related to Migration

Description

Most projects start with the manual provisioning of resources in the AWS console. As the project grows, teams begin to adopt infrastructure as code tools, resulting in a mix of code-provisioned and user-provisioned resources. If teams are not using immutable infrastructure the lingering artifacts from a previously used approach can impact the overall infrastructure’s behavior. Or the quick hotfixes that we’re all guilty of using to address a network issue or workload don’t make their way back into the IaC repository. 

Prevention

When you move to IaC, move fully to IaC. There are tools to help you to overcome your original clickops. For example, you can start by exporting all resources as Terraform files (using tools like TerraFormer) and transitioning to 100% infrastructure as code resources can eliminate confusion.

After that, revoke all console access from your users and accept some breaking-glass accounts that are needed to quickly perform an action when things go sideways. In AWS, this can be e.g. done by invoking

aws iam delete-login-profile --user-name username

Reason# 2: Frameworks Built on Frameworks

Description

Kubernetes is hard, Terraform is hard.

Working with sophisticated tools such as Kubernetes and Terraform can present challenges. To streamline their usage, developers have created tools like TerraGrunt and Terraspace. With these tools, certain configurations remain immutable while others can change inadvertently. This gives rise to disparities and "drift". It’s a particular challenge as the team grows, and new tools are adopted into existing projects.

Prevention

The way to resolve this is to choose a particular framework and be consistent with the adoption and use of the newly chosen framework. No mix and match. Ideally, study the framework before you commit to it, and ensure that it covers all the features of the sub-tool you need today and in the future.

Reason # 3: Inline Configuration vs. Separate Files

Description

Look at this HELM chart as an example. 

You can see that the contents of a typical my.cnf file, i.e. the MySQL/MariaDB configuration file, is being posted inside the values file. While this is a valid option, it is important to again ensure that consistency is maintained. We generally recommend storing this information in a separate file in your code repository, since this would aid the organization of the project and maintainability by a human.

If consistency is not maintained, situations may occur where certain environments use the values from the chart, and others use a configuration file inside the repository.

Prevention

In order to instruct your testing and deploying efforts in a repeatable manner, again, do not mix and match. If asked which strategy is the better, we from CoGuard would recommend keeping things in separate files and reference those. This makes it generally easier to keep consistency.

Reason # 4: Environment Variables vs. Configuration Files

Description

Many tools offer configuration options through both configuration files and environment variables. Depending on the specific tool, environment variables may take precedence or not. It's common for developers to mix both approaches, especially when seeking advice from platforms like Stack Overflow or using AI assistants like CoPilot/ChatGPT. However, this mixing can lead to unexpected issues as it may not be clear which setting takes precedence.

Prevention

A general guideline is to use environment variables for anything that is a secret or specific to the environment, such as a domain name. Other configuration settings should be stored in versioned configuration files.

Additionally, when defining environment variables, it is important to have a deploy script in place. This ensures that the values are properly managed and tracked, rather than being directly set on the machines without any record.

Reason # 5: Versioning Your Configuration Files

Description

Configuration files play a vital role in software projects, particularly in deployment and release management. Unfortunately, they are often overlooked in this context. It is crucial to recognize the importance of versioning configuration files to ensure consistency and facilitate proper deployment and release management.

Take for example referenced Docker images. In your Kubernetes/Docker-compose files you may find lines like

image: mysql:8

or similar. In this case, you are trusting the configuration file inside that container, unless you mount it in somehow. CoGuard would detect that and scan the file anyway for you, and make you aware of this gray area in your repository. Once you see it in the report, you should move it into the repository.

Prevention

Anytime you are adopting a new tool in your technology stack, check if it has a configuration file. If it does, version it. Run the coguard-cli in order to detect if you are thoughtlessly using such a project today.

Using CoGuard to Avoid Configuration Drift

The NIST Special Publication 800-128 provides some guidance for avoiding configuration drift. We built CoGuard to help teams identify configuration files for IaC, containers, applications and networks across your infrastructure. 

• Implementation of configuration tools that automate the creation, modification and deployment of configuration settings across your server and network infrastructure using established templates. Manual efforts are prone to human error, and slower than automated processes, meaning that setting may remain in a vulnerable state longer.
  
  CoGuard can help teams discover the configuration files in infrastructure source code repositories, cloud management administration panels, default configurations used by applications inside containers, and others. 
• Use a repository of benchmarks, templates, and configurations that can be used as templates to identify configuration drift. Consider using CIS or NIST as starting points to build your configurations.
  
  CoGuard scans configurations using a variety of security benchmarks and best practices including CIS, NIST, PCI DSS, SoC2 and others. The CoGuard results allow for the ability to start with the existing configuration, make adjustments to harden your infrastructure, and track changes over time. 
• Implementation of continuous monitoring of configuration changes so that improper modifications can be identified. Monitoring efforts should be supplemented with regular audits.
  
  CoGuard can be added to your CI/CD pipeline and can halt the build based on a variety of parameters including the volume or severity of issues, changes from baseline, etc. This does not change the build and release process but allows configuration changes to be included in the existing processes.
  
• Standardize your configuration change management processes to minimize the chance of configuration drift. Every configuration change should be approved and documented using this system.
  
  Understanding the importance of build and release processes and the impact on teams is something done with small startup teams all the way through large military providers. We happen to work with your team to provide coaching and examples of configuration change management processes that work for engineering and infrastructure teams of any size. 

Get started by finding your configuration files

pip3 install coguard-cli 
coguard folder ./INFRASTRUCTURE_REPO/ –dry-run=true

Get started by finding the configuration files in your build repository, including containers and default configurations.This is the first step in hardening your configuration files in your  infrastructure.

Get started for free »

‍