Kafka is s a well-engineered scalable messaging system, but along with it comes a large number of different configurations and one in particular that you need to look out for.
Kafka is a service that gained tremendous popularity since its release into the Apache incubator in 2011. And rightfully so, as it is a well-engineered scalable messaging system.
The number of different configurations is large, as you can see in the documentation.
There is one configuration you need to look out for though, depending on your needs:
By default, this parameter is set to false, as it comes with a risk of data loss in exchange for a larger resilience to node-failures. In your applications, if you do not care that all messages are delivered at every moment, then you can set it to true and have your system survive an even larger number of node failures. However I would argue that this is something that should be a conscientious choice.
AWS’s hosted Kafka MSK system, however, is setting this parameter by default to true (see documentation). Hence, they assume that their users are fine with the potential data-loss to node-recovery tradeoff. This is a bold assumption, and it is something which should be checked for every individual case.
This is why we included a rule in CoGuard which fails if it encounters default configurations for MSK. This may be more of an availability problem than a security issue, but since Kafka is often also used to transfer logs, it also falls under the security category.
Contact us to learn more: info@coguard.io
