Aaryamann Challani

Engineer and amateur cook writing about privacy, cryptography, and distributed systems

← Back to posts
April 11, 2021

Introduction

Secret Management for Kubernetes is tricky to implement at times.

There are many avenues for developers working on Kubernetes to access the secrets, if they haven't been setup properly

There are a few good open-source solutions to manage secrets on K8s -

  • Hashicorp Vault
  • Sealed Secrets
  • External Secrets

Hashicorp Vault

As mentioned on their website,

Hashicorp Vault allows Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.

Their offering is great, and offers quite a few features that certain teams may find appealing -

  • Huge variety of tooling
  • Multi-Party signing for Unlocking and Locking the Vault 🔒
  • Secret history/rollbacks 📘
  • Support for multiple backend data storages, i.e etcd, GCS, etc.
  • Robust RBAC and Advanced Permissioning System.

Vault is essentially a one-stop solution for all-things secret management. Hashicorp has a cloud offering as well, to make setup much easier!

There are a few things to keep in mind -

  • Unless you're setting it up on their cloud provider it takes time and some technical knowledge.
  • If you're using GCP/Azure, you can use their respective secret managers. It works better with their services.
  • Expect latency if your Vault is in a different geographical location.

Sealed Secrets

Have you ever wanted to check in your secrets to version control without worrying about it getting compromised?

Or maybe you already checked in secrets accidently 🤦🏽 and got in trouble? Then this solution is for you!

Sealed Secrets is a controller on Kubernetes, which solves a major problem with k8s secret management - Versioning. Sure, it is versioned using the kubeapi server, but for teams that wish to utilize the gitOps approach and have a "hands-off" experience with Kubernetes this is a great option.

Basically, the Operator you create creates an RSA keypair, which is used to encrypt and decrpyt the secrets. Once you create a object of type SealedSecret(which is encrypted), the controller automatically creates a Kubernetes Secret with the same name, and the desired secret payload.

Here are a few advantages to Sealed Secrets -

  • Version Control on Secrets 🤯
  • Labels for different "types" of secrets, like certificates, etc
  • Tool called kubeseal that manages the encryption locally
  • Namespace-specific sealing. Also supports cluster-wide secret creation ✅

This product comes from Bitnami, which is a leader in Kubernetes management.

There are a few things to keep in mind -

  • Key rotation. They have it documented here
  • Each user that wants to create secrets must have access to the certificate. You would need to implement a distribution mechanism
  • No good way to test the integrity of the secret created unless you're an administrator

External Secrets

This is my secret management of choice!

External Secrets, was produced by GoDaddy to have a generalized secret management interface, with support for multiple backends. This is by far one of the most extensible and flexible secret management options available.

Suppose your organization starts off small. You can use this controller for using the secret manager on whichever cloud you're on. Eventually when you start to scale you may have other ideas regarding secret management and wish to switch to a different provider(Hashicorp Vault, for instance).

The operator has support for multiple backends, like GCP Secret Manager, AWS Secret Manager, Hashicorp Vault, etc. As you may know, each of these services can be modified to suit the usecase as well. The controller will pick up the config from them.

Transitioning from one secret manager to another is super easy. You can pass in your secret-accessor credentials to the controller, and update the configs to the backend that you require. If the secrets exist in the destination manager, they will be picked up! Otherwise the secrets will continue to use the same backend 🚀

Some advantages of External Secrets -

  • Support for multiple backend managers
  • Automagically manages new secret versions 🧙🏾‍♂️
  • Pods can be injected using regular secret volumes
  • Tons of examples and assured code quality since the product is by goDaddy
  • Can be used with Version Control, since secrets aren't checked into the repo. Only the name is stored. 🙌

There are a few things to keep in mind -

  • For secret managers that don't support webhooks, REST calls will be made frequently.
  • Secrets must be namespaced

This were my views on few of the best secret management systems on Kubernetes. Thanks for reading!