Secrets Management

Imagine you are deploying an application that requires access to a database. You have non-sensitive configuration data (like the database’s port number) and highly sensitive data (like the database password). Storing both in the same place—or worse, hardcoding the password in your container image—is a major security risk.

In Kubernetes, while ConfigMaps handle non-sensitive configuration, Secrets are specifically designed to store and manage sensitive information such as passwords, OAuth tokens, and SSH keys.

1. Secrets vs. ConfigMaps: The “Keycard” and the “Manual”

If a ConfigMap is an “Instruction Manual” left on a desk for anyone to read, a Secret is the “Keycard” hidden in your pocket. They share a similar Key-Value structure, but Secrets have crucial differences:

  1. Obfuscation (Not Encryption): By default, K8s Secrets are simply Base64 encoded, not encrypted. Base64 is an encoding scheme, meaning it can be easily reversed by anyone.
  2. RAM-Backed Storage: When a Secret is mounted into a Pod as a volume, it is stored in tmpfs (a RAM-backed filesystem) on the Node. It is never written to the Node’s physical disk, reducing the risk of a compromised disk exposing secrets.
  3. Strict RBAC: You can (and should) use Role-Based Access Control (RBAC) to restrict who can read Secret resources independently of ConfigMap resources.

[!WARNING] Base64 is NOT Encryption. Anyone with permission to run kubectl get secret <name> -o yaml can decode your secrets. By default, Secrets are stored unencrypted in the cluster’s underlying ETCD database.

Secret Types

Kubernetes provides built-in types to handle common secret formats:

  • Opaque: The default type for arbitrary user-defined key-value pairs.
  • kubernetes.io/tls: specifically for storing TLS certificates and private keys.
  • kubernetes.io/dockerconfigjson: Used to store credentials for pulling images from private container registries.

2. Interactive: The Base64 “Encryption” Illusion

Prove to yourself that Kubernetes Secrets are merely encoded strings. Type a secret below to see how it looks in Kubernetes, and notice how easily it can be reversed.

Base64 Encoder/Decoder

echo -n 'hunter2' | base64
⇄
echo 'aHVudGVyMg==' | base64 --decode

3. Consuming Secrets in Pods

Just like ConfigMaps, Secrets can be exposed to Pods in two main ways: Environment Variables or Volume Mounts.

Option A: Environment Variables

Injecting secrets as environment variables is simple but has a major drawback: they do not hot-reload. If the Secret gets updated, the Pod must be restarted to pick up the new value. Additionally, poorly written applications might dump environment variables to logs during a crash.

Mounting a Secret as a volume places the values into files (e.g., /etc/secrets/db-password). Kubernetes dynamically updates these files when the Secret changes, allowing applications to hot-reload credentials without a restart.

4. Production Security Patterns

The default state of Kubernetes Secrets is insufficient for production. How do you securely manage Secrets when following GitOps practices (where all K8s manifests are stored in a Git repository)? Committing Base64-encoded passwords to GitHub is a critical security violation.

Here are the industry-standard solutions:

1. External Secrets Operator (ESO)

The Enterprise Standard: Instead of creating K8s Secrets directly, you store your passwords in an external Enterprise Vault (like AWS Secrets Manager, Azure KeyVault, or HashiCorp Vault). You then deploy the External Secrets Operator in your Kubernetes cluster. ESO authenticates with the external vault, fetches the secret, and automatically generates a native K8s Secret inside the cluster.

2. Sealed Secrets (Bitnami)

The Asymmetric Encryption Approach:

  • A controller runs inside the K8s cluster and generates a public/private key pair.
  • Developers use a CLI tool (kubeseal) on their local machines, leveraging the public key to encrypt the Secret into a SealedSecret manifest.
  • This SealedSecret is fully encrypted and safe to commit to Git.
  • Once applied to the cluster, the controller uses its private key to decrypt the SealedSecret and generate a standard K8s Secret.

3. Encryption at Rest (ETCD)

Even if you use ESO or Sealed Secrets, the resulting native Secret is still stored in the cluster’s ETCD database as plain Base64. To protect against infrastructure-level breaches (e.g., someone stealing the ETCD backup files), you must enable Encryption at Rest at the API server level. This requires passing an --encryption-provider-config flag to the kube-apiserver, pointing to a configuration file that dictates which resources (like secrets) should be encrypted using a cryptographic provider (like AES-CBC, or a KMS).

5. Solutions: Secret Manifest & Pod Usage

Below is a complete example of defining an Opaque Secret and securely injecting it into a Pod using environment variables.

Secret Manifest
Pod Usage
apiVersion: v1
kind: Secret
metadata:
  name: db-auth
# Opaque is the default for arbitrary key/value pairs
type: Opaque
data:
  # Values must be explicitly base64 encoded before applying
  # echo -n 'admin' | base64 -> YWRtaW4=
  username: YWRtaW4=
  # echo -n 'p@ssword' | base64 -> cEBzc3dvcmQ=
  password: cEBzc3dvcmQ=

apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  containers:
  - name: app
    image: my-app:1.0
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: db-auth
          key: username
    - name: DB_PASS
      valueFrom:
        secretKeyRef:
          name: db-auth
          key: password