Implementing Secrets Management in Kubernetes with External Secrets Operator and AWS Secrets Manager

Introduction

Managing sensitive data such as API keys, database passwords, and TLS certificates is one of the most critical security challenges in cloud-native environments. While Kubernetes provides native Secret objects, these are only base64-encoded — not truly encrypted — and managing them at scale across multiple clusters becomes unmanageable. Storing secrets in Git repositories, even encrypted, introduces additional complexity and risk.

The External Secrets Operator (ESO) solves this problem by synchronizing secrets from external provider APIs (like AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, or GCP Secret Manager) directly into Kubernetes Secrets. This allows teams to use a centralized, audited secrets backend while keeping their Kubernetes-native tooling.

In this tutorial, you will learn how to:

• Install the External Secrets Operator on an Amazon EKS cluster
• Configure IAM roles and policies for secure AWS integration
• Create a SecretStore resource to define the backend connection
• Define ExternalSecret resources that synchronize secrets automatically
• Consume the resulting Kubernetes Secrets inside pods
• Apply security best practices for secrets management in production

Prerequisites

Before beginning this tutorial, ensure you have the following:

• A running Kubernetes cluster (EKS, Kops, or any CNCF-certified distribution)
• kubectl configured with cluster admin access
• helm v3.8+ installed on your local machine
• AWS CLI configured with permissions to manage Secrets Manager and IAM
• An AWS account with Secrets Manager enabled in your target region

Tip: If you’re using an EKS cluster, you can use IAM Roles for Service Accounts (IRSA) to avoid managing long-lived AWS credentials entirely. This is the recommended approach for production environments.

Step 1: Install the External Secrets Operator

The External Secrets Operator can be installed via Helm from the official Helm chart repository. This is the preferred installation method as it handles CRD upgrades automatically.

First, add the External Secrets Helm repository:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

Next, install the operator into the external-secrets namespace:

helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --set installCRDs=true

Verify that the operator pods are running:

kubectl get pods -n external-secrets

NAME                                                READY   STATUS    RESTARTS   AGE
external-secrets-6f7c8d9b4f-abc12                  1/1     Running   0          45s
external-secrets-cert-controller-7d8e9c0b3f-def34  1/1     Running   0          45s
external-secrets-webhook-9a0b1c2d3e-ghi56          1/1     Running   0          45s

You should see three pods: the main operator, the cert-controller (for managing webhook certificates), and the webhook (for validating admission hooks).

Step 2: Set Up AWS IAM Permissions (IRSA Method)

For production deployments, the recommended approach is to use IAM Roles for Service Accounts (IRSA). This binds an IAM role to the Kubernetes ServiceAccount that ESO uses, eliminating the need for static AWS credentials.

2.1 Create the IAM Policy

Create a policy that grants read access to secrets matching a specific naming pattern. Save the following as eso-secrets-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecrets"
      ],
      "Resource": [
        "arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp-*"
      ]
    }
  ]
}

aws iam create-policy \
  --policy-name ExternalSecretsOperatorPolicy \
  --policy-document file://eso-secrets-policy.json

2.2 Create the IAM Role with Trust Policy

Create a trust policy that allows the external-secrets ServiceAccount in your cluster to assume the role:

# Get your OIDC provider URL
aws eks describe-cluster --name your-cluster --query "cluster.identity.oidc.issuer" --output text

# Example output: https://oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED1234567890ABCDEF

Create the trust policy file trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED1234567890ABCDEF"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED1234567890ABCDEF:sub": "system:serviceaccount:external-secrets:external-secrets"
        }
      }
    }
  ]
}

aws iam create-role \
  --role-name external-secrets-role \
  --assume-role-policy-document file://trust-policy.json

aws iam attach-role-policy \
  --role-name external-secrets-role \
  --policy-arn arn:aws:iam::123456789012:policy/ExternalSecretsOperatorPolicy

2.3 Annotate the ServiceAccount

Annotate the ESO ServiceAccount with the IAM role ARN so IRSA associates the two:

kubectl annotate serviceaccount external-secrets \
  -n external-secrets \
  eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/external-secrets-role

Note: If IRSA is not available in your environment, you can create a Kubernetes Secret with AWS credentials (access key + secret key) and reference it from the SecretStore. However, this is less secure and should only be used for development or testing.

Step 3: Create a SecretStore

The SecretStore resource defines how ESO connects to your external secrets provider. There are two types:

• SecretStore — namespaced, only accessible within its own namespace
• ClusterSecretStore — cluster-wide, accessible from any namespace

Create a namespaced SecretStore that connects to AWS Secrets Manager. Save this as secret-store.yaml:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-store
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

Apply it to your cluster:

kubectl apply -f secret-store.yaml

Check that the SecretStore is ready:

kubectl get secretstore aws-secrets-store -o jsonpath='{.status.conditions[0].type}'

Ready

If the status shows Ready, the operator has successfully validated the connection to AWS Secrets Manager.

Step 4: Create Secrets in AWS Secrets Manager

For testing, create two secrets in AWS Secrets Manager. First, a database credential:

aws secretsmanager create-secret \
  --name myapp-database \
  --secret-string '{"username":"dbadmin","password":"s3cur3P@ssw0rd!","host":"mydb.example.com","port":5432}' \
  --region eu-west-1

Next, an API key:

aws secretsmanager create-secret \
  --name myapp-api-key \
  --secret-string 'sk-abc123def456ghi789jkl' \
  --region eu-west-1

Step 5: Create ExternalSecret Resources

An ExternalSecret tells ESO what secret to fetch, what Kubernetes Secret to create, and how to transform the data.

5.1 Simple Key-Value Secret

For the API key (a plain string), we fetch it and store it directly:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-api-key
  namespace: default
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: aws-secrets-store
    kind: SecretStore
  target:
    name: myapp-api-key-secret
    creationPolicy: Owner
  data:
    - secretKey: api-key
      remoteRef:
        key: myapp-api-key

kubectl apply -f external-secret-api-key.yaml

5.2 JSON Secret with Property Extraction

For the database credential (a JSON object), we can extract individual fields:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-database
  namespace: default
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: aws-secrets-store
    kind: SecretStore
  target:
    name: myapp-database-secret
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: myapp-database
        property: username
    - secretKey: password
      remoteRef:
        key: myapp-database
        property: password
    - secretKey: host
      remoteRef:
        key: myapp-database
        property: host
    - secretKey: port
      remoteRef:
        key: myapp-database
        property: port

kubectl apply -f external-secret-database.yaml

5.3 Verify the Generated Secrets

Check that the Kubernetes Secrets were created:

kubectl get secrets

NAME                    TYPE     DATA   AGE
myapp-api-key-secret    Opaque   1      10s
myapp-database-secret   Opaque   4      5s

Decode and inspect them:

kubectl get secret myapp-api-key-secret -o jsonpath='{.data.api-key}' | base64 -d

sk-abc123def456ghi789jkl

Step 6: Consume Secrets in a Pod

Once the ExternalSecret synchronizes the data into a Kubernetes Secret, you consume it just like any other Secret — as environment variables, volumes, or in kustomize overlays.

Here’s a sample pod that uses both secrets:

apiVersion: v1
kind: Pod
metadata:
  name: myapp-consumer
  namespace: default
spec:
  containers:
    - name: myapp
      image: nginx:alpine
      env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: myapp-api-key-secret
              key: api-key
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: myapp-database-secret
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: myapp-database-secret
              key: password
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: myapp-database-secret
              key: host
        - name: DB_PORT
          valueFrom:
            secretKeyRef:
              name: myapp-database-secret
              key: port

kubectl apply -f consumer-pod.yaml

Verify that the environment variables are set correctly:

kubectl exec myapp-consumer -- printenv | grep -E '^(API_KEY|DB_)'

API_KEY=sk-abc123def456ghi789jkl
DB_USERNAME=dbadmin
DB_PASSWORD=s3cur3P@ssw0rd!
DB_HOST=mydb.example.com
DB_PORT=5432

Step 7: Implement PushSecret (Optional)

ESO also supports PushSecret — the inverse workflow where you can push a Kubernetes Secret into an external provider. This is useful for CI/CD pipelines that generate secrets at build time and need to store them centrally.

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: push-myapp-secret
  namespace: default
spec:
  secretStoreRefs:
    - name: aws-secrets-store
      kind: SecretStore
  refreshInterval: "1h"
  data:
    - match:
        secretKey: myapp-api-key
        remoteRef:
          remoteKey: myapp-api-key-pushed

Best Practices

Practice	Recommendation
Least Privilege IAM	Scope IAM policies to specific secrets using ARN patterns with wildcards (e.g., myapp-*). Avoid using "Resource": "*".
Use IRSA	Always prefer IAM Roles for ServiceAccounts over static access keys. This eliminates credential rotation and limits blast radius.
Refresh Intervals	Set refreshInterval to balance security and API costs. Every 1 hour is a good default; use shorter intervals for high-rotation secrets.
Creation Policy	Use creationPolicy: Owner so ESO manages the lifecycle of the target Secret. Use creationPolicy: Merge if you need to combine multiple ExternalSecrets or add annotations.
Encryption at Rest	Enable KMS customer-managed keys (CMK) on your AWS Secrets Manager secrets for additional encryption control.
Auditing	Enable AWS CloudTrail to log all GetSecretValue calls. Monitor for unexpected access patterns using CloudWatch alarms.
Namespace Isolation	Use namespaced SecretStore instead of ClusterSecretStore when possible to enforce tenant boundaries.

Troubleshooting Common Issues

SecretStore Not Ready

If the SecretStore status is not “Ready”, check the operator logs:

kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

Common causes include: incorrect IAM role ARN, missing trust policy conditions, or the OIDC provider not being associated with the EKS cluster.

ExternalSecret Not Syncing

Describe the ExternalSecret to see status conditions:

kubectl describe externalsecret myapp-database

Look for events like SecretSyncError or SecretSynced in the output. Common issues include referencing non-existent remote keys or incorrect property paths in JSON secrets.

Webhook Timeout on Apply

If applying ExternalSecret resources times out, verify the webhook pod is running and reachable:

kubectl get pods -n external-secrets -l app.kubernetes.io/name=external-secrets-webhook

Network policies or misconfigured TLS certificates can also cause webhook failures.

Conclusion

The External Secrets Operator bridges the gap between centralized secrets management and Kubernetes-native workflows. By integrating with AWS Secrets Manager, you get:

• Centralized management — all secrets in one auditable, backup-enabled service
• Automatic synchronization — secrets update automatically when the backend changes
• No vendor lock-in — ESO supports 20+ providers including Vault, Azure Key Vault, GCP Secret Manager, and more
• GitOps compatibility — ExternalSecret manifests can live in Git repos without exposing sensitive data

By following the steps in this tutorial, you have implemented a production-ready secrets management pipeline that keeps sensitive data out of your cluster’s etcd store, out of your Git repositories, and under proper access control in AWS Secrets Manager.

Next Steps: Explore the External Secrets Operator’s advanced features such as templating with templateEngine, deploying with Flux CD or ArgoCD for GitOps-driven secrets, and configuring PushSecret for CI/CD workflows that need to create secrets dynamically.