Kubernetes secrets engine

The Kubernetes Secrets Engine for Vault generates Kubernetes service account tokens, and optionally service accounts, role bindings, and roles. The created service account tokens have a configurable TTL and any objects created are automatically deleted when the Vault lease expires.

For each lease, Vault will create a service account token attached to the defined service account. The service account token is returned to the caller.

To learn more about service accounts in Kubernetes, visit the Kubernetes service account and Kubernetes RBAC documentation.

Setup

The Kubernetes Secrets Engine must be configured in advance before it can perform its functions. These steps are usually completed by an operator or configuration management tool.

1. By default, Vault will connect to Kubernetes using its own service account. If using the standard Helm chart, this service account is created automatically by default and named after the Helm release (often vault, but this can be configured via the Helm value server.serviceAccount.name).

   It's necessary to ensure that the service account Vault uses will have permissions to manage service account tokens, and optionally manage service accounts, roles, and role bindings. These permissions can be managed using a Kubernetes role or cluster role. The role is attached to the Vault service account with a role binding or cluster role binding.

   For example, a minimal cluster role to create service account tokens is:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: k8s-minimal-secrets-abilities
   rules:
   - apiGroups: [""]
     resources: ["serviceaccounts/token"]
     verbs: ["create"]
   

   Similarly, you can create a more permissive cluster role with full permissions to manage tokens, service accounts, bindings, and roles.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: k8s-full-secrets-abilities
   rules:
   - apiGroups: [""]
     resources: ["serviceaccounts", "serviceaccounts/token"]
     verbs: ["create", "update", "delete"]
   - apiGroups: ["rbac.authorization.k8s.io"]
     resources: ["rolebindings", "clusterrolebindings"]
     verbs: ["create", "update", "delete"]
   - apiGroups: ["rbac.authorization.k8s.io"]
     resources: ["roles", "clusterroles"]
     verbs: ["bind", "escalate", "create", "update", "delete"]
   

   Create this role in Kubernetes (e.g., with kubectl apply -f).

   Moreover, if you want to use label selection to configure the namespaces on which a role can act, you will need to grant Vault permissions to read namespaces.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: k8s-full-secrets-abilities-with-labels
   rules:
   - apiGroups: [""]
     resources: ["namespaces"]
     verbs: ["get"]
   - apiGroups: [""]
     resources: ["serviceaccounts", "serviceaccounts/token"]
     verbs: ["create", "update", "delete"]
   - apiGroups: ["rbac.authorization.k8s.io"]
     resources: ["rolebindings", "clusterrolebindings"]
     verbs: ["create", "update", "delete"]
   - apiGroups: ["rbac.authorization.k8s.io"]
     resources: ["roles", "clusterroles"]
     verbs: ["bind", "escalate", "create", "update", "delete"]
   

   Note: Getting the right permissions for Vault will require some trial and error most likely since Kubernetes has strict protections against privilege escalation. You can read more in the Kubernetes RBAC documentation.

   Note: Protect the Vault service account, especially if you use broader permissions for it, as it is essentially a cluster administrator account.

2. Create a role binding to bind the role to Vault's service account and grant Vault permission to manage tokens.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: vault-token-creator-binding
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: k8s-minimal-secrets-abilities
   subjects:
   - kind: ServiceAccount
     name: vault
     namespace: vault
   

   For more information on Kubernetes roles, service accounts, bindings, and tokens, visit the Kubernetes RBAC documentation.

3. If Vault will not be automatically managing roles or service accounts (see Automatically Managing Roles and Service Accounts), then you will need to set up a service account that Vault will issue tokens for.

   Note: It is highly recommended that the service account that Vault issues tokens for is NOT the same service account that Vault itself uses.

   The examples we will use will under the namespace test, which you can create if it does not already exist.

   $ kubectl create namespace test
   namespace/test created
   

   Here is a simple set up of a service account, role, and role binding in the Kubernetes test namespace with basic permissions we will use for this document:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: test-service-account-with-generated-token
     namespace: test
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: test-role-list-pods
     namespace: test
   rules:
   - apiGroups: [""]
     resources: ["pods"]
     verbs: ["list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: test-role-abilities
     namespace: test
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: test-role-list-pods
   subjects:
   - kind: ServiceAccount
     name: test-service-account-with-generated-token
     namespace: test
   

   You can create these objects with kubectl apply -f.

4. Enable the Kubernetes Secrets Engine:

   $ vault secrets enable kubernetes
   Success! Enabled the kubernetes Secrets Engine at: kubernetes/
   

   By default, the secrets engine will mount at the same name as the engine, i.e., kubernetes/ here. This can be changed by passing the -path argument when enabling.

5. Configure the mount point. An empty config is allowed.

   $ vault write -f kubernetes/config
   

   Configuration options are available as specified in the API docs.

6. You can now configure Kubernetes Secrets Engine to create a Vault role (not the same as a Kubernetes role) that can generate service account tokens for the given service account:

   $ vault write kubernetes/roles/my-role \
       allowed_kubernetes_namespaces="*" \
       service_account_name="test-service-account-with-generated-token" \
       token_default_ttl="10m"
   

Generating credentials

After a user has authenticated to Vault and has sufficient permissions, a write to the creds endpoint for the Vault role will generate and return a new service account token.

$ vault write kubernetes/creds/my-role \
    kubernetes_namespace=test

Key                        Value
–--                        -----
lease_id                   kubernetes/creds/my-role/31d771a6-...
lease_duration             10m0s
lease_renwable             false
service_account_name       test-service-account-with-generated-token
service_account_namespace  test
service_account_token      eyJHbGci0iJSUzI1NiIsImtpZCI6ImlrUEE...

You can use the service account token above (eyJHbG...) with any Kubernetes API request that its service account is authorized for (through role bindings).

$ curl -sk $(kubectl config view --minify -o 'jsonpath={.clusters[].cluster.server}')/api/v1/namespaces/test/pods \
    --header "Authorization: Bearer eyJHbGci0iJSUzI1Ni..."
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "1624"
  },
  "items": []
}

When the lease expires, you can verify that the token has been revoked.

$ curl -sk $(kubectl config view --minify -o 'jsonpath={.clusters[].cluster.server}')/api/v1/namespaces/test/pods \
    --header "Authorization: Bearer eyJHbGci0iJSUzI1Ni..."
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

TTL

Kubernetes service account tokens have a time-to-live (TTL). When a token expires it is automatically revoked.

You can set a default (token_default_ttl) and a maximum TTL (token_max_ttl) when creating or tuning the Vault role.

$ vault write kubernetes/roles/my-role \
    allowed_kubernetes_namespaces="*" \
    service_account_name="new-service-account-with-generated-token" \
    token_default_ttl="10m" \
    token_max_ttl="2h"

You can also set a TTL (ttl) when you generate the token from the credentials endpoint. The TTL of the token will be given the default if not specified (and cannot exceed the maximum TTL of the role, if present).

$ vault write kubernetes/creds/my-role \
    kubernetes_namespace=test \
    ttl=20m

Key                        Value
–--                        -----
lease_id                   kubernetes/creds/my-role/31d771a6-...
lease_duration             20m0s
lease_renwable             false
service_account_name       new-service-account-with-generated-token
service_account_namespace  test
service_account_token      eyJHbGci0iJSUzI1NiIsImtpZCI6ImlrUEE...

You can verify the token's TTL by decoding the JWT token and extracting the iat (issued at) and exp (expiration time) claims.

$ echo 'eyJhbGc...' | cut -d'.' -f2 | base64 -d  | jq -r '.iat,.exp|todate'
2022-05-20T17:14:50Z
2022-05-20T17:34:50Z

Audiences

Kubernetes service account tokens have audiences.

You can set default audiences (token_default_audiences) when creating or tuning the Vault role. The Kubernetes cluster default audiences for service account tokens will be used if not specified.

$ vault write kubernetes/roles/my-role \
    allowed_kubernetes_namespaces="*" \
    service_account_name="new-service-account-with-generated-token" \
    token_default_audiences="custom-audience"

You can also set audiences (audiences) when you generate the token from the credentials endpoint. The audiences of the token will be given the default audiences if not specified.

$ vault write kubernetes/creds/my-role \
    kubernetes_namespace=test \
    audiences="another-custom-audience"

Key                        Value
–--                        -----
lease_id                   kubernetes/creds/my-role/SriWQf0bPZ...
lease_duration             768h
lease_renwable             false
service_account_name       new-service-account-with-generated-token
service_account_namespace  test
service_account_token      eyJHbGci0iJSUzI1NiIsImtpZCI6ImlrUEE...

You can verify the token's audiences by decoding the JWT.

$ echo 'eyJhbGc...' | cut -d'.' -f2 | base64 -d
{"aud":["another-custom-audience"]...

Automatically managing roles and service accounts

When configuring the Vault role, you can pass in parameters to specify that you want to automatically generate the Kubernetes service account and role binding, and optionally generate the Kubernetes role itself.

If you want to configure the Vault role to use a pre-existing Kubernetes role, but generate the service account and role binding automatically, you can set the kubernetes_role_name parameter.

$ vault write kubernetes/roles/auto-managed-sa-role \
    allowed_kubernetes_namespaces="test" \
    kubernetes_role_name="test-role-list-pods"

You can then get credentials with the automatically generated service account.

$ vault write kubernetes/creds/auto-managed-sa-role \
    kubernetes_namespace=test
Key                          Value
---                          -----
lease_id                     kubernetes/creds/auto-managed-sa-role/cujRLYjKZUMQk6dkHBGGWm67
lease_duration               768h
lease_renewable              false
service_account_name         v-token-auto-man-1653001548-5z6hrgsxnmzncxejztml4arz
service_account_namespace    test
service_account_token        eyJHbGci0iJSUzI1Ni...

Furthermore, Vault can also automatically create the role in addition to the service account and role binding by specifying the generated_role_rules parameter, which accepts a set of JSON or YAML rules for the generated role.

$ vault write kubernetes/roles/auto-managed-sa-and-role \
    allowed_kubernetes_namespaces="test" \
    generated_role_rules='{"rules":[{"apiGroups":[""],"resources":["pods"],"verbs":["list"]}]}'

You can then get credentials in the same way as before.

$ vault write kubernetes/creds/auto-managed-sa-and-role \
    kubernetes_namespace=test
Key                          Value
---                          -----
lease_id                     kubernetes/creds/auto-managed-sa-and-role/pehLtegoTP8vCkcaQozUqOHf
lease_duration               768h
lease_renewable              false
service_account_name         v-token-auto-man-1653002096-4imxf3ytjh5hbyro9s1oqdo3
service_account_namespace    test
service_account_token        eyJHbGci0iJSUzI1Ni...

API

The Kubernetes Secrets Engine has a full HTTP API. Please see the Kubernetes Secrets Engine API docs for more details.