Secrets in Kubernetes and HashiCorp Vault

In this article, I will try to explain the features that Kubernetes provides to store sensitive data, credentials, auth token as Secret that will run under a Pod. Also, we will see a more robust approach to store Secrets using HashiCorp Vault.

Image for post
Image for post

Kubernetes Secrets

In application development, sensitive data storage is a critical operation. A single key or credential leak can bridge an entire database system in a network. So, Kubernetes provide flexibility where we can store a small size of data in different format like in a file system, environmental variable, and mount them into a volume. The Secrets will be mounted to a Pod to be accessible.

Secrets as Environmental Variables

The secrets can be mounted as an env variable and in a Pod definition, each container will contain a secret key that can be accessed from env[].valueFrom.secretKeyRef.

  • Create Secret literals using kubectl
$ kubectl create secret generic mongosecret — from-literal=’username=demoadmin’ — from-literal=’password=demopwd123'
  • Deploy Secret as Environment variable in Pod

So, under a container, multiple env can be created. In this Pod deployment, we have created two Secrets as ‘SECRET_USERNAME’ and ‘SECRET_PASSWORD’.

If we get into the Pod shell

$ kubectl exec -it <<pod-name>> — /bin/sh# echo $SECRET_USERNAME
# demoadmin
# echo $SECRET_PASSWORD
# demopwd123

Potential Risk using Kubernetes Secrets

There are many different possibilities of risk involved with using Kubernetes Secrets if the Pods are not configured properly.

  • In a shared environment, if the administrator creates secrets in the cluster and may be possible for unauthorized access to the environment variable by the malicious actor.
  • Avoid writing environmental variables into the logs as they are widely visible in the plain text.
  • In a large scale cloud infrastructure, there is a higher chance of misconfiguration of microservices, containers can easily create security vulnerabilities to gain access to the container images to read envs.
  • Creating various levels of access policy among containers, pods are highly essential as every node in Kubernetes environment talk to each other that give all access to file systems, database cluster, and many other exposed endpoints.
  • Frequent security assessment and audit of each container has to be an essential practice for the network administrator.

A robust approach with HashiCorp Vault

The HashiCorp Vault provides a variety of pluggable component architecture that store secrets with access-controls, policies, token management, and key sharing encryption algorithms. The access control to sensitive data can be changed dynamically on lease or Time to Live (TTL).

Vault Seal/Unseal

The initial status of the vault is sealed and cannot decrypt any data stored in the physical storage. Vault follows the Shamir Key Sharing Algorithm that creates multiple pieces of the master key. So, the vault requires a master key to unlock the encryption key to decrypt the data that is stored inside the vault.

$ export VAULT_ADDR=http://127.0.0.1:8200
$ vault operator init
Unseal Key 1: SVJHjpSU/1Y3I5LqHJM5oQ+WCyQ52KJSb6/4939phE97
Unseal Key 2: cPBI2gDQX7o3igJwml34fo1wN7mM0WZO++fW5Al0Vc3J
Unseal Key 3: pUhQ5HsnBvaCOtjRNdNo5d6FMw9iWKN0/UZTAeyM11kB
Unseal Key 4: yTR7BKgEkMG+hSClBu9yrx13sLMioXukTJsONUrQClWY
Unseal Key 5: prDHpyCzyDAu2g1MvG81ammS1cgqV8pQCQ+WZvb55zpu
Initial Root Token: s.su8bzw3WALaNfGCWVdrhALdK

The initialization of the vault creates 5 key shares and a threshold of 3 keys to unseal the vault. These unseal keys are only visible in the local environment but in the real scenario, these keys won’t be visible altogether, and also they will be encrypted using several tools like Keybase and HashiCorp’s PGP.

$ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.5.2
Cluster Name vault-cluster-29ef6044
Cluster ID 00081a9e-4f31-3902-4b8c-3b047fefad34
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active

The administrator can also login with root token.
$ vault login s.su8bzw3WALaNfGCWVdrhALdK

Create Static Secret with Vault using Minikube

We will create a sample static secret using Vault under a minikube environment similar to the environmental variable secret using Kubernetes.

  • Install Consul Helm Chart

HashiCorp provides a Consul Helm chart that provides a configured Consul to run under a Vault Dev Server Mode.

Add HashiCorp Helm repository

$ helm repo add hashicorp https://helm.releases.hashicorp.com
“hashicorp” has been added to your repositories
  • Deploy Consul Client and Server Pods
$ cat helm-consul-values-yaml
$ helm install consul hashicorp/consul — values helm-consul-values.yaml
NAME: consul
LAST DEPLOYED: Tue Sep 29 23:26:53 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Consul!
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
consul-consul-ffqnh 1/1 Running 0 5m46s
consul-consul-server-0 1/1 Running 0 5m45s

Both Client and Server Pods deployed successfully.

  • Install Vault Helm Chart

The Vault Helm Chart will run in high-availability mode and will create a file storage backend.

$ cat helm-vault-values.yaml
$ helm install vault hashicorp/vault — values helm-vault-values.yaml 
NAME: vault
LAST DEPLOYED: Tue Sep 29 23:43:38 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Vault!

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
consul-consul-ffqnh 1/1 Running 0 18m
consul-consul-server-0 1/1 Running 0 18m
vault-0 0/1 Running 0 95s
vault-1 0/1 Running 0 93s
vault-2 0/1 ContainerCreating 0 92s
vault-agent-injector-857cdd9594-xq8cv 0/1 ContainerCreating 0 95s

This installation will create Vault Pods and Vault Agent but they are not ready as the status shows (0/1) because the Vault seal status is true.

$ kubectl exec -it vault-0 -- vault status
Key Value
— — — — -
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version n/a
HA Enabled true
command terminated with exit code 2

Unseal the Vault

Initialize the Vault with -key-shares=5 and -key-threshold=3

$ kubectl exec -it vault-0 -- vault operator init -key-shares=5 -key-threshold=3 -format=json > vault-keys.json
$ cat vault-keys.json

Now Unseal the vault-0 with the unseal keys. We need to enter at-least 3 unseal keys to change the seal status false.

$ kubectl exec -it vault-0 -- /bin/sh
$ vault operator unseal
Unseal Key (will be hidden): SMY/5QwJkqc4bHAWiEg0bA5h50M+5kkkE4q6iHP14rG/
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce db5bf091-4695-9238-61e2-66ae08e97b8e
Version 1.5.2
HA Enabled true
$ vault operator unseal
Unseal Key (will be hidden): 0oO5su7BJ7uiAFfX0lnB9Omb8ZkWU8/CN6gPb5YqNbec
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce db5bf091-4695-9238-61e2-66ae08e97b8e
Version 1.5.2
HA Enabled true
$ vault operator unseal
Unseal Key (will be hidden): PK0EY++Y/zL5OgeHMlwA3dC7wn8gF6nsDDHMLBOD/NvD
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.5.2
Cluster Name vault-cluster-ea4ff059
Cluster ID 34e1ad16-6aca-5085-3442-0acf00a3ebbd
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>

Now vault-0 is unseal, similar will follow for vault-1, vault-2.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
consul-consul-ffqnh 1/1 Running 0 38m
consul-consul-server-0 1/1 Running 0 38m
vault-0 1/1 Running 0 22m
vault-1 1/1 Running 0 22m
vault-2 1/1 Running 0 22m
vault-agent-injector-857cdd9594-xq8cv 1/1 Running 0 22m

All Vault Pods and Vault Agents are now running.

Vault Web UI

Vault Web Interface can be accessible at http://localhost:8200/ui from the web browser, so please follow the below command for port-forwarding at 8200.

$ kubectl port-forward vault-0 8200:8200
Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200

Image for post
Image for post
Vault Sign in — Use root token to authenticate
  • Set Secret in Vault

Now, we can proceed to create a secret in Vault at the path secret/webapp/config. The secrets will be stored as key-value pairs in the Vault secret engine and login into the Vault via root token is required to enable the key-value secret engine.

$ kubectl exec -it vault-0 — /bin/sh
$ vault login
Token (will be hidden): s.4XLyZOkSzGynYL0GDCCTruJ7
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.4XLyZOkSzGynYL0GDCCTruJ7
token_accessor wkHbTPmh5pnxdbKQNwbT7b1L
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
$ vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/

Create Secret at path secret/webapp/config

$ vault kv put secret/webapp/config username=”demoadmin” password=”demopwd123"
Key Value
— — — — -
created_time 2020–09–29T18:45:21.598063156Z
deletion_time n/a
destroyed false
version 1
$ vault kv get secret/webapp/config
====== Metadata ======
Key Value
--- -----
created_time 2020-09-29T18:45:21.598063156Z
deletion_time n/a
destroyed false
version 1
====== Data ======
Key Value
--- -----
password demopwd123
username demoadmin

The static secret as username and password is created at Vault secret engine.

Image for post
Image for post
Vault Secret Dashboard

Configure Vault with Authentication

Vault will authenticate web applications via JWT token with limited access to read secrets. The Kubernetes Service Account authentication method will provide a token to the clients. Vault will authenticate web applications via JWT token with limited access to read secrets. The Kubernetes Service Account authentication method will provide a token to the clients.

$ kubectl exec -it vault-0 — /bin/sh
$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Configure the Kubernetes authentication with Service Account Token

$ vault write auth/kubernetes/config \
token_reviewer_jwt=”$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)” \
kubernetes_host=”https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

This will configure the token_reviewer_jwt and kubernetes_ca_cert that will mount with containers during the creation in Kubernetes.

Grant Read Access Policy

The secret path secret/webapp/config requires few policy access agreements that will enable clients to read secrets from the path.

$ vault policy write webapp — <<EOF
path “secret/data/webapp/config” {
capabilities = [“read”]
}
EOF
Success! Uploaded policy: webapp

Additional role-based control requires to apply this policy in the authentication engine.
$ vault write auth/kubernetes/role/webapp \
bound_service_account_names=vault \
bound_service_account_namespaces=default \
policies=webapp \
ttl=24h
Success! Data written to: auth/kubernetes/role/webapp

The authentication token remains alive for 24 hours.

Verify Vault Setup from minikube dashboard

Type the following command to navigate to the Kubernetes dashboard in the browser.

$ minikube dashboard
Image for post
Image for post
Vault Workloads
Image for post
Image for post
Vault Container Server Data
Image for post
Image for post
Vault Seal Events
Image for post
Image for post
Vault Stores Service Account Token in Kubernetes Secrets

Read Vault Secrets from Web Application

As we have completed all the steps to create secrets, enable authentication in Vault, configuring read access policy, So now we can apply Deployments in Kubernetes that will mount the VAULT_ADDR, JWT_PATH, and SERVICE_PORT as an env into a container.

$ cat go-cache-poc-app.yaml
$ kubectl apply -f go-cache-poc-app.yaml
Image for post
Image for post
Vault Envs[VAULT_ADDR, JWT_PATH & SERVICE_PORT] mounted into the Web App Pod deployment

Go Application Code

  • The first request will retrieve the client_token using the Service Account Token(JWT).
  • In the second request, the client_token will be used as the “X-Vault-Token” header to send a Get request that will retrieve the secret from Vault.
  1. Vault HTTP Request to receive client_token

API: http://localhost:8200/v1/auth/kubernetes/login

Method: POST

Request Body:

Response:

2. Vault HTTP Request to read the secret

API: http://localhost:8200/v1/secret/data/webapp/config

Method: Get

Headers

X-Vault-Token: <<client_token_received_from_vault_request_1>>

Response:

So, this is the high-level practical implementation of HashiCorp Vault and comparison with the Secrets management in Kubernetes. I hope
the article may have provided you the knowledge regarding the potential benefits of using HashiCorp Vault to store sensitive information over Kubernetes Secret management.

Please also check the project on Kubernetes at my Github.

I hope you find this article useful :)

Thanks

Written by

Working on various software development projects, [Android, Go, Blockchain, Node.js, MongoDB, JavaScript, Kubernetes, Docker]

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store