As with any cloud application, microservices need to have access to sensitive pieces of information such as database passwords, API keys, etc. These values are never committed in the codebase and dummy values are used for local development.

To ensure that these secrets are stored in a cryptographically secure way, we leverage HashiCorp Vault and its Kubernetes auth backend. Developers can specify the secrets that they need access to as mounted volumes in their deployment. These will automatically be fetched at runtime by an init container using the deployment’s service account to authenticate with HashiCorp Vault. This allows us to have fine-grained policies around what secrets each microservice can access, ensure that tokens are short-lived and secrets can be rotated on demand. Unusual activity can easily be flagged based on detailed audit trails.

We also use HashiCorp Vault for internal or corp services. Of these, our main use case is issuance of short-lived (~16h) certificates to access our Kubernetes clusters. Engineers will generally log in once a day (using their LDAP credentials) to gain access to the cluster. These certificates map to RBAC roles, allowing us to easily update permissions across different teams or organisations. Engineers can also request higher-privileged certificates if necessary, though these only last 30 to 60 minutes.