Vault supports only SASL_SSL. This means that in addition to selecting either SCRAM-SHA-256 or SCRAM-SHA-512 for authentication, a CA or CA chain must be provided in order to successfully establish the connection between Vault services and Kafka over TLS. This can be either a CA certificate/CA chain or a supported Public CA.

This is optional; skip this step if you are using a Public CA.

If you are using Vault-installed Kafka, create the certificates using the PKI engine or the same way as you would for mTLS. This is so that the brokers have certificates and clients have the CA chain.

If you are using Vault-installed Kafka, you only need to provide a CA certificate. You have two options:

Add a CA certificate to your secrets manager. If you are using HashiCorp Vault, add it at the path secret/{SECRET_PREFIX}/kafka-ca/ca-certificates.crt. If you are using AWS Secrets Manager, add it under {TM_IAM_PREFIX}/{SECRET_PREFIX}/kafka-ca/ca-certificates.crt. Proceed to step 2.

Inject it with the CA Injector webhook. Proceed to step 3.

If Vault has already been installed, replace any existing certificates by running the command:

kubectl exec -it -n tm-system vault-installer -- /deployment-tools/rotate-certs kafka_certs

Run the Vault Installer with the --skip_crown_k8s flag to generate client credentials in the secrets manager.

Discover the client credentials in HashiCorp Vault. This can be achieved by iterating over all secrets ending in /kafka under the secrets path set up for Vault to use, then checking if they have "sasl_scram_username" and "sasl_scram_password" fields.

Register the usernames and passwords found in HashiCorp Vault to Kafka. There are multiple ways of doing this, the most common of which is documented by Confluent.

You need to repeat step 4 and step 5 for every Vault upgrade, as new services requiring SASL-SCRAM credentials to access Kafka may be introduced.

You may wish to keep a historical record of credentials registered to aid with deregistering credentials from Kafka when they no longer appear in the kafka_principals_info.json artifact. It is recommended to only deregister credentials after the Vault installation process is complete to avoid downtime.

Configure the Vault values.yaml file to enable SASL-SCRAM by selecting a SCRAM mechanism, point to the Kafka cluster's SASL_SSL port and enable use of the System CA if a Public CA will be used, using the settings described in Security levels and configuration for Kafka and Vault.

Run the Vault installer to apply the changes.