How you interact with Kafka topics within Vault will depend on your Kafka hosting solution and internal processes. If a lightweight approach is required for consuming messages, Thought Machine recommends the use of Kafkacat for interacting with Vault DLQ Kafka topics in error scenarios. Instructions for how to download and use Kafkacat can be found here, while providing the -h option when running Kafkacat will provide more information about the available functionality. Remember that wherever you choose to run Kafkacat, it will need access to your Kafka broker.

If you are using Kafkacat to consume and read messages from a DLQ, we recommend:

Running via Docker to ensure the latest version of Kafkacat is used. The command should start with docker run --rm confluentinc/cp-kafkacat kafkacat -b -t -C.

Using the -J option to ensure the headers are displayed along with the message body for the consumed messages. Alternatively you can specify a format via -f which must include the %h header token.

If you wish to only see the headers, you can provide a format that only contains the header token, e.g. -f ‘Headers: <%h>’. This may be useful if the original message contains data the user is not authorised to see (note however that PII data can sometimes occur in the error message, which will always be present in the headers). For a full list of all the format tokens and what data will be displayed, you can run docker run --rm confluentinc/cp-kafkacat kafkacat -h and consult the ‘Format string tokens:’ section. Only the data associated with a given token will be displayed when the -f option is used.

Kafkacat will produce output to Stdout by default.