Breaking News: Grepper is joining You.com. Read the official announcement!

What should be the kafka consumer concurrency level for maximum concurrency and throughput for a single topic with 5 partitions?

Pragya Keshap answered on February 21, 2023 Popularity 1/10 Helpfulness 1/10

answer What should be the kafka consumer concurrency level for maximum concurrency and throughput for a single topic with 5 partitions?

What should be the kafka consumer concurrency level for maximum concurrency and throughput for a single topic with 5 partitions?

Comment

We can have at most 5 different threads, one per each partition. Additional threads will be idle.

List<PartitionInfo> partitionInfos = null;
partitionInfos = consumer.partitionsFor("topic"); 

if (partitionInfos != null) {
    for (PartitionInfo partition : partitionInfos)
        partitions.add(new TopicPartition(partition.topic(),
            partition.partition()));
    consumer.assign(partitions); 

    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(1000);

        for (ConsumerRecord<String, String> record: records) {
            System.out.printf("topic = %s, partition = %s, offset = %d,
                customer = %s, country = %s\n",
                record.topic(), record.partition(), record.offset(),
                record.key(), record.value());
        }
        consumer.commitSync();
    }
}

https://learning.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html

Popularity 1/10 Helpfulness 1/10 Language java

Source: Grepper

Tags: concurrency consumer throughput whatever

Link to this answer
Share Copy Link

Contributed on Feb 21 2023

Pragya Keshap

0 Answers Avg Quality 2/10

What should be the kafka consumer concurrency level for maximum concurrency and throughput for a single topic with 5 partitions?

Contents

More Related Answers

What should be the kafka consumer concurrency level for maximum concurrency and throughput for a single topic with 5 partitions?

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.