Breaking News: Grepper is joining You.com. Read the official announcement!

How are partitions distributed in an Apache Kafka cluster?

Add Answer

Pragya Keshap answered on February 6, 2023 Popularity 3/10 Helpfulness 1/10

answer How are partitions distributed in an Apache Kafka cluster?

How are partitions distributed in an Apache Kafka cluster?

Comment

Partitions of a Kafka topic are distributed across servers in a Kafka cluster.

Each Kafka server handles the data and requests with its share of partitions.

Partitions can be replicated across multiple servers to ensure fault tolerance.

Every partition has one Kafka server that plays the role of a leader for that partition.

The leader takes care of all the read and write requests for that particular partition.

A leader can have zero or more followers.

The leader and follower relationship is such that the followers passively replicate the leader.

In the case where the leader fails, one of the followers can take on the role of the leader.

This is how the leader and follower concept works in a Kafka cluster.

public class Sender {

	public static void main(String[] args) {
		AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(Config.class);
		context.getBean(Sender.class).send("test", 42);
	}

	private final KafkaTemplate<Integer, String> template;

	public Sender(KafkaTemplate<Integer, String> template) {
		this.template = template;
	}

	public void send(String toSend, int key) {
		this.template.send("topic1", key, toSend);
	}

}

public class Listener {

    @KafkaListener(id = "listen1", topics = "topic1")
    public void listen1(String in) {
        System.out.println(in);
    }

}

@Configuration
@EnableKafka
public class Config {

    @Bean
    ConcurrentKafkaListenerContainerFactory<Integer, String>
                        kafkaListenerContainerFactory(ConsumerFactory<Integer, String> consumerFactory) {
        ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
                                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
        return factory;
    }

    @Bean
    public ConsumerFactory<Integer, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerProps());
    }

    private Map<String, Object> consumerProps() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        // ...
        return props;
    }

    @Bean
    public Sender sender(KafkaTemplate<Integer, String> template) {
        return new Sender(template);
    }

    @Bean
    public Listener listener() {
        return new Listener();
    }

    @Bean
    public ProducerFactory<Integer, String> producerFactory() {
        return new DefaultKafkaProducerFactory<>(senderProps());
    }

    private Map<String, Object> senderProps() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.LINGER_MS_CONFIG, 10);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        //...
        return props;
    }

    @Bean
    public KafkaTemplate<Integer, String> kafkaTemplate(ProducerFactory<Integer, String> producerFactory) {
        return new KafkaTemplate<Integer, String>(producerFactory);
    }

}

https://medium.com/event-driven-utopia/understanding-kafka-topic-partitions-ae40f80552e8

Popularity 3/10 Helpfulness 1/10 Language java

Source: Grepper

Tags: distributed java

Link to this answer
Share Copy Link

Contributed on Feb 06 2023

Pragya Keshap

0 Answers Avg Quality 2/10

How are partitions distributed in an Apache Kafka cluster?

Contents

More Related Answers

How are partitions distributed in an Apache Kafka cluster?

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.