Kafka doesn’t hold data in perpetuity. The admin can configure Kafka to delete the messages for a topic in two ways:

Specify a retention time after which messages are deleted.

Specify the data size to be reached before messages are deleted.

In either scenario, Kafka will not wait for consumers to read messages and delete them when the deletion criteria is met. Data for a partition isn’t a contiguous file. Rather, the data is broken into chunks of files called segments. Each segment can be at most 1GB in size or contain a week’s worth of data, whichever is smaller. The segment currently being written to is known as the active segment and is closed as soon as it reaches 1GB when a new file is opened for writing. Having segments makes deleting stale data much easier than attempting to delete messages in one long contiguous file.

Note that the active segment can never be deleted. This has consequences on the deletion policy. Say you configure Kafka to retain data for only one day. If the topic doesn’t experience high traffic, it is possible that the active segment also contains data for previous days and the data for those days will not be deleted since it is in the active segment. On the other hand, if we configure Kafka to store a week’s worth of data and a new segment gets created for each day, then we shall see the partition contain seven segments most of the time.

Also, note that the broker keeps an open handle to all the segments in all the partitions, including the inactive ones.