Tag Archives: Partition

Kafka Server Properties Configuration

Kafka provide server level properties for configuration of  Broker, Socket, Zookeeper, Buffering, Retention etc.

broker.id :  This broker id which is unique integer value in Kafka cluster.


Socket Server Settings :

listeners: default value is PLAINTEXT://:9092  where socket servers listens and if not configured will take from java.net.InetAddress.getCanonicalHostName()

Format: security_protocol://host_name:port


advertised.listeners: Need to set this value if listeners value is not set. Broker will advertise this listener value to producers and consumers.

Example: PLAINTEXT://your.host.name:9092


num.network.threads: Threads handling network requests.


num.io.threads: Number of threads handling I/O for disk.


socket.send.buffer.bytes: Buffer size used by socket server to keep records for sending.


socket.receive.buffer.bytes: Buffer size used by socket server to keep records for sending.


socket.request.max.bytes: max size of request that the socket server will accept.


Log Basics

log.dirs: a comma separated list of directories under which to store log files.


num.partitions: The default number of logs per topic. More partitions allow greater parallelism for consumption, but this will also result in more files across the brokers.


num.recovery.threads.per.data.dir: The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.This value is recommended to be increased for installations with data dirs located in RAID array.


Log Flush Policy

log.flush.interval.messages:  Messages are immediately written to the file system but by default we only fsync() to sync the OS cache lazily. The following configurations control the flush of data to disk.There are a few important trade-offs here:

  1. Durability: Unflushed data may be lost if you are not using replication.
  2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
  3.  Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.

The settings below allow one to configure the flush policy to flush data after a period of time or every N messages (or both). This can be done globally and overridden on a per-topic basis.

The number of messages to accept before forcing a flush of data to disk.


The maximum amount of time a message can sit in a log before we force a flush.


Log Retention Policy

The following configurations control the disposal of log segments. The policy can be set to delete segments after a period of time, or after a given size has accumulated. A segment will be deleted whenever either of these criteria are met. Deletion always happens from the end of the log.

log.retention.hours:The minimum age of a log file to be eligible for deletion


log.retention.bytes:A size-based retention policy for logs. Segments are pruned from the log as long as the remaining segments don’t drop below log.retention.bytes.


log.segment.bytes:The maximum size of a log segment file. When this size is reached a new log segment will be created.


log.retention.check.interval.ms: The interval at which log segments are checked to see if they can be deleted according to the retention policies



zookeeper.connect: Zookeeper connection string is a comma separated host:port pairs, each corresponding to a zk server. e.g. “,,”. We can also append an optional root string to the urls to specify theroot directory for all kafka znodes.


zookeeper.connection.timeout.ms: Timeout in ms for connecting to zookeeper.


Read More on Kafka


Integrate Filebeat, Kafka, Logstash, Elasticsearch and Kibana


Logstash Input and Output to/from Kafka Example

Logstash can take input from Kafka to parse data  and send parsed output to Kafka for streaming to other Application.

Kafka Input Configuration in Logstash

Below are basic configuration for Logstash to consume messages from Logstash. For more information about Logstash, Kafka Input configuration  refer this elasticsearch site Link

input {
   kafka {
       bootstrap_servers => 'KafkaServer:9092'
       topics => ["TopicName"]
       codec => json {}

bootstrap_servers : Default value is “localhost:9092”. Here it takes list of all servers connections in the form of  host1:port1,host2:port2  to establish the initial connection to the cluster. It will connect with other if one server is down.

topics: List of topics to subscribe from where it will consume messages.

Kafka Output Configuration in Logstash

Below are basic configuration for Logstash to publish messages to Logstash. For more information about Logstash, Kafka Output configuration  refer this elasticsearch site Link

output {
        kafka {
        bootstrap_servers => "localhost:9092"
        topic_id => 'TopicName'

bootstrap_servers : Default value is “localhost:9092”. Here it takes list of all servers connections in the form of  host1:port1,host2:port2   and producer will only use it for getting metadata(topics, partitions and replicas) .The socket connections for sending the actual data will be established based on the broker information returned in the metadata.

topic_id: Topic name where messages will publish.

Read More on Kafka


Integrate Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Setup Kafka Cluster for Multi/Distributed Servers/Brokers

For setting up Kafka Cluster for Multi Broker/ Server on single Machine follow below steps:

In below example we will create Kafka cluster with three brokers on single machine. All steps are same as configured for Kafka Cluster with Single Server on same machine additionally created two more file for additional brokers and run it on same Cluster.

Download and Installation

Download Latest version of Kafka from link download , copy it to installation directory and run below command to install it.

tar -zxvf kafka_2.11-

Configuration Changes for Zookeeper and Server

Make below changes  in zookeeper.properties configuration file in config directory.



clientPort is the port where client will connect. By Default port is 2181 if port will update in zookeeper.properties have to update in below server.properties too.

Make below changes  in server.properties configuration file in config directory.



By default server.properties file have above fields with default values.

broker.id : Represents broker unique id by which zookeeper recognize brokers in Kafka Cluster. If Kafka Cluster is having multiple server this broker id will in incremental order for servers.

listeners : Each broker runs on different port by default port for broker is 9092 and can change also.

log.dir:  keep path of logs where Kafka will store steams records. By default point /tmp/kafka-logs.

For more change on property for server.properties file follow  link Kafka Server Properties Configuration.

Multi Server/Broker :

For creating three brokers create two more copy of server.properties configuration file as server1.properties and server2.properties and make below changes in files so that configuration will ready with three brokers on Kafka cluster .

Create copy of server.properties file.

cp config/server.properties config/server1.properties
cp config/server.properties config/server2.properties

make below changes corresponding to each configuration file.





Start Zookeeper and Servers

Run below files as below in Kafka directory

screen -d -m bin/zookeeper-server-start.sh config/zookeeper.properties
screen -d -m bin/kafka-server-start.sh config/server.properties
screen -d -m bin/kafka-server-start.sh config/server1.properties
screen -d -m bin/kafka-server-start.sh config/server2.properties

Check status of Zookeeper & Servers

Below commands will return the port of Zookeeper and Servers processes Id

ps aux | grep zookeeper.properties
ps aux | grep server.properties
ps aux | grep server1.properties
ps aux | grep server2.properties

 Now Kafka is ready to create topic publish and subscribe messages also.

Create a Topic and Check Status

Create topic with user defined name and by passing replication and number partitions for topic. For more info about how partition stores in Kafka Cluster Env follow link for Kafka Introduction and Architecture.

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic multi-test

Created topic "multi-test".

above command will create a topic multi-test with configured partition as 1 and replica as 3.

List of available Topics  in Zookeeper

Run below command to get list of topics

bin/kafka-topics.sh --list --zookeeper localhost:2181


Description of Topic

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic multi-test

Topic:multi-test    PartitionCount:1   ReplicationFactor:3     Configs:
Topic: multi-test   Partition: 0    Leader: 2       Replicas: 2,0,1 Isr: 2,0,1

In above command response .The first line gives a summary of all the partitions, each additional line provide information about one partition. We have only one line for this topic  because  there is one partition.

  • “leader” is the broker responsible for all reads and writes for the given partition. Each broker will be the leader for a randomly selected portion of the partitions.
  • “replicas” is the list of brokers that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
  • “isr” is the set of “in-sync” replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.

In above example broker 2 is the leader for one partition for the topic and replicas of these portion are stored  on broker 0 and  then 1.  If any message publish for topic will store in partition 2 first then in brokers 0 and 1 in sequence.

For all the request for this topic will taken care by Broker 2  and if broker is busy or fail by some reason like shutdown then broker 0 will become lead.  See below example I have stopped broker 2 and run below command again and there lead is showing as 0.

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic multi-test
Topic:multi-test     PartitionCount:1       ReplicationFactor:3     Configs:
Topic: multi-test    Partition: 0    Leader: 0       Replicas: 2,0,1 Isr: 0,1

Publish Messages to Topic

To test topic push your messages to topic by running below command

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic multi-test

Input Messages:
Hi Dear
How r u doing?
Where are u these days?

These message after publish to Topic will retain as logs retention is configured for server even it’s read by consumer or not. To get information about Retention Policy configuration follow link Kafka Server Properties Configuration.

Subscribe Messages by Consumer from Topic

Run below command to get all published messages from multi-test Topic. It will return all these messages from beginning.

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic multi-test

Output Messages:
Hi Dear
How r u doing?
Where are u these days?

Read More on Kafka


Integrate Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Kafka Introduction and Architecture

Kafka is Open source distributed, Steam Processing, Message Broker platform written in Java and Scala developed by Apache Software Foundation.

Kafka is massively use for enterprise infrastructure to process stream data or transaction logs on real time. Kafka provide unified, fault-tolerant, high throughput, low latency platform for dealing real time data feeds.

 Important Points about Kafka :

  • Publish/subscribe messaging system.
  • Robust queue able to handle high volume of data.
  • Work for Online and Offline message consumption.
  • In Kafka Cluster each server/Node work like broker and each broker is responsible for published record and may have zero or more partitions per topic.
  • Each records in partition consists fields key, value and timestamp.
  • Kafka use TCP Protocol to communicate between clients and servers.
  • Kafka provide Producer, Consumer, Streams and Connector java API to publish and consume from topics.

Initial Release: January, 2011

Current Release: 0.10.20

Kafka Cluster Architecture?

 Before going to discuss about Kafka Cluster Architecture . Let’s introduce about terminology use of Kafka that will easy to understand about Architecture and flow.


Broker is stateless instance of Kafka server in Cluster. We define broker by giving unique id for each server instance. Kafka cluster can have multiple broker instances and each broker can handle hundred thousands of reads and write request or TB data per seconds of messages without any performance impact.


Kafka Cluster use Zookeeper for managing and coordinating brokers. Producer and Consumer will get notification if new broker added to cluster or if any fail so that producer and consumer can decide about to point available broker.


Topic is a category to keeps steams of records which are publish to it. Topic can have zero, one or many consumers for reading data. We can create our own topics by application and manually also.

Topic stored data on portitions and distribute over servers based on number of partition configure per topic and available brokers.


A partition stores records in sequential orders and will continually to append it. Each record in partition having sequential id number called as offset. Individual Log partition allows the records to scale up to available single server capacity.

How Topic will partitioned for  brokers/servers/nodes?

Suppose, Need to create a topic with N partitions for Kafka Cluster having M brokers.

If (M==N) : Each broker will have one partition.

If (M>N): First available N broker will take one partition for each.

If (M<N): Some brokers may have more than one partitions.

Kafka cluster will retain these partition  as configured for retention policy in server.properties file while it’s consumed or not by default it’s configure for two months and we can modify based on our storage capacity. Kafka performance don’t impact based on data size because it read and write data based on offset values.

 Kafka Cluster Architecture with Multi distrubuted servers

 Detail about above Kafka Cluster for Multi/distributed servers.

Kafka Cluster: Having three servers  and each server is having corresponding  brokers as id 1, 2 and 3.

Zookeeper: Zookeeper runs over Kafka Cluster which keeps detail of availability of brokers and update producers and consumers.

Brokers: 1,2 and 3 which are having topics as T1, T2 and T3 stored in partitions.

Topics: Topics T1, T2 is partitioned as 3 and distributed over the servers 1, 2 and 3 while Topic 3 is partitioned as 1 that is stored in server 3 only.

Partition: Each partitioned for topic is having different no of records from offset 0 to some value where 0 represents oldest records.

Producers: APP1, APP2 and APP3 is writing to different topics on T1, T2 and T3 which are created by Applications or manually.

Consumers: Topic T3 is consumed by applications APP5 and APP6 while Topic T1 is consumed by APP4 and T2 is consumed by APP5 only.  One topic can be consumed by multiple APPs.

How Kafka Cluster Flow works for Producers and Consumers?

 I will divide above architectures in two parts as “Producer to Kafka Cluster” and “Kafka Cluster to Consumer” because producer and consumer runs parallel and independent of each others.

Producer to Kafka Cluster

  • Create Topic manually or by application with configuration for portioned and replica.
  • Producer will connect with Kafka Cluster with topic name . Kafka cluster will check in Zookeeper for available broker and send broker id to Producer .
  • Producer will publish message to available broker to store in sequential order to partition. If anything got change with Kafka cluster servers like add or fail server Zookeeper updated to Producer.
  • If replication is configured for topic will keep copy of partition on another server for fault tolerant.

Kafka Cluster to Consumer

  • Consumer will point to Topic on Kafka Cluster  as required by Application.
  • Consumer will subscribe records from Topic based on required offset value (like beginning, now or from last).
  • If consumer wants records from now  Zookeeper will send offset value to Consumer to start read records from brokers partitions.
  • If  required offset is not exist in Broker partition where Consumer was pointing reading data then Zookeeper will return available broker id  with partition detail to Consumer.
  • If in between one broker is down during Consumer is reading records from it. Zookeeper will send will send available broker id  with partition detail to Consumer.

Kafka Cluster with Single Server: Will create no of partition on the same server per topic.

Kafka Cluster with Multi Server/Distributed: Topic partitions logs are distributed on all the servers in the Kafka cluster and each server is able to handle data and requests for share partitions. If replication is configure servers will keep number of copies of partition logs distributed to servers for fault tolerance.

Kafka Cluster Load balance for multi-server or distributed servers?

For each Topic partitions log having one server/broker as “leader” while others are followers (if multi server/distributed). Leader handles all read and write requests from producer and consumer while followers make replica of lead server partition. If somehow leader fail or server down for one machine then one of followers will become leader and rest server will follower. For more detail go to Kafka Cluster with multi server on same machine.

Read More on Kafka


Integrate Filebeat, Kafka, Logstash, Elasticsearch and Kibana