For setting up Kafka Cluster for Single Broker . Follow below steps :
Download and Installation
Download Latest version of Kafka from link download , copy it to installation directory and run below command to install it.
tar -zxvf kafka_2.11-0.10.0.0
Configuration Changes for Zookeeper and Server
Make below changes in zookeeper.properties configuration file in config directory.
clientPort is the port where client will connect. By Default port is 2181 if port will update in zookeeper.properties have to update in below server.properties too.
Make below changes in server.properties configuration file in config directory.
config/server.properties: broker.id=0 listeners=PLAINTEXT://:9092 log.dir=/tmp/kafka-logs zookeeper.connect=localhost:2181
By default server.properties file have above fields with default values.
broker.id : Represents broker unique id by which zookeeper recognize brokers in Kafka Cluster. If Kafka Cluster is having multiple server this broker id will in incremental order for servers.
listeners : Each broker runs on different port by default port for broker is 9092 and can change also.
log.dir: keep path of logs where Kafka will store steams records. By default point /tmp/kafka-logs.
For more change on property for server.properties file follow link Kafka Server Properties Configuration.
Start Zookeeper and Server
Run below files as below in Kafka directory
screen -d -m bin/zookeeper-server-start.sh config/zookeeper.properties screen -d -m bin/kafka-server-start.sh config/server.properties
Check status of Zookeeper & Server
Below commands will return the port of Zookeeper and Server processes
ps aux | grep zookeeper.properties ps aux | grep server.properties
Now Kafka is ready to create topic publish and subscribe messages also.
Create a Topic and Check Status
Create topic with user defined name and by passing replication and number partitions for topic. For more info about how partition stores in Kafka Cluster Env follow link for Kafka Introduction and Architecture.
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test Result: Created topic "test".
above command will create topic test with configured partition as 1 and replica as 1.
List of available Topics in Zookeeper
Run below command to get list of topics
bin/kafka-topics.sh --list --zookeeper localhost:2181 Result: test
Description of Topic
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test Result: Topic:test PartitionCount:1 ReplicationFactor:1 Configs: Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
In above command response .The first line gives a summary of all the partitions, each additional line provide information about one partition. We have only one line for this topic because there is one partition.
- “leader” is the broker responsible for all reads and writes for the given partition. Each broker will be the leader for a randomly selected portion of the partitions.
- “replicas” is the list of brokers that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
- “isr” is the set of “in-sync” replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.
In above example broker 1 is the leader for one partition for the topic. Topic is not having any replica and is on server 0 because of one server on cluster.
Publish Messages to Topic
To test topic push your messages to topic by running below command
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test Input Messages: Hi Dear How r u doing? Where are u these days?
These message after publish to Topic will retain as logs retention is configured for server even it’s read by consumer or not. To get information about Retention Policy configuration follow link Kafka Server Properties Configuration.
Subscribe Messages by Consumer from Topic
Run below command to get all published messages from test Topic. It will return all these messages from beginning.
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic test Output Messages: Hi Dear How r u doing? Where are u these days?
Read More on Kafka
- Kafka Introduction and Architecture
- Kafka Server Properties Configuration
- Setup Kafka Cluster for Multi/Distributed Servers/Brokers
- Integrate Java with Kafka
- Integrate Logstash with Kafka
- Integrate Filebeat with Kafka
- Kafka and Zookeeper Common Issues