Quick reference for kafka
Download as TXT, PDF1 like889 views
This document provides instructions for setting up single and multi-broker Kafka clusters. For a single broker cluster, it describes how to start Zookeeper, the Kafka broker, create and view topics, and add partitions. For a multi-broker cluster, it explains how to configure additional brokers, create a replicated topic, kill a broker, and view that the topic remains accessible.
![********************************************************************************
*****************************************************
Single Broker Cluster
********************************************************************************
*****************************************************
Start Zookeeper [seperate terminal]
---------------
bin/zookeeper-server-start.sh config/zookeeper.properties
Start Kafka broker [seperate terminal]
------------------
bin/kafka-server-start.sh config/server.properties
________________________________________________________________________________
_____________________________________________________
Create Topic
------------
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1
--partitions 1 --topic testtopic
List Topic
----------
bin/kafka-topics.sh --list --zookeeper localhost:2181
Display Topic Information
-------------------------
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic testtopic
Add Partitions to a Topic
-------------------------
bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic testopic
--partitions 2
________________________________________________________________________________
_____________________________________________________
Start Producer [seperate terminal]
--------------
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testtopic
Start Consumer [seperate terminal]
--------------
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-
beginning
________________________________________________________________________________
_____________________________________________________
Note: To understand the flow refer Kafka Tutorial.mp4
********************************************************************************
*****************************************************
Multi Broker Cluster
********************************************************************************
*****************************************************
Create config file for 2 more Kafka Brokers
-------------------------------------------
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
Edit config file 2 Kafka Brokers created above
----------------------------------------------
config/server-1.properties:
broker.id=1
port=9093](https://image.slidesharecdn.com/quickreferenceforkafka-151208155711-lva1-app6892/85/Quick-reference-for-kafka-1-320.jpg)
![log.dir=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
Start 2 newly created Kafka Brokers [seperate terminal]
-----------------------------------
JMX_PORT=9997 bin/kafka-server-start.sh config/server-1.properties &
JMX_PORT=9998 bin/kafka-server-start.sh config/server-2.properties &
Create Topic
------------
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3
--partitions 1 --topic my-replicated-topic
List Topic
----------
bin/kafka-topics.sh --list --zookeeper localhost:2181
Start Producer [seperate terminal]
--------------
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-
replicated-topic
Start Consumer [seperate terminal]
--------------
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning
--topic my-replicated-topic
Kill one of the Kafka Brokers
-----------------------------
pkill -9 -f server-1.properties
List Topic
----------
bin/kafka-topics.sh --list --zookeeper localhost:2181
Start Consumer [seperate terminal]
--------------
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning
--topic my-replicated-topic](https://image.slidesharecdn.com/quickreferenceforkafka-151208155711-lva1-app6892/85/Quick-reference-for-kafka-2-320.jpg)
Quick reference for kafka
- 1. ******************************************************************************** ***************************************************** Single Broker Cluster ******************************************************************************** ***************************************************** Start Zookeeper [seperate terminal] --------------- bin/zookeeper-server-start.sh config/zookeeper.properties Start Kafka broker [seperate terminal] ------------------ bin/kafka-server-start.sh config/server.properties ________________________________________________________________________________ _____________________________________________________ Create Topic ------------ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testtopic List Topic ---------- bin/kafka-topics.sh --list --zookeeper localhost:2181 Display Topic Information ------------------------- bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic testtopic Add Partitions to a Topic ------------------------- bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic testopic --partitions 2 ________________________________________________________________________________ _____________________________________________________ Start Producer [seperate terminal] -------------- bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testtopic Start Consumer [seperate terminal] -------------- bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from- beginning ________________________________________________________________________________ _____________________________________________________ Note: To understand the flow refer Kafka Tutorial.mp4 ******************************************************************************** ***************************************************** Multi Broker Cluster ******************************************************************************** ***************************************************** Create config file for 2 more Kafka Brokers ------------------------------------------- cp config/server.properties config/server-1.properties cp config/server.properties config/server-2.properties Edit config file 2 Kafka Brokers created above ---------------------------------------------- config/server-1.properties: broker.id=1 port=9093
- 2. log.dir=/tmp/kafka-logs-1 config/server-2.properties: broker.id=2 port=9094 log.dir=/tmp/kafka-logs-2 Start 2 newly created Kafka Brokers [seperate terminal] ----------------------------------- JMX_PORT=9997 bin/kafka-server-start.sh config/server-1.properties & JMX_PORT=9998 bin/kafka-server-start.sh config/server-2.properties & Create Topic ------------ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic List Topic ---------- bin/kafka-topics.sh --list --zookeeper localhost:2181 Start Producer [seperate terminal] -------------- bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my- replicated-topic Start Consumer [seperate terminal] -------------- bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic Kill one of the Kafka Brokers ----------------------------- pkill -9 -f server-1.properties List Topic ---------- bin/kafka-topics.sh --list --zookeeper localhost:2181 Start Consumer [seperate terminal] -------------- bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic