Apache Kafka Tutorial for Beginners | Part 4

Part 4: Command-line Kafka Producer and Consumer

Let us understand how to use Command-line Kafka Producer and Consumer in the Kafka multi-node cluster/single node cluster.

Kafka Command-line Utility:

Navigate to Kafka installation location/directory, if you are not added the Kafka installation directory in Path variable.

cd /opt/ctv/big_data_science/kafka_2.11-2.3.0

Command to list the Topics, Create Topic

bin/kafka-topics.sh --zookeeper localhost:2181 --list

bin/kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1 --topic msgtopic1

Command to Describe Topic

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic msgtopic1

Command-line Kafka Producer

bin/kafka-console-producer.sh --broker-list 34.73.102.250:9092 --topic msgtopic1

Command-line Kafka Consumer

Read the latest messages from Kafka topic

bin/kafka-console-consumer.sh --bootstrap-server 34.73.102.250:9092 --topic msgtopic1

Read all the messages from beginning(older messages also) in the Kafka topic

bin/kafka-console-consumer.sh --bootstrap-server 34.73.102.250:9092 --topic msgtopic1 --from-beginning

How to use Consumer Group

bin/kafka-console-consumer.sh --bootstrap-server 34.73.102.250:9092 --topic msgtopic1 --from-beginning --consumer-property group.id=console-consumer-group

How to increase Partitions for particular Kafka Topic

bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic msgtopic1 --partitions 3

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic msgtopic1

How to check parition assignment & Offsets details

bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --bootstrap-server 34.73.102.250:9092 --describe --group console-consumer-group

Happy Learning !!!