Module 3.3: Apache Kafka for Message Layer

Prerequisite

• Java 1.8
• Scala 2.12.8
• Apache Kafka

Walk-through

In this article, we are going to discuss about how we are going to use Apache Kafka as message layer in the "Real-Time Meetup.com RSVP Message Processing Application".

Apache Kafka is open source framework for publish-subscribe(pub-sub) messaging system. We can build Apache Kafka Server(broker) as single node cluster or multi-node cluster.

Create a kafka topic called "meetuprsvptopic" to store the stream data(RSVP JSON Message) as string message.

kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic meetuprsvptopic

Command to list topics in the kafka server/broker

kafka-topics.sh --list --zookeeper localhost:2181

Command to run the command-line the kafka producer

kafka-console-producer.sh --broker-list localhost:9092 --topic meetuprsvptopic

Command to run the command-line the kafka consumer

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic meetuprsvptopic

Summary

In this article, we have successfully learned how to use Apache Kafka as message layer in the "Real-Time Meetup.com RSVP Message Processing Application". Please go through all these steps and provide your feedback and post your queries/doubts if you have. Thank you. Appreciated.

You can build this project and run on your Spark and Hadoop cluster or reach to us for FREE Spark and Hadoop VM.

Request for FREE Spark and Hadoop VM

Click here for more details about FREE Spark and Hadoop VM

Happy Learning !!!