Node Kafka Producer And Consumer Example
To set up a Node.js project where producer.js
sends messages to Kafka using a cron job, and consumer.js
consumes those messages, follow these steps:
1. Initialize the Node.js Project
First, create a new directory for your project and initialize it with npm
:
2. Install Required Dependencies
You will need kafkajs
for interacting with Kafka and node-cron
for scheduling the cron job:
3. Create the Producer Script (producer.js
)
This script will use node-cron
to schedule a job that sends a message to Kafka at regular intervals:
4. Create the Consumer Script (consumer.js
)
This script will consume messages from the Kafka topic:
5. Running the Producer and Consumer
You can run these scripts independently in separate terminal sessions:
6. Kafka Setup
Ensure that Kafka is running and accessible. If you’re running Kafka locally, you should have it set up with at least one broker, and the topic (my-topic
) should be created.
7. Explanation
-
producer.js
: This script usesnode-cron
to schedule a job that sends a message to the Kafka topic every minute. ThesendMessage
function is responsible for connecting to Kafka, sending the message, and then disconnecting. -
consumer.js
: This script connects to Kafka as a consumer, subscribes to the specified topic, and logs any messages it receives.
This setup allows you to independently run the producer and consumer, with the producer automatically sending messages at intervals defined by the cron job.
Code
Please check the code in GitHub.