Secure Kafka with SSL(Shell Script Automation)
Securing Kafka with SSL (using Java KeyStore, JKS) involves several steps both on the server (Kafka broker) side and the client (producer/consumer) side. Here’s a detailed guide:
1. Generate SSL Certificates and Keystores
Step 1: Create a Certificate Authority (CA)
The CA will sign the certificates for your Kafka brokers and clients.
# Generate CA private key
openssl genpkey -algorithm RSA -out ca-key.pem
# Generate CA certificate
openssl req -new -x509 -key ca-key.pem -out ca-cert.pem -days 365 \
-subj "/CN=MyKafkaCA"
Step 2: Create Keystores and Certificates for Kafka Brokers
You need to create a keystore for each Kafka broker.
- Generate a keystore:
keytool -genkeypair -alias kafka-broker -keyalg RSA -keystore kafka.broker.keystore.jks \
-storepass broker-keystore-password -keypass broker-key-password \
-dname "CN=kafka-broker, OU=IT, O=MyOrg, L=City, ST=State, C=US"
- Create a Certificate Signing Request (CSR):
keytool -certreq -alias kafka-broker -keystore kafka.broker.keystore.jks \
-file kafka-broker.csr -storepass broker-keystore-password
- Sign the CSR with the CA:
openssl x509 -req -CA ca-cert.pem -CAkey ca-key.pem -in kafka-broker.csr \
-out kafka-broker-cert.pem -days 365 -CAcreateserial -passin pass:ca-key-password
- Import the CA certificate and the signed certificate into the keystore:
# Import CA certificate
keytool -importcert -alias ca-cert -file ca-cert.pem -keystore kafka.broker.keystore.jks \
-storepass broker-keystore-password -noprompt
# Import broker certificate
keytool -importcert -alias kafka-broker -file kafka-broker-cert.pem \
-keystore kafka.broker.keystore.jks -storepass broker-keystore-password -noprompt
Step 3: Create Truststore for Kafka Brokers
The truststore contains the CA certificate.
keytool -importcert -alias ca-cert -file ca-cert.pem -keystore kafka.broker.truststore.jks \
-storepass broker-truststore-password -noprompt
Step 4: Repeat for Each Kafka Broker
If you have multiple brokers, repeat the steps to generate unique keystores and certificates for each one.
2. Configure Kafka Brokers
Add the following configurations to each broker’s server.properties
file:
# Enable SSL for inter-broker communication and client connections
listeners=SSL://kafka-broker-hostname:9093
# Specify keystore and truststore locations and passwords
ssl.keystore.location=/path/to/kafka.broker.keystore.jks
ssl.keystore.password=broker-keystore-password
ssl.key.password=broker-key-password
ssl.truststore.location=/path/to/kafka.broker.truststore.jks
ssl.truststore.password=broker-truststore-password
ssl.client.auth=required
# Enable SSL encryption
security.inter.broker.protocol=SSL
3. Create Keystore and Truststore for Kafka Clients
Step 1: Generate a Keystore for the Client
keytool -genkeypair -alias kafka-client -keyalg RSA -keystore kafka.client.keystore.jks \
-storepass client-keystore-password -keypass client-key-password \
-dname "CN=kafka-client, OU=IT, O=MyOrg, L=City, ST=State, C=US"
Step 2: Create a CSR and Sign it with the CA
keytool -certreq -alias kafka-client -keystore kafka.client.keystore.jks \
-file kafka-client.csr -storepass client-keystore-password
openssl x509 -req -CA ca-cert.pem -CAkey ca-key.pem -in kafka-client.csr \
-out kafka-client-cert.pem -days 365 -CAcreateserial -passin pass:ca-key-password
Step 3: Import the CA Certificate and Signed Certificate into the Client Keystore
keytool -importcert -alias ca-cert -file ca-cert.pem -keystore kafka.client.keystore.jks \
-storepass client-keystore-password -noprompt
keytool -importcert -alias kafka-client -file kafka-client-cert.pem \
-keystore kafka.client.keystore.jks -storepass client-keystore-password -noprompt
Step 4: Create a Truststore for the Client
keytool -importcert -alias ca-cert -file ca-cert.pem -keystore kafka.client.truststore.jks \
-storepass client-truststore-password -noprompt
4. Configure Kafka Clients
For both producers and consumers, you need to configure SSL settings in your client code.
const { Kafka } = require('kafkajs');
const fs = require('fs');
const kafka = new Kafka({
clientId: 'your-client-id',
brokers: ['kafka-broker-hostname:9093'],
ssl: {
rejectUnauthorized: true, // Optionally, set to false for self-signed certs
ca: [fs.readFileSync('/path/to/ca-cert.pem')],
key: fs.readFileSync('/path/to/kafka.client.keystore.jks'),
cert: fs.readFileSync('/path/to/kafka-client-cert.pem'),
},
});
5. Test the Setup
-
Start the Kafka Broker: Make sure the broker is configured correctly with SSL and start it.
-
Run the Kafka Client: Ensure your producer/consumer is configured with SSL and attempt to connect to the Kafka broker. If everything is configured correctly, the client should be able to produce and consume messages securely over SSL.
6. Additional Security Configurations
- SASL Authentication: You can combine SSL with SASL for additional security (e.g.,
SASL_SSL
). - Authorization: Implement Access Control Lists (ACLs) to restrict access to specific topics or consumer groups.
By following these steps, you will have a Kafka cluster secured with SSL, ensuring encrypted communication between Kafka brokers and clients.
Shell Script for cert generation
#!/bin/bash
# Variables
DAYS_VALID=365
KEYSTORE_PASSWORD="keystore-password"
TRUSTSTORE_PASSWORD="truststore-password"
CA_CERT="ca-cert.pem"
CA_KEY="ca-key.pem"
CA_SUBJECT="/CN=MyKafkaCA"
BROKER_ALIAS="kafka-broker"
CLIENT_ALIAS="kafka-client"
BROKER_SUBJECT="CN=kafka-broker"
CLIENT_SUBJECT="CN=kafka-client"
# Paths
BROKER_KEYSTORE="kafka.broker.keystore.jks"
BROKER_TRUSTSTORE="kafka.broker.truststore.jks"
CLIENT_KEYSTORE="kafka.client.keystore.jks"
CLIENT_TRUSTSTORE="kafka.client.truststore.jks"
BROKER_CSR="kafka-broker.csr"
CLIENT_CSR="kafka-client.csr"
BROKER_CERT="kafka-broker-cert.pem"
CLIENT_CERT="kafka-client-cert.pem"
# Step 1: Create a Certificate Authority (CA)
echo "Creating Certificate Authority (CA)..."
openssl genpkey -algorithm RSA -out $CA_KEY
openssl req -new -x509 -key $CA_KEY -out $CA_CERT -days $DAYS_VALID -subj $CA_SUBJECT
# Step 2: Create Keystore and CSR for Kafka Broker
echo "Creating keystore and CSR for Kafka Broker..."
keytool -genkeypair -alias $BROKER_ALIAS -keyalg RSA -keystore $BROKER_KEYSTORE \
-storepass $KEYSTORE_PASSWORD -keypass $KEYSTORE_PASSWORD -dname $BROKER_SUBJECT
keytool -certreq -alias $BROKER_ALIAS -keystore $BROKER_KEYSTORE \
-file $BROKER_CSR -storepass $KEYSTORE_PASSWORD
# Step 3: Sign Broker CSR with CA
echo "Signing Broker CSR with CA..."
openssl x509 -req -CA $CA_CERT -CAkey $CA_KEY -in $BROKER_CSR \
-out $BROKER_CERT -days $DAYS_VALID -CAcreateserial
# Step 4: Import CA Cert and Broker Cert into Broker Keystore
echo "Importing CA and Broker certificates into Broker keystore..."
keytool -importcert -alias ca-cert -file $CA_CERT -keystore $BROKER_KEYSTORE \
-storepass $KEYSTORE_PASSWORD -noprompt
keytool -importcert -alias $BROKER_ALIAS -file $BROKER_CERT \
-keystore $BROKER_KEYSTORE -storepass $KEYSTORE_PASSWORD -noprompt
# Step 5: Create Truststore for Kafka Broker
echo "Creating Truststore for Kafka Broker..."
keytool -importcert -alias ca-cert -file $CA_CERT -keystore $BROKER_TRUSTSTORE \
-storepass $TRUSTSTORE_PASSWORD -noprompt
# Step 6: Create Keystore and CSR for Kafka Client
echo "Creating keystore and CSR for Kafka Client..."
keytool -genkeypair -alias $CLIENT_ALIAS -keyalg RSA -keystore $CLIENT_KEYSTORE \
-storepass $KEYSTORE_PASSWORD -keypass $KEYSTORE_PASSWORD -dname $CLIENT_SUBJECT
keytool -certreq -alias $CLIENT_ALIAS -keystore $CLIENT_KEYSTORE \
-file $CLIENT_CSR -storepass $KEYSTORE_PASSWORD
# Step 7: Sign Client CSR with CA
echo "Signing Client CSR with CA..."
openssl x509 -req -CA $CA_CERT -CAkey $CA_KEY -in $CLIENT_CSR \
-out $CLIENT_CERT -days $DAYS_VALID -CAcreateserial
# Step 8: Import CA Cert and Client Cert into Client Keystore
echo "Importing CA and Client certificates into Client keystore..."
keytool -importcert -alias ca-cert -file $CA_CERT -keystore $CLIENT_KEYSTORE \
-storepass $KEYSTORE_PASSWORD -noprompt
keytool -importcert -alias $CLIENT_ALIAS -file $CLIENT_CERT \
-keystore $CLIENT_KEYSTORE -storepass $KEYSTORE_PASSWORD -noprompt
# Step 9: Create Truststore for Kafka Client
echo "Creating Truststore for Kafka Client..."
keytool -importcert -alias ca-cert -file $CA_CERT -keystore $CLIENT_TRUSTSTORE \
-storepass $TRUSTSTORE_PASSWORD -noprompt
echo "SSL certificates generation completed!"
Test:
# bin/zookeeper-server-start.sh config/zookeeper.properties
# bin/kafka-server-start.sh /Users/larry/IdeaProjects/pkslow-samples/other/kafka-ssl-security/server.properties
Code
Please check the code in GitHub.