Apache Kafka
Apache Kafka is a distributed messaging system using components such as Publisher/Subscriber/Broker. It is popular due to the fact that system is design to store message in fault tolerant way and also its support to build real-time streaming data pipeline and applications.
In this message broker system, we create a topic(category) and list of producers which send message on a topic to brokers and then message from brokers are either broadcast or parallel processed by list of consumer registered to that topic.In this, the communication between producer and consumer are performed using TCP protocol.
ZooKeeper also integral part of the system, which help in co-ordination of distributed brokers and consumers.
This is the simple working model as shown below.
In this tutorial, I will discuss the steps for installing simple Kafka messaging system.
Installing Apache Kafka
Step 1: Create user (Optional Step)
[code language="java"]
[root@localhost ~]$ sudo useradd kafka
[root@localhost ~]$ sudo passwd kafka
Changing password for user kafka.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@localhost ~]$ su - kafka
[/code]
Step 2: Download tar file
Download the latest code from the link or wget the code (version 2.11) as shown below.
[code language="java"]
[kafka@localhost ~]$ wget http://apache.osuosl.org/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
--2016-12-19 13:10:48-- http://wget/
....
Connecting to apache.osuosl.org (apache.osuosl.org)|64.50.236.52|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34373824 (33M) [application/x-gzip]
Saving to: ‘kafka_2.11-0.10.1.0.tgz’
100%[======================================>] 34,373,824 2.46MB/s in 13s
2016-12-19 13:11:01 (2.60 MB/s) - ‘kafka_2.11-0.10.1.0.tgz’ saved [34373824/34373824]
[/code]
Step 3: Untar the file
Untar the file using below command
[code language="java"]
[kafka@localhost ~]$ tar -xvf kafka_2.11-0.10.1.0.tgz
[kafka@localhost ~]$ cd kafka_2.11-0.10.1.0/
[/code]
The code base has some important directory as shown below
Step 4: Start the server
Kafka server require Zookeeper, so first start it in as shown below:
[code language="java"]
# Run the zookeeper in background process on port 2181.
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/zookeeper-server-start.sh config/zookeeper.properties &
[2] 29678
[1] Exit 143 nohup bin/zookeeper-server-start.sh config/zookeeper.properties > logs/zookeeper_kafka.out
nohup: ignoring input and redirecting stderr to stdout
#Verify if it process is running
[kafka@localhost kafka_2.11-0.10.1.0]$ jps
29678 QuorumPeerMain
29987 Jps
[/code]
Now, start the kafka server as shown below
[code language="java"]
#Run the kafka server in background
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-server-start.sh config/server.properties &
[3] 30228
...
[2016-12-19 14:46:39,543] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,48] in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
# Verify if server running
[kafka@localhost kafka_2.11-0.10.1.0]$ jps
29678 QuorumPeerMain
30501 Jps
30228 Kafka
[/code]
Step 5: Create a topic
Let create a topic "demo" with single partition and single replica as shown below
[code language="java"]
#Create topic "demo"
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic demo
Created topic "demo".
#Verify if topic exists
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-topics.sh --list --zookeeper localhost:2181
demo
[/code]
Step 6: Create a producer
Kafka comes with a command line producer that can take input from file or from keyboard input.
[code language="java"]
#Run the producer to send message on topic demo
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo
[/code]
Step 7: Create a consumer
Kafka comes with command line consumer that show the message on console
[code language="java"]
#Receive message on consumer
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo
[/code]
Hope you were able to setup the basic kafka messaging system. Please let me know i you face any issues while configuring.
Happy Coding!!!!
Apache Kafka is a distributed messaging system using components such as Publisher/Subscriber/Broker. It is popular due to the fact that system is design to store message in fault tolerant way and also its support to build real-time streaming data pipeline and applications.
In this message broker system, we create a topic(category) and list of producers which send message on a topic to brokers and then message from brokers are either broadcast or parallel processed by list of consumer registered to that topic.In this, the communication between producer and consumer are performed using TCP protocol.
ZooKeeper also integral part of the system, which help in co-ordination of distributed brokers and consumers.
This is the simple working model as shown below.
In this tutorial, I will discuss the steps for installing simple Kafka messaging system.
Installing Apache Kafka
Step 1: Create user (Optional Step)
[code language="java"]
[root@localhost ~]$ sudo useradd kafka
[root@localhost ~]$ sudo passwd kafka
Changing password for user kafka.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@localhost ~]$ su - kafka
[/code]
Step 2: Download tar file
Download the latest code from the link or wget the code (version 2.11) as shown below.
[code language="java"]
[kafka@localhost ~]$ wget http://apache.osuosl.org/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
--2016-12-19 13:10:48-- http://wget/
....
Connecting to apache.osuosl.org (apache.osuosl.org)|64.50.236.52|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34373824 (33M) [application/x-gzip]
Saving to: ‘kafka_2.11-0.10.1.0.tgz’
100%[======================================>] 34,373,824 2.46MB/s in 13s
2016-12-19 13:11:01 (2.60 MB/s) - ‘kafka_2.11-0.10.1.0.tgz’ saved [34373824/34373824]
[/code]
Step 3: Untar the file
Untar the file using below command
[code language="java"]
[kafka@localhost ~]$ tar -xvf kafka_2.11-0.10.1.0.tgz
[kafka@localhost ~]$ cd kafka_2.11-0.10.1.0/
[/code]
The code base has some important directory as shown below
| Folder | Usage |
| bin | Contains daemons to start Server, Zoopkeper, Publisher, Subscriber or create topics. |
| config | Contains properties file for each components |
| libs | Contain internal jars required by system |
Step 4: Start the server
Kafka server require Zookeeper, so first start it in as shown below:
[code language="java"]
# Run the zookeeper in background process on port 2181.
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/zookeeper-server-start.sh config/zookeeper.properties &
[2] 29678
[1] Exit 143 nohup bin/zookeeper-server-start.sh config/zookeeper.properties > logs/zookeeper_kafka.out
nohup: ignoring input and redirecting stderr to stdout
#Verify if it process is running
[kafka@localhost kafka_2.11-0.10.1.0]$ jps
29678 QuorumPeerMain
29987 Jps
[/code]
Now, start the kafka server as shown below
[code language="java"]
#Run the kafka server in background
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-server-start.sh config/server.properties &
[3] 30228
...
[2016-12-19 14:46:39,543] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,48] in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
# Verify if server running
[kafka@localhost kafka_2.11-0.10.1.0]$ jps
29678 QuorumPeerMain
30501 Jps
30228 Kafka
[/code]
Step 5: Create a topic
Let create a topic "demo" with single partition and single replica as shown below
[code language="java"]
#Create topic "demo"
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic demo
Created topic "demo".
#Verify if topic exists
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-topics.sh --list --zookeeper localhost:2181
demo
[/code]
Step 6: Create a producer
Kafka comes with a command line producer that can take input from file or from keyboard input.
[code language="java"]
#Run the producer to send message on topic demo
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo
[/code]
Step 7: Create a consumer
Kafka comes with command line consumer that show the message on console
[code language="java"]
#Receive message on consumer
[kafka@localhost kafka_2.11-0.10.1.0]$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo
[/code]
Hope you were able to setup the basic kafka messaging system. Please let me know i you face any issues while configuring.
Happy Coding!!!!
