Nifi Multi Node Cluster on Docker

Introduction

I will run Apache Nifi multi node cluster on docker. I will use external Zookeeper for it.

Multi Node Cluster

I will use docker-compose to launch a 3 node Apache Nifi cluster. There will be a separate node for zookeeper.

I am using custom Docker file so that I don't download the 1.5GB+ zip file multiple times and slow down the container launch. I have manually downloaded Nifi and Nifi-toolkit zip file for 1.14.0 version.

FROM openjdk:8-jre

ARG UID=1000
ARG GID=1000
ARG NIFI_VERSION=1.14.0

ENV NIFI_BASE_DIR=/opt/nifi
ENV NIFI_HOME ${NIFI_BASE_DIR}/nifi-current
ENV NIFI_TOOLKIT_HOME ${NIFI_BASE_DIR}/nifi-toolkit-current

ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs

ADD sh/ ${NIFI_BASE_DIR}/scripts/
RUN chmod -R +x ${NIFI_BASE_DIR}/scripts/*.sh

# Setup NiFi user and create necessary directories
RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
    && useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
    && mkdir -p ${NIFI_BASE_DIR} \
    && chown -R nifi:nifi ${NIFI_BASE_DIR} \
    && apt-get update \
    && apt-get install -y jq xmlstarlet procps nano vim iputils-ping

USER nifi

# Download, validate, and expand Apache NiFi Toolkit binary.
ADD nifi-toolkit-${NIFI_VERSION}-bin.zip ${NIFI_BASE_DIR}/
RUN unzip ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} ${NIFI_TOOLKIT_HOME} \
    && ln -s ${NIFI_TOOLKIT_HOME} ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}

# Download, validate, and expand Apache NiFi binary.
ADD nifi-${NIFI_VERSION}-bin.zip ${NIFI_BASE_DIR}/
RUN unzip ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} ${NIFI_HOME} \
    && mkdir -p ${NIFI_HOME}/conf \
    && mkdir -p ${NIFI_HOME}/database_repository \
    && mkdir -p ${NIFI_HOME}/flowfile_repository \
    && mkdir -p ${NIFI_HOME}/content_repository \
    && mkdir -p ${NIFI_HOME}/provenance_repository \
    && mkdir -p ${NIFI_HOME}/state \
    && mkdir -p ${NIFI_LOG_DIR} \
    && ln -s ${NIFI_HOME} ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}

VOLUME ${NIFI_LOG_DIR} \
       ${NIFI_HOME}/conf \
       ${NIFI_HOME}/database_repository \
       ${NIFI_HOME}/flowfile_repository \
       ${NIFI_HOME}/content_repository \
       ${NIFI_HOME}/provenance_repository \
       ${NIFI_HOME}/state

# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
RUN echo "#!/bin/sh\n" > $NIFI_HOME/bin/nifi-env.sh

# Web HTTP(s) & Socket Site-to-Site Ports
EXPOSE 8080 8443 10000 8000

WORKDIR ${NIFI_HOME}

ENTRYPOINT ["../scripts/start.sh"]

You can create a local image using below command: 

$  docker build -t my_nifi -f Dockerfile_manual .

This will generate the image in few seconds.

Th Let's now create a  docker-compose : 

version: "3"
services:
  zk01:
    hostname: zk01
    container_name: zk01
    image: 'bitnami/zookeeper:3.7'
    ports:
      - '2181'
      - '2888'
      - '3888'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_SERVER_ID=1
      - ZOO_SERVERS=0.0.0.0:2888:3888
    networks:
      - nifinet

  nifi01:
    image: my_nifi:latest
    container_name: nifi01
    hostname: nifi01
    ports:
      - 6980:8080
    volumes:
      - /Users/aagarwal/dev/docker/java_cluster/nifi_conf1:/opt/nifi/nifi-current/conf
    networks:
      - nifinet
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zk01:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_SENSITIVE_PROPS_KEY=testpassword

  nifi02:
    image: my_nifi:latest
    container_name: nifi02
    hostname: nifi02
    ports:
      - 6979:8080
    volumes:
      - /Users/aagarwal/dev/docker/java_cluster/nifi_conf2:/opt/nifi/nifi-current/conf
    networks:
      - nifinet
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zk01:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_SENSITIVE_PROPS_KEY=testpassword

  nifi03:
    image: my_nifi:latest
    container_name: nifi03
    hostname: nifi03
    ports:
      - 6978:8080
    volumes:
      - /Users/aagarwal/dev/docker/java_cluster/nifi_conf3:/opt/nifi/nifi-current/conf
    networks:
      - nifinet
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zk01:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_SENSITIVE_PROPS_KEY=testpassword

networks:
  nifinet:
    driver: bridge

Use below command to create the cluster:

$  docker-compose -f docker-compose.yaml up

This will launch the 1 node for zookeeper and 3 nodes for Nifi.

This will take 10-15 minutes for the Nifi nodes for them to be available.

Open the browser  http://localhost:6979/nifi :

The nifi is ready for use.

Happy Coding !!!

Running Multi Node Zookeeper Cluster on Docker

Introduction 

I am working on installing Apache Nifi on multi node cluster. For this I needed a Zookeeper multi node cluster. 

Zookeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. 

Multi Node Cluster

We will run 3 node cluster on docker. We will use below docker-compose.yaml as below:

version: "3"
services:
  zk01:
    hostname: zk01
    container_name: zk01
    image: 'bitnami/zookeeper:3.7'
    ports:
      - '2181'
      - '2888'
      - '3888'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_SERVER_ID=1
      - ZOO_SERVERS=0.0.0.0:2888:3888,zk02:2888:3888,zk03:2888:3888
    networks:
      - zk_net

  zk02:
    hostname: zk02
    container_name: zk02
    image: 'bitnami/zookeeper:3.7'
    ports:
      - '2181'
      - '2888'
      - '3888'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_SERVER_ID=2
      - ZOO_SERVERS=zk01:2888:3888,0.0.0.0:2888:3888,zk03:2888:3888
    networks:
      - zk_net

networks:
  zk_net:
    driver: bridge

I am using bitnami-zookeeper image for it. You can use  latest version instead of 3.7 version if interested. 

This will launch a 3 node zookeeper cluster without publishing/exposing the nodes to the host machines. 

$  docker-compose -f docker-compose.yaml up

This will show the logs of all the 3 nodes as they start running.

After few minutes the log will be like below:

We can check the status by running below command:

$  docker ps

This will show all the 3 instances running.

You can now login to one of the instance/node using below command.

$ docker exec -it zk02 /bin/bash
I have no name!@zk02:/$

This will log you in to the zk02 node.

Testing the Cluster

We will now create some entry on zk02 and it will be reflected/replicated to all the other Zookeeper nodes immediately. We will start zookeeper cli for it.

$ zkCli.sh -server zk02:2181
/opt/bitnami/java/bin/java
Connecting to zk02:2181
2021-10-12 05:56:18,058 [myid:] - INFO  [main:Environment@98] - Client environment:zookeeper.version=3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC
2021-10-12 05:56:18,064 [myid:] - INFO  [main:Environment@98] - Client environment:host.name=zk02
2021-10-12 05:56:18,065 [myid:] - INFO  [main:Environment@98] - Client environment:java.version=11.0.12
2021-10-12 05:56:18,069 [myid:] - INFO  [main:Environment@98] - Client environment:java.vendor=BellSoft

This will open below zk cli command prompt: 

2021-10-12 05:56:18,195 [myid:zk02:2181] - INFO  [main-SendThread(zk02:2181):ClientCnxn$SendThread@1005] - Socket connection established, initiating session, client: /172.28.0.2:43996, server: zk02/172.28.0.2:2181
2021-10-12 05:56:18,249 [myid:zk02:2181] - INFO  [main-SendThread(zk02:2181):ClientCnxn$SendThread@1438] - Session establishment complete on server zk02/172.28.0.2:2181, session id = 0x20010a8acf10000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: zk02:2181(CONNECTED) 0] 

Now run create : 

[zk: zk02:2181(CONNECTED) 0] create /hello world
Created /hello

We can query it as below

[zk: zk02:2181(CONNECTED) 1] get /hello
world

We can also check by logging into other nodes. We will follow the above steps to login and query on other node.

Clean up

We can delete the entry:

[zk: zk02:2181(CONNECTED) 2] delete /hello

Happy Coding !!

Docker in simple terms

Docker is the open source software build on concept of virtualization. Using it, minimal portable operating system can be installed in its containers.Like on servers, we can install docker software and then can run Windows OS in one or two container and CentOS on other containers.Docker light weight isolated containers can be easily managed and can be stop,start,kill or restart at any point of time.The containers internally will shared the same linux instance and uses resources/namespace secure isolation therefore better option than hardware virtualization where in we specify the resource allocation (like Hard disk size/RAM usage) for each VMs at time of creation.

Docker container is created using read only template called Docker images. For eg. an image can contain Ubuntu operating system with Apache tomcat installed. You can either download the existing image or build your own images.All the docker images public and private are stored in docker registry called Docker Hub.

Lets briefly go through simple steps to follow to setup docker on a server machine:

a. Download Docker

On window 7,I install boot2docker (https://github.com/boot2docker/windows-installer/releases->docker-install.exe) and run the exe, it will install Docker client and virtual box. You run the Docker client tool.

You can even go through the steps mentioned at http://docs.docker.com/installation/windows/.

On CentOS machine,follow the below steps:

1. Make sure yum package is up-to date
$ sudo yum update

2. Run the installation script
$ curl -sSL https://get.docker.com/ | sh

Output:
[sudo] password for xxx:
+ sudo -E sh -c 'sleep 3; yum -y -q install docker-engine'
warning: rpmts_HdrFromFdno: Header V4 RSA/SHA1 Signature, key ID 2c52609d: NOKEY
Importing GPG key 0x2C52609D:
Userid: "Docker Release Tool (releasedocker) <docker@docker.com>"
From : https://yum.dockerproject.org/gpg

Remember that you will have to log out and back in for this to take effect!

3. Start the docker service
$ sudo service docker start

4. Test if successfully installed docker:
On docker client tool/bash terminal, run the below command:
$sudo docker run hello-world

Output:
Unable to find image 'hello-world:latest' locally
latest: Pulling from hello-world

535020c3e8ad: Pull complete
af340544ed62: Pull complete
Digest: sha256:a68868bfe696c00866942e8f5ca39e3e31b79c1e50feaee4ce5e28df2f051d5c
Status: Downloaded newer image for hello-world:latest

Hello from Docker.
This message shows that your installation appears to be working correctly.

This command should download the image "hello-world" from registry(Docker Hub) and then create container and run it and then produce the output "Hello from Docker".

b. Managing the docker images
Docker image can be pulled from registry (Docker Hub) or can be build using Docker file(shown in step e below).

Pulling ubuntu image from registry:
$ sudo docker pull ubuntu

Output:
latest: Pulling from ubuntu
d3a1f33e8a5a: Pull complete
c22013c84729: Pull complete
d74508fb6632: Pull complete
91e54dfb1179: Already exists
ubuntu:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:fde8a8814702c18bb1f39b3bd91a2f82a8e428b1b4e39d1963c5d14418da8fba
Status: Downloaded newer image for ubuntu:latest

List all the downloaded docker images on server:
$ sudo docker images

Output:
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ubuntu latest 91e54dfb1179 2 weeks ago 188.3 MB
hello-world latest af340544ed62 4 weeks ago 960 MB

Delete docker images from machine:
Delete the ubuntu docker image whose Image Id (91e54dfb1179)

$ sudo docker rmi 91e54dfb1179

Output:
Untagged: ubuntu:latest
Deleted: 91e54dfb11794fad694460162bf0cb0a4fa710cfa3f60979c177d920813e267c
Deleted: d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82
Deleted: c22013c8472965aa5b62559f2b540cd440716ef149756e7b958a1b2aba421e87
Deleted: d3a1f33e8a5a513092f01bb7eb1c2abf4d711e5105390a3fe1ae2248cfde1391

Delete all docker images from machine:
If we need to remove all downloaded docker images from machine

$ sudo docker rmi $(sudo docker images -q)

Output:
Untagged: ubuntu:latest
Deleted: 91e54dfb11794fad694460162bf0cb0a4fa710cfa3f60979c177d920813e267c
Deleted: d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82
Deleted: c22013c8472965aa5b62559f2b540cd440716ef149756e7b958a1b2aba421e87
Deleted: d3a1f33e8a5a513092f01bb7eb1c2abf4d711e5105390a3fe1ae2248cfde1391
Untagged: hello-world:latest
Deleted: af340544ed62de0680f441c71fa1a80cb084678fed42bae393e543faea3a572c
Deleted: 535020c3e8add9d6bb06e5ac15a261e73d9b213d62fb2c14d752b8e189b2b912

d. Creating the Docker Container
Docker container can be run from image or start existing docker container

Started a Docker Container using Ubuntu image ( or can even use Image ID) in interactive bash shell (-i) and pseudo-TTY connected (-t) to container's stdin. Note: In below we code,we specify Docker to connect Docker Container bash shell.

$ sudo docker run -it ubuntu /bin/bash
Output:
root@0ba57ee1a68a:/# uname -a
Linux 0ba57ee1a68a 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Once we stop the container, they can be restarted but Docker Container connect command cannot be modified.Note: 0ba57ee1a68a is Container ID.

$ sudo docker ps -a
Output:
[sudo] password for xxx:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0ba57ee1a68a 91e54dfb1179 "/bin/bash" 10 minutes ago Exited (0) 15 seconds ago sad_cori

$ sudo docker restart 0ba57ee1a68a

0ba57ee1a68a

$ sudo docker attach 0ba57ee1a68a
Output:
root@0ba57ee1a68a:/# uname -a
Linux 0ba57ee1a68a 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

e. Managing the Docker containers
Docker containers can be listed, stop,remove,killed.

List all Docker containers:
$ sudo docker ps -a
Output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0ba57ee1a68a 91e54dfb1179 "/bin/bash" 24 minutes ago Exited (1) 15 seconds ago

Stop running Docker Container:

$ sudo docker stop 0ba57ee1a68a
Output:
0ba57ee1a68a

Remove the Docker Container:
$ sudo docker rm 0ba57ee1a68a
Output:
0ba57ee1a68a

Remove all Docker Containers:
$ sudo docker rm $(sudo docker ps -a -q)
Output:
3d8ae1bb867e

b. Building the Docker images (from docker file)
Docker can build image by reading the instructions from the Dockerfile (it is a text file that contain all commands need to build a image).
Created a simple dockerfile where in instruction to install Java 8 on Ubuntu machine and then boot with bash shell.
Dockerfile attached below:

FROM ubuntu
MAINTAINER Pooja Gupta <pooja.gupta@jbksoft.com>

# setup Java
RUN RUNLEVEL=1 DEBIAN_FRONTEND=noninteractive apt-get install -y wget
RUN mkdir /opt/java
RUN wget -O /opt/java/jdk-8u25-linux-x64.tar.gz --no-cookies --no-check-certificate --header \
"Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \
"http://download.oracle.com/otn-pub/java/jdk/8u25-b17/jdk-8u25-linux-x64.tar.gz"

# change dir to Java installation dir

WORKDIR /opt/java/

RUN tar -zxf jdk-8u25-linux-x64.tar.gz

# setup environment variables

RUN update-alternatives --install /usr/bin/javac javac /opt/java/jdk1.8.0_25/bin/javac 100

RUN update-alternatives --install /usr/bin/java java /opt/java/jdk1.8.0_25/bin/java 100

RUN update-alternatives --display java

RUN java -version

# Expose the ports we're interested in
EXPOSE 8080 9990

# Set the default command to run on boot
CMD ["/bin/bash"]

Now, we can build the image and then run the container using image.Note:ubuntu-with-java is the repository name of the image created and Image Id is b37cc178d4c7 for me.

$ sudo docker build -t ubuntu-with-java8 .

Output:
.....
Removing intermediate container 182805e359f2
Step 11 : EXPOSE 8080 9990
---> Running in e4c2eebbb475
---> 346fd2a5f340
Removing intermediate container e4c2eebbb475
Step 12 : CMD /bin/bash
---> Running in 8372b7ae20e7
---> b37cc178d4c7
Removing intermediate container 8372b7ae20e7
Successfully built b37cc178d4c7

$ sudo docker run -it ubuntu-with-java8
Output:
root@66a414638a22:/opt/java#
root@66a414638a22:/opt/java# java -version
java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)