Showing posts with label Zookeeper. Show all posts
Showing posts with label Zookeeper. Show all posts

Monday, October 11, 2021

Nifi Multi Node Cluster on Docker

Introduction

I will run Apache Nifi multi node cluster on docker. I will use external Zookeeper for it.

Multi Node Cluster

I will use docker-compose to launch a 3 node Apache Nifi cluster. There will be a separate node for zookeeper.

I am using custom Docker file so that I don't download the 1.5GB+ zip file multiple times and slow down the container launch. I have manually downloaded Nifi and Nifi-toolkit zip file for 1.14.0 version.

FROM openjdk:8-jre

ARG UID=1000
ARG GID=1000
ARG NIFI_VERSION=1.14.0

ENV NIFI_BASE_DIR=/opt/nifi
ENV NIFI_HOME ${NIFI_BASE_DIR}/nifi-current
ENV NIFI_TOOLKIT_HOME ${NIFI_BASE_DIR}/nifi-toolkit-current

ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs

ADD sh/ ${NIFI_BASE_DIR}/scripts/
RUN chmod -R +x ${NIFI_BASE_DIR}/scripts/*.sh

# Setup NiFi user and create necessary directories
RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
    && useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
    && mkdir -p ${NIFI_BASE_DIR} \
    && chown -R nifi:nifi ${NIFI_BASE_DIR} \
    && apt-get update \
    && apt-get install -y jq xmlstarlet procps nano vim iputils-ping

USER nifi

# Download, validate, and expand Apache NiFi Toolkit binary.
ADD nifi-toolkit-${NIFI_VERSION}-bin.zip ${NIFI_BASE_DIR}/
RUN unzip ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION} ${NIFI_TOOLKIT_HOME} \
    && ln -s ${NIFI_TOOLKIT_HOME} ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_VERSION}

# Download, validate, and expand Apache NiFi binary.
ADD nifi-${NIFI_VERSION}-bin.zip ${NIFI_BASE_DIR}/
RUN unzip ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
    && rm ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip \
    && mv ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} ${NIFI_HOME} \
    && mkdir -p ${NIFI_HOME}/conf \
    && mkdir -p ${NIFI_HOME}/database_repository \
    && mkdir -p ${NIFI_HOME}/flowfile_repository \
    && mkdir -p ${NIFI_HOME}/content_repository \
    && mkdir -p ${NIFI_HOME}/provenance_repository \
    && mkdir -p ${NIFI_HOME}/state \
    && mkdir -p ${NIFI_LOG_DIR} \
    && ln -s ${NIFI_HOME} ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}

VOLUME ${NIFI_LOG_DIR} \
       ${NIFI_HOME}/conf \
       ${NIFI_HOME}/database_repository \
       ${NIFI_HOME}/flowfile_repository \
       ${NIFI_HOME}/content_repository \
       ${NIFI_HOME}/provenance_repository \
       ${NIFI_HOME}/state

# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
RUN echo "#!/bin/sh\n" > $NIFI_HOME/bin/nifi-env.sh

# Web HTTP(s) & Socket Site-to-Site Ports
EXPOSE 8080 8443 10000 8000

WORKDIR ${NIFI_HOME}

ENTRYPOINT ["../scripts/start.sh"]
You can create a local image using below command: 

$  docker build -t my_nifi -f Dockerfile_manual .
This will generate the image in few seconds.


Th Let's now create a  docker-compose : 

version: "3"
services:
  zk01:
    hostname: zk01
    container_name: zk01
    image: 'bitnami/zookeeper:3.7'
    ports:
      - '2181'
      - '2888'
      - '3888'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_SERVER_ID=1
      - ZOO_SERVERS=0.0.0.0:2888:3888
    networks:
      - nifinet

  nifi01:
    image: my_nifi:latest
    container_name: nifi01
    hostname: nifi01
    ports:
      - 6980:8080
    volumes:
      - /Users/aagarwal/dev/docker/java_cluster/nifi_conf1:/opt/nifi/nifi-current/conf
    networks:
      - nifinet
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zk01:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_SENSITIVE_PROPS_KEY=testpassword

  nifi02:
    image: my_nifi:latest
    container_name: nifi02
    hostname: nifi02
    ports:
      - 6979:8080
    volumes:
      - /Users/aagarwal/dev/docker/java_cluster/nifi_conf2:/opt/nifi/nifi-current/conf
    networks:
      - nifinet
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zk01:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_SENSITIVE_PROPS_KEY=testpassword

  nifi03:
    image: my_nifi:latest
    container_name: nifi03
    hostname: nifi03
    ports:
      - 6978:8080
    volumes:
      - /Users/aagarwal/dev/docker/java_cluster/nifi_conf3:/opt/nifi/nifi-current/conf
    networks:
      - nifinet
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zk01:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_SENSITIVE_PROPS_KEY=testpassword

networks:
  nifinet:
    driver: bridge

Use below command to create the cluster:

$  docker-compose -f docker-compose.yaml up
This will launch the 1 node for zookeeper and 3 nodes for Nifi.


This will take 10-15 minutes for the Nifi nodes for them to be available.

Open the browser  http://localhost:6979/nifi :

The nifi is ready for use.

Happy Coding !!!

Running Multi Node Zookeeper Cluster on Docker

Introduction 

I am working on installing Apache Nifi on multi node cluster. For this I needed a Zookeeper multi node cluster. 

Zookeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. 

Multi Node Cluster

We will run 3 node cluster on docker. We will use below docker-compose.yaml as below:

version: "3"
services:
  zk01:
    hostname: zk01
    container_name: zk01
    image: 'bitnami/zookeeper:3.7'
    ports:
      - '2181'
      - '2888'
      - '3888'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_SERVER_ID=1
      - ZOO_SERVERS=0.0.0.0:2888:3888,zk02:2888:3888,zk03:2888:3888
    networks:
      - zk_net

  zk02:
    hostname: zk02
    container_name: zk02
    image: 'bitnami/zookeeper:3.7'
    ports:
      - '2181'
      - '2888'
      - '3888'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_SERVER_ID=2
      - ZOO_SERVERS=zk01:2888:3888,0.0.0.0:2888:3888,zk03:2888:3888
    networks:
      - zk_net

networks:
  zk_net:
    driver: bridge

I am using bitnami-zookeeper image for it. You can use  latest version instead of 3.7 version if interested. 

This will launch a 3 node zookeeper cluster without publishing/exposing the nodes to the host machines. 

$  docker-compose -f docker-compose.yaml up
This will show the logs of all the 3 nodes as they start running.



After few minutes the log will be like below:



We can check the status by running below command:

$  docker ps
This will show all the 3 instances running.


You can now login to one of the instance/node using below command.

$ docker exec -it zk02 /bin/bash
I have no name!@zk02:/$
This will log you in to the zk02 node.

Testing the Cluster

We will now create some entry on zk02 and it will be reflected/replicated to all the other Zookeeper nodes immediately. We will start zookeeper cli for it.

$ zkCli.sh -server zk02:2181
/opt/bitnami/java/bin/java
Connecting to zk02:2181
2021-10-12 05:56:18,058 [myid:] - INFO  [main:Environment@98] - Client environment:zookeeper.version=3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC
2021-10-12 05:56:18,064 [myid:] - INFO  [main:Environment@98] - Client environment:host.name=zk02
2021-10-12 05:56:18,065 [myid:] - INFO  [main:Environment@98] - Client environment:java.version=11.0.12
2021-10-12 05:56:18,069 [myid:] - INFO  [main:Environment@98] - Client environment:java.vendor=BellSoft

This will open below zk cli command prompt: 

2021-10-12 05:56:18,195 [myid:zk02:2181] - INFO  [main-SendThread(zk02:2181):ClientCnxn$SendThread@1005] - Socket connection established, initiating session, client: /172.28.0.2:43996, server: zk02/172.28.0.2:2181
2021-10-12 05:56:18,249 [myid:zk02:2181] - INFO  [main-SendThread(zk02:2181):ClientCnxn$SendThread@1438] - Session establishment complete on server zk02/172.28.0.2:2181, session id = 0x20010a8acf10000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: zk02:2181(CONNECTED) 0] 
Now run create : 
[zk: zk02:2181(CONNECTED) 0] create /hello world
Created /hello
We can query it as below
[zk: zk02:2181(CONNECTED) 1] get /hello
world
We can also check by logging into other nodes. We will follow the above steps to login and query on other node.

Clean up

We can delete the entry:
[zk: zk02:2181(CONNECTED) 2] delete /hello

Happy Coding !!