Apache Hadoop
Hadoop is open source framework for writing and running distributed application. It consists of scale out fault tolerant distribute file system (HDFS) and data processing system (Map Reduce).
Today, I will walk through the steps for set up Hadoop Cluster which involve 2 or more commodity machine. I will be configuring the set up using 2 machines.
Prerequisites:
Network accessible : Machines should be connect through network by either through Ethernet hubs or switch or routers. Therefore, cluster machines should have same subnetting IP address like 192.168.1.x.
Multi Node Hadoop Cluster Setup
1. Set up Hadoop on each machine
Please follow the steps provide in the tutorial and set up single node setup on each machine. Then stop the processes as shown in Step 8 of the tutorial.
2. Change each nodes hosts files to include all machine in cluster .
In my case, I have just 2 machine connected through network with IP Address (192.168.1.1, 192.168.1.2). Therefore, I have included the below 2 lines to file as shown below:
hduser@pooja:~$ sudo vi /etc/hosts
192.168.1.1 master
192.168.1.2 slave1
We will be creating a passwordless ssh between master and all slaves machine in network.
3.1 Master machine ssh set up with itself
We have already set up password less ssh to localhost/itslef when configure Hadoop on each machine. Here, we will just verify if setup is proper.
hduser@pooja:~$ ssh master
The authenticity of host 'master (192.168.1.101)' can't be established.
ECDSA key fingerprint is ad:3c:12:c3:b1:d2:60:a4:8f:76:00:1d:15:b7:f5:41.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.1.101' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64)
* Documentation: https://help.ubuntu.com/
385 packages can be updated.
268 updates are security updates.
New release '16.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Thu Aug 24 13:51:11 2017 from localhost
$
3.2 Master machine ssh set up with slave nodes
hduser@pooja:~$ ssh-copy-id -i /home/hduser/.ssh/id_rsa.pub hduser@slave1
The authenticity of host 'slaves (192.168.1.2)' can't be established.
The ECDSA key fingerprint is: b3: 7d: 41: 89: 03: 15: 04: 1c: 84: e3: d1: 69: 1f: c8: 5d.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hduser@slave1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hduser@slave1'"
and check to make sure that only the key(s) you wanted were added.
Note: In the bold line, we specify the password of hduser@slave1 machine.
3.2.2 Verify the authorization_keys file of slave1 machine
Make sure you have a key enter from master node as shown below.
hduser@prod01:~$ cat .ssh/authorized_keys
ssh-rsa fffA3zwdi0eWSkJvDWzD9du...kSRTRLZbzVY9ahLZNLFz+p1QU3HXuY3tLr hduser@pooja
3.2.3 Confirm passwordless ssh from master machine
hduser@pooja:~$ ssh slave1
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64)
* Documentation: https://help.ubuntu.com/
334 packages can be updated.
224 updates are security updates.
New release '16.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Thu Aug 24 13:50:50 2017 from localhost
$
4. Hadoop Configuration changes
4.1 Changes to masters files
This file specify the list of machine that run the name node and secondary name node (name node will always start on the master node but the secondary name node can run on any slave node if cluster started using start-dfs.sh from the particular slave node). Basically, Secondary namenode merge the fsimage and edit log periodically to keep edit log in size.
In our case we will specify the master machine only.
In our case we will specify the master machine only.
hduser@pooja:~$ vi $HADOOP_HOME/etc/hadoop/masters
4.2 Changes to slave files
This file specify the list of machine that run the datanodes and node masters.
In our case we will specify the master and slave1, if you have more slaves, you can specify them here and can remove master node.
hduser@pooja:~$ vi $HADOOP_HOME/etc/hadoop/slaves
4.3 Changes in core-site.xml for all machine in cluster.
Now, the namenode process will be running on master and not on localhost.
Therefore, we need to change the value of fs-default-name property to hdfs://master:9000 as shown below.
hduser@pooja:~$ vi $HADOOP_HOME/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
</configuration>
Note: Make sure you make changes to core-site.xml in slave nodes as well
4.4 Changes in hdfs-site.xml of all slave nodes (This is optional step)
Remove property "dfs.namenode.dir" as now namenode won't be running on slave machine.
5. Starting hadoop cluster
From the master machine run the below commands
5.1 Start HDFS
hduser@pooja:~$ start-dfs.sh
5.2 Start Yarn
hduser@pooja:~$ start-yarn.sh
5.3 Verify the running process on master
5.3.1 Process runnining on master machine.
hduser@pooja:~$ jps
6821 SecondaryNameNode
7126 NodeManager
6615 DataNode
7628 Jps
6444 NameNode
6990 ResourceManager
5.3.2 Process running on slave node
hduser@prod01:~$ jps
9749 NodeManager
9613 DataNode
9902 Jps
5.3.3 Run the PI Mapreduce job from the hadoop-examples jar.
hduser@pooja:~$ yarn jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 4 1000
5.3.3 Run the PI Mapreduce job from the hadoop-examples jar.
hduser@pooja:~$ yarn jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 4 1000
6. Stop the Cluster
In the master node, stop the processes.
6.1 Stop yarn
hduser@pooja:~$ stop-yarn.sh
stopping yarn daemons
stopping resourcemanager
master: stopping nodemanager
slave1: stopping nodemanager
master: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
slave1: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
no proxyserver to stop
hduser@pooja:~$ stop-dfs.sh
17/08/24 18:42:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namodes on [master]
Master: stopping forgive
master: stopping datanode
slave1: stopping datanode
Stopping secondary namodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
17/08/24 18:42:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I hope you are able to follow my instruction to set up Hadoop Cluster. If still facing issue, I love to address them, please do write your problems !!!
Happy Coding !!!
No comments:
Post a Comment