Hadoop Multi Node Cluster Setup

on


 

Public Address                  Private Address
3.88.49.42 172.31.94.116   - MASTER AND SLAVE
52.91.201.178 172.31.87.192 - SLAVE
18.234.205.75 172.31.85.82 - SLAVE


Generate the ssh key and ad all the node keys in all the nodes under /home/ubuntu/.ssh/authorized_keys ( create this file if not exists ) 

Generate the key (ssh-keygen -t rsa) - 4 times press enter 

node 1 - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWEt73sdIKxCvYNej2oFhfuCArd7SE1P8L+VxdiELLlcrWCsBNVPBD0pPxeIXOuj5gJd8h5uYtzQYdtKJdWY/5nmR5mlPh2sxUFiOX4MGziAly2rmA7NJ8GFbzdfp8smV4IlGx+LdwpotaRp+nVY1xDSQZUpj6rEiCdOT7QWeFcpqtI0yw74EmwIJDlnHO0= ubuntu@ip-172-31-85-82
node 2 - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfknmGNEMgGbUdEMGEuBHZBGG04MsELX8gvhc6dI5wavyaOIQeDKGFaVKpuObjsvc2EoU2ExHJa5cgs6Rdmac5ShkUZHuxF4p29ZWWLq4RlPBaGRkNEGyl/1M98= ubuntu@ip-172-31-87-192
node 3 - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCscwmQoL+C5Tlls+pA5A1mDuhrEcHhUZLdvpHQ6cKUuZz6SAlv9P2ZxkkIEY0WLonLSzaron6GsPJy/jQcuaBKR0fjNep0SLZwu9NuXIwXQnSUBi7KGrtcRcHtS7sjGNOwFkE3WUYU0An01jZVfSGJ91rH5eTly8hqF9jm2LQJmUKzbANJwwenW1XqjjD4+slEGk= ubuntu@ip-172-31-94-116



vi  etc/hadoop/core-site.xml

<property>
<name>fs.default.name</name>
<value>hdfs://172.31.94.116:50000</value>
</property>

vi  etc/hadoop/yarn-site.xml

<property>
<name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>localhost</value>
</property>
<property>
<description>The address of the applications manager interface in the RM.</description>
<name>yarn.resourcemanager.address</name>
<value>localhost:8032</value>
</property>


vi etc/hadoop/hdfs-site.xml

<property>
<name>dfs.namenode.name.dir</name>
<value>/home/ubuntu/hadoop2-dir/namenode-dir</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/ubuntu/hadoop2-dir/datanode-dir</value>
</property>



cp etc/hadoop/mapred-site.xml.template  etc/hadoop/mapred-site.xml

vi etc/hadoop/mapred-site.xml

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

slaves  - ONLY IN MASTER NODE

172.31.94.116
172.31.87.192
172.31.85.82



vi etc/hadoop/hadoop-env.sh
export JAVA_HOME=/home/username/jdk1.8.0_45

vi etc/hadoop/mapred-env.sh
export JAVA_HOME=/home/username/jdk1.8.0_ 45

vi etc/hadoop/yarn-env.sh
export JAVA_HOME=/home/username/jdk1.8.0_45


Foramt name node 

 bin/hadoop namenode -format

Comments

Post a Comment