Hadoop2.6.0自動(dòng)化部署腳本（一）
作者：王森豐
最近自己寫了一個(gè)Hadoop自動(dòng)化部署腳本，包括Hadoop集群自動(dòng)化部署腳本和Hadoop增加單節(jié)點(diǎn)自動(dòng)化部署腳本。需要快速部署Hadoop集群的童鞋可以使用該腳本。這些腳本我在用5臺虛擬機(jī)進(jìn)行了測試，如果在使用中還有bug，歡迎指出。本文主要介紹Hadoop集群自動(dòng)化部署腳本，安裝的Hadoop版本為2.6.0。
1 概述
最近自己寫了一個(gè)Hadoop自動(dòng)化部署腳本，包括Hadoop集群自動(dòng)化部署腳本和Hadoop增加單節(jié)點(diǎn)自動(dòng)化部署腳本。需要快速部署Hadoop集群的童鞋可以使用該腳本。這些腳本我在用5臺虛擬機(jī)進(jìn)行了測試，如果在使用中還有bug，歡迎指出。本文主要介紹Hadoop集群自動(dòng)化部署腳本，安裝的Hadoop版本為2.6.0。
2 依賴
安裝Hadoop2.6.0集群需要依賴JDK和Zookeeper。本文安裝的JDK版本為jdk-7u60-linux-x64，Zookeeper版本為zookeeper-3.4.6。
3 各文件及配置說明
該部署腳本由兩部分構(gòu)成：root用戶下執(zhí)行的腳本和Hadoop啟動(dòng)用戶下執(zhí)行的腳本。這些腳本都只需要在一臺服務(wù)器上執(zhí)行即可，執(zhí)行腳本的服務(wù)器作為Hadoop的Master服務(wù)器。下面分別進(jìn)行說明。
3.1 root腳本說明
root腳本的目錄結(jié)構(gòu)如下：
- conf — 配置文件目錄
- init.conf
- expect — expect腳本目錄
- password.expect
- scp.expect
- otherInstall.expect
- file — 安裝文件目錄
- hadoop-2.6.0.tar.gz
- jdk-7u60-linux-x64.tar.gz
- zookeeper-3.4.6.tar.gz
- installRoot.sh — 腳本執(zhí)行文件
3.1.1 conf目錄
該目錄下的init.conf文件為root執(zhí)行腳本使用的配置文件，在執(zhí)行腳本之前需要對該配置文件進(jìn)行修改。文件內(nèi)容如下：
- #jdk file and version
- JDK_FILE_TAR=jdk-7u60-linux-x64.tar.gz
- #jdk unpack name
- JDK_FILE=jdk1.7.0_60
- #java home
- JAVAHOME=/usr/java
- #Whether install the package for dependence,0 means no,1 means yes
- IF_INSTALL_PACKAGE=1
- #host conf
- ALLHOST="hadoop1master hadoop1masterha hadoop1slave1 hadoop1slave2 hadoop1slave3"
- ALLIP="192.168.0.180 192.168.0.184 192.168.0.181 192.168.0.182 192.168.0.183"
- #zookeeper conf
- ZOOKEEPER_TAR=zookeeper-3.4.6.tar.gz
- ZOOKEEPERHOME=/usr/local/zookeeper-3.4.6
- SLAVELIST="hadoop1slave1 hadoop1slave2 hadoop1slave3"
- #hadoop conf
- HADOOP_TAR=hadoop-2.6.0.tar.gz
- HADOOPHOME=/usr/local/hadoop-2.6.0
- HADOOP_USER=hadoop2
- HADOOP_PASSWORD=hadoop2
- #root conf: $MASTER_HA $SLAVE1 $SLAVE2 $SLAVE3
- ROOT_PASSWORD="hadoop hadoop hadoop hadoop"
下面是個(gè)別參數(shù)的解釋及注意事項(xiàng)：
- ALLHOST為Hadoop集群各個(gè)服務(wù)器的hostname，使用空格分隔;ALLIP為Hadoop集群各個(gè)服務(wù)器的ip地址，使用空格分隔。要求ALLHOST和ALLIP要一一對應(yīng)。
- SLAVELIST為zookeeper集群部署的服務(wù)器的hostname。
- ROOT_PASSWORD為除了Master服務(wù)器以外的其他服務(wù)器root用戶的密碼，使用逗號隔開。(在實(shí)際情況下，可能各個(gè)服務(wù)器的root密碼并不相同。)
3.1.2 expect目錄
該目錄下包含password.expect、scp.expect、otherInstall.expect三個(gè)文件。password.expect用來設(shè)置hadoop啟動(dòng)用戶的密碼;scp.expect用來遠(yuǎn)程傳輸文件;otherInstall.expect用來遠(yuǎn)程執(zhí)行其他服務(wù)器上的installRoot.sh。這三個(gè)文件都在installRoot.sh中被調(diào)用。
password.expect文件內(nèi)容如下：
- #!/usr/bin/expect -f
- set user [lindex $argv 0]
- set password [lindex $argv 1]
- spawn passwd $user
- expect "New password:"
- send "$password\r"
- expect "Retype new password:"
- send "$password\r"
- expect eof
其中argv 0和argv 1都是在installRoot.sh腳本中進(jìn)行傳值的。其他兩個(gè)文件argv *也是這樣傳值的。
scp.expect文件內(nèi)容如下：
- #!/usr/bin/expect -f
- # set dir, host, user, password
- set dir [lindex $argv 0]
- set host [lindex $argv 1]
- set user [lindex $argv 2]
- set password [lindex $argv 3]
- set timeout -1
- spawn scp -r $dir $user@$host:/root/
- expect {
- "(yes/no)?"
- {
- send "yes\n"
- expect "*assword:" { send "$password\n"}
- }
- "*assword:"
- {
- send "$password\n"
- }
- }
- expect eof
otherInstall.expect文件內(nèi)容如下：
- #!/usr/bin/expect -f
- # set dir, host, user, password
- set dir [lindex $argv 0]
- set name [lindex $argv 1]
- set host [lindex $argv 2]
- set user [lindex $argv 3]
- set password [lindex $argv 4]
- set timeout -1
- spawn ssh -q $user@$host "$dir/$name"
- expect {
- "(yes/no)?"
- {
- send "yes\n"
- expect "*assword:" { send "$password\n"}
- }
- "*assword:"
- {
- send "$password\n"
- }
- }
- expect eof
3.1.3 file目錄
這里就是安裝Hadoop集群及其依賴所需的安裝包。
3.1.4 installRoot.sh腳本
該腳本是在root用戶下需要執(zhí)行的腳本，文件內(nèi)容如下：
- #!/bin/bash
- if [ $USER != "root" ]; then
- echo "[ERROR]:Must run as root"; exit 1
- fi
- # Get absolute path and name of this shell
- readonly PROGDIR=$(readlink -m $(dirname $0))
- readonly PROGNAME=$(basename $0)
- hostname=`hostname`
- source /etc/profile
- # import init.conf
- source $PROGDIR/conf/init.conf
- echo "install start..."
- # install package for dependence
- if [ $IF_INSTALL_PACKAGE -eq 1 ]; then
- yum -y install expect >/dev/null 2>&1
- echo "expect install successful."
- # yum install openssh-clients #scp
- fi
- #stop iptables or open ports, now stop iptables
- service iptables stop
- chkconfig iptables off
- FF_INFO=`service iptables status`
- if [ -n "`echo $FF_INFO | grep "Firewall is not running"`" ]; then
- echo "Firewall is already stop."
- else
- echo "[ERROR]:Failed to shut down the firewall.Exit shell."
- exit 1
- fi
- #stop selinux
- setenforce 0
- SL_INFO=`getenforce`
- if [ $SL_INFO == "Permissive" -o $SL_INFO == "disabled" ]; then
- echo "selinux is already stop."
- else
- echo "[ERROR]:Failed to shut down the selinux. Exit shell."
- exit 1
- fi
- #host config
- hostArr=( $ALLHOST )
- IpArr=( $ALLIP )
- for (( i=0; i <= ${#hostArr[@]}; i++ ))
- {
- if [ -z "`grep "${hostArr[i]}" /etc/hosts`" -o -z "`grep "${IpArr[i]}" /etc/hosts`" ]; then
- echo "${IpArr[i]} ${hostArr[i]}" >> /etc/hosts
- fi
- }
- #user config
- groupadd $HADOOP_USER && useradd -g $HADOOP_USER $HADOOP_USER && $PROGDIR/expect/password.expect $HADOOP_USER $HADOOP_PASSWORD >/dev/null 2>&1
- # check jdk
- checkOpenJDK=`rpm -qa | grep java`
- # already install openJDK ,uninstall
- if [ -n "$checkOpenJDK" ]; then
- rpm -e --nodeps $checkOpenJDK
- echo "uninstall openJDK successful"
- fi
- # A way of exception handling. `java -version` perform error then perform after ||.
- `java -version` || (
- [ ! -d $JAVAHOME ] && ( mkdir $JAVAHOME )
- tar -zxf $PROGDIR/file/$JDK_FILE_TAR -C $JAVAHOME
- echo "export JAVA_HOME=$JAVAHOME/$JDK_FILE" >> /etc/profile
- echo 'export JAVA_BIN=$JAVA_HOME/bin' >> /etc/profile
- echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile
- echo 'export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar' >> /etc/profile
- echo 'export JAVA_HOME JAVA_BIN PATH CLASSPATH' >> /etc/profile
- echo "sun jdk done"
- )
- # check zookeeper
- slaveArr=( $SLAVELIST )
- if [[ "${slaveArr[@]}" =~ $hostname ]]; then
- `zkServer.sh status` || [ -d $ZOOKEEPERHOME ] || (
- tar -zxf $PROGDIR/file/$ZOOKEEPER_TAR -C /usr/local/
- chown -R $HADOOP_USER:$HADOOP_USER $ZOOKEEPERHOME
- echo "export ZOOKEEPER_HOME=$ZOOKEEPERHOME" >> /etc/profile
- echo 'PATH=$PATH:$ZOOKEEPER_HOME/bin' >> /etc/profile
- echo "zookeeper done"
- )
- fi
- # check hadoop2
- `hadoop version` || [ -d $HADOOPHOME ] || (
- tar -zxf $PROGDIR/file/$HADOOP_TAR -C /usr/local/
- chown -R $HADOOP_USER:$HADOOP_USER $HADOOPHOME
- echo "export HADOOP_HOME=$HADOOPHOME" >> /etc/profile
- echo 'PATH=$PATH:$HADOOP_HOME/bin' >> /etc/profile
- echo 'HADOOP_HOME_WARN_SUPPRESS=1' >> /etc/profile
- echo "hadoop2 done"
- )
- source /etc/profile
- #ssh config
- sed -i "s/^#RSAAuthentication\ yes/RSAAuthentication\ yes/g" /etc/ssh/sshd_config
- sed -i "s/^#PubkeyAuthentication\ yes/PubkeyAuthentication\ yes/g" /etc/ssh/sshd_config
- sed -i "s/^#AuthorizedKeysFile/AuthorizedKeysFile/g" /etc/ssh/sshd_config
- sed -i "s/^GSSAPIAuthentication\ yes/GSSAPIAuthentication\ no/g" /etc/ssh/sshd_config
- sed -i "s/^#UseDNS\ yes/UseDNS\ no/g" /etc/ssh/sshd_config
- service sshd restart
- # install other servers
- rootPasswdArr=( $ROOT_PASSWORD )
- if [ $hostname == ${hostArr[0]} ]; then
- i=0
- for node in $ALLHOST; do
- if [ $hostname == $node ]; then
- echo "this server, do nothing"
- else
- # cope install dir to other server
- $PROGDIR/expect/scp.expect $PROGDIR $node $USER ${rootPasswdArr[$i]}
- $PROGDIR/expect/otherInstall.expect $PROGDIR $PROGNAME $node $USER ${rootPasswdArr[$i]}
- i=$(($i+1)) #i++
- echo $node" install successful."
- fi
- done
- # Let the environment variables take effect
- su - root
- fi
這個(gè)腳本主要干了下面幾件事：
- 如果在配置文件中設(shè)置了IF_INSTALL_PACKAGE=1，則安裝expect，默認(rèn)是安裝expect。如果服務(wù)器上已經(jīng)有了expect，則可以設(shè)置IF_INSTALL_PACKAGE=0。
- 關(guān)閉防火墻，停止selinux。
- 將Hadoop集群的各個(gè)機(jī)器host及ip對應(yīng)關(guān)系寫到/etc/hosts文件中。
- 新建Hadoop啟動(dòng)用戶及用戶組。
- 安裝jdk、zookeeper、hadoop并設(shè)置環(huán)境變量。
- 修改ssh配置文件/etc/ssh/sshd_config。
- 如果判斷執(zhí)行腳本的機(jī)器是Master機(jī)器，則拷貝本機(jī)的root腳本到其他機(jī)器上并執(zhí)行。
注意：在執(zhí)行該腳本之前，需要確保Hadoop集群安裝的各個(gè)服務(wù)器上能夠執(zhí)行scp命令，如果不能執(zhí)行，需要在各個(gè)服務(wù)器上安裝openssh-clients，執(zhí)行腳本為：yum –y install openssh-clients。
3.2 hadoop腳本說明
hadoop腳本的目錄結(jié)構(gòu)如下：
- bin — 腳本目錄
- config_hadoop.sh
- config_ssh.sh
- config_zookeeper.sh
- ssh_nopassword.expect
- start_all.sh
- conf — 配置文件目錄
- init.conf
- template — 配置文件模板目錄
- core-site.xml
- hadoop-env.sh
- hdfs-site.xml
- mapred-site.xml
- mountTable.xml
- myid
- slaves
- yarn-env.sh
- yarn-site.xml
- zoo.cfg
- installCluster.sh — 腳本執(zhí)行文件
3.2.1 bin腳本目錄
該目錄中包含installCluster.sh腳本中調(diào)用的所有腳本，下面一一說明。
3.2.1.1 config_hadoop.sh
該腳本主要是創(chuàng)建Hadoop所需目錄，以及配置文件的配置，其中的參數(shù)均在init.conf中。
- #!/bin/bash
- # Get absolute path of this shell
- readonly PROGDIR=$(readlink -m $(dirname $0))
- # import init.conf
- source $PROGDIR/../conf/init.conf
- for node in $ALL; do
- # create dirs
- ssh -q $HADOOP_USER@$node "
- mkdir -p $HADOOPDIR_CONF/hadoop2/namedir
- mkdir -p $HADOOPDIR_CONF/hadoop2/datadir
- mkdir -p $HADOOPDIR_CONF/hadoop2/jndir
- mkdir -p $HADOOPDIR_CONF/hadoop2/tmp
- mkdir -p $HADOOPDIR_CONF/hadoop2/hadoopmrsys
- mkdir -p $HADOOPDIR_CONF/hadoop2/hadoopmrlocal
- mkdir -p $HADOOPDIR_CONF/hadoop2/nodemanagerlocal
- mkdir -p $HADOOPDIR_CONF/hadoop2/nodemanagerlogs
- "
- echo "$node create dir done."
- for conffile in $CONF_FILE; do
- # copy
- scp $PROGDIR/../template/$conffile $HADOOP_USER@$node:$HADOOPHOME/etc/hadoop
- # update
- ssh -q $HADOOP_USER@$node "
- sed -i 's%MASTER_HOST%${MASTER_HOST}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%MASTER_HA_HOST%${MASTER_HA_HOST}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%SLAVE1%${SLAVE1}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%SLAVE2%${SLAVE2}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%SLAVE3%${SLAVE3}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%HDFS_CLUSTER_NAME%${HDFS_CLUSTER_NAME}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%VIRTUAL_PATH%${VIRTUAL_PATH}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%DFS_NAMESERVICES%${DFS_NAMESERVICES}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%NAMENODE1_NAME%${NAMENODE1_NAME}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%NAMENODE2_NAME%${NAMENODE2_NAME}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%NAMENODE_JOURNAL%${NAMENODE_JOURNAL}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%HADOOPDIR_CONF%${HADOOPDIR_CONF}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%ZOOKEEPER_ADDRESS%${ZOOKEEPER_ADDRESS}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%YARN1_NAME%${YARN1_NAME}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%YARN2_NAME%${YARN2_NAME}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%HADOOPHOME%${HADOOPHOME}%g' $HADOOPHOME/etc/hadoop/$conffile
- sed -i 's%JAVAHOME%${JAVAHOME}%g' $HADOOPHOME/etc/hadoop/$conffile
- # update yarn.resourcemanager.ha.id for yarn_ha
- if [ $conffile == 'yarn-site.xml' ]; then
- if [ $node == $MASTER_HA_HOST ]; then
- sed -i 's%YARN_ID%${YARN2_NAME}%g' $HADOOPHOME/etc/hadoop/$conffile
- else
- sed -i 's%YARN_ID%${YARN1_NAME}%g' $HADOOPHOME/etc/hadoop/$conffile
- fi
- fi
- "
- done
- echo "$node copy hadoop template done."
- done
3.2.1.2 config_ssh.sh和ssh_nopassword.expect
這兩個(gè)文件是配置ssh無密碼登錄的，ssh_nopassword.expect被config_ssh.sh調(diào)用。
config_ssh.sh文件如下：
- #!/bin/bash
- # Get absolute path of this shell
- readonly PROGDIR=$(readlink -m $(dirname $0))
- # import init.conf
- source $PROGDIR/../conf/init.conf
- # Get hostname
- HOSTNAME=`hostname`
- # Config ssh nopassword login
- echo "Config ssh on master"
- # If the directory "~/.ssh" is not exist, then execute mkdir and chmod
- [ ! -d ~/.ssh ] && ( mkdir ~/.ssh ) && ( chmod 700 ~/.ssh )
- # If the file "~/.ssh/id_rsa.pub" is not exist, then execute ssh-keygen and chmod
- [ ! -f ~/.ssh/id_rsa.pub ] && ( yes|ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa ) && ( chmod 600 ~/.ssh/id_rsa.pub )
- echo "Config ssh nopassword for cluster"
- # For all node, including master and slaves
- for node in $ALL; do
- # execute bin/ssh_nopassword.expect
- $PROGDIR/ssh_nopassword.expect $node $HADOOP_USER $HADOOP_PASSWORD $HADOOPDIR_CONF/.ssh/id_rsa.pub >/dev/null 2>&1
- echo "$node done."
- done
- echo "Config ssh successful."
ssh_nopassword.expect文件如下：
- #!/usr/bin/expect -f
- set host [lindex $argv 0]
- set user [lindex $argv 1]
- set password [lindex $argv 2]
- set dir [lindex $argv 3]
- spawn ssh-copy-id -i $dir $user@$host
- expect {
- yes/no
- {
- send "yes\r";exp_continue
- }
- -nocase "password:"
- {
- send "$password\r"
- }
- }
- expect eof
3.2.1.3 config_zookeeper.sh
該文件主要是對zookeeper的配置，文件內(nèi)容如下：
- #!/bin/bash
- # Get absolute path of this shell
- readonly PROGDIR=$(readlink -m $(dirname $0))
- # import init.conf
- source $PROGDIR/../conf/init.conf
- #update conf
- sed -i "s%ZOOKEEPERHOME%${ZOOKEEPERHOME}%g" $PROGDIR/../template/zoo.cfg
- sed -i "s%ZOOKEEPER_SLAVE1%${ZOOKEEPER_SLAVE1}%g" $PROGDIR/../template/zoo.cfg
- sed -i "s%ZOOKEEPER_SLAVE2%${ZOOKEEPER_SLAVE2}%g" $PROGDIR/../template/zoo.cfg
- sed -i "s%ZOOKEEPER_SLAVE3%${ZOOKEEPER_SLAVE3}%g" $PROGDIR/../template/zoo.cfg
- zookeeperArr=( "$ZOOKEEPER_SLAVE1" "$ZOOKEEPER_SLAVE2" "$ZOOKEEPER_SLAVE3" )
- myid=1
- for node in ${zookeeperArr[@]}; do
- scp $PROGDIR/../template/zoo.cfg $HADOOP_USER@$node:$ZOOKEEPERHOME/conf
- echo $myid > $PROGDIR/../template/myid
- ssh -q $HADOOP_USER@$node "
- [ ! -d $ZOOKEEPERHOME/data ] && ( mkdir $ZOOKEEPERHOME/data )
- [ ! -d $ZOOKEEPERHOME/log ] && ( mkdir $ZOOKEEPERHOME/log )
- "
- scp $PROGDIR/../template/myid $HADOOP_USER@$node:$ZOOKEEPERHOME/data
- myid=`expr $myid + 1` #i++
- echo "$node copy zookeeper template done."
- done
3.2.1.4 start_all.sh
該腳本主要用來啟動(dòng)zookeeper及Hadoop全部組件，文件內(nèi)容如下：
- #!/bin/bash
- source /etc/profile
- # Get absolute path of this shell
- readonly PROGDIR=$(readlink -m $(dirname $0))
- # import init.conf
- source $PROGDIR/../conf/init.conf
- # start zookeeper
- zookeeperArr=( "$ZOOKEEPER_SLAVE1" "$ZOOKEEPER_SLAVE2" "$ZOOKEEPER_SLAVE3" )
- for znode in ${zookeeperArr[@]}; do
- ssh -q $HADOOP_USER@$znode "
- source /etc/profile
- $ZOOKEEPERHOME/bin/zkServer.sh start
- "
- echo "$znode zookeeper start done."
- done
- # start journalnode
- journalArr=( $JOURNALLIST )
- for jnode in ${journalArr[@]}; do
- ssh -q $HADOOP_USER@$jnode "
- source /etc/profile
- $HADOOPHOME/sbin/hadoop-daemon.sh start journalnode
- "
- echo "$jnode journalnode start done."
- done
- # format zookeeper
- $HADOOPHOME/bin/hdfs zkfc -formatZK
- # format hdfs
- $HADOOPHOME/bin/hdfs namenode -format -clusterId $DFS_NAMESERVICES
- # start namenode
- $HADOOPHOME/sbin/hadoop-daemon.sh start namenode
- # sign in master_ha, sync from namenode to namenode_ha
- ssh -q $HADOOP_USER@$MASTER_HA_HOST "
- $HADOOPHOME/bin/hdfs namenode -bootstrapStandby
- "
- # start zkfc on master
- $HADOOPHOME/sbin/hadoop-daemon.sh start zkfc
- # start namenode_ha and datanode
- $HADOOPHOME/sbin/start-dfs.sh
- # start yarn
- $HADOOPHOME/sbin/start-yarn.sh
- # start yarn_ha
- ssh -q $HADOOP_USER@$MASTER_HA_HOST "
- source /etc/profile
- $HADOOPHOME/sbin/yarn-daemon.sh start resourcemanager
- "
- echo "start all done."
4 集群自動(dòng)化部署流程
4.1 root腳本的執(zhí)行
選擇一臺服務(wù)器作為Hadoop2.6.0的主節(jié)點(diǎn)，使用root用戶執(zhí)行。
- 確保Hadoop集群所在服務(wù)器可以執(zhí)行scp命令：在各個(gè)服務(wù)器上執(zhí)行scp，如果提示命令沒有找到，執(zhí)行安裝命令：yum –y install openssh-clients。
- 執(zhí)行以下操作：
- 執(zhí)行cd ~，進(jìn)入/root目錄下
- 將root腳本所在目錄打成tar包(假設(shè)打包后的文件名為root_install.tar.gz)，執(zhí)行rz -y，上傳root_install.tar.gz(若無法找到rz命令，執(zhí)行安裝命令：yum -y install lrzsz)
- 執(zhí)行tar -zxvf root_install.tar.gz解壓
- 執(zhí)行cd root_install，進(jìn)入到root_install目錄中
- 執(zhí)行. /installRoot.sh，開始安裝jdk、zookeeper、Hadoop，等待安裝結(jié)束
- 檢查/etc/hosts、/etc/profile的配置，執(zhí)行java -version、hadoop version命令檢查jdk和Hadoop的安裝情況。若出現(xiàn)java、hadoop命令找不到的情況，重新登錄一次服務(wù)器再進(jìn)行檢查。
4.2 hadoop腳本的執(zhí)行
在主節(jié)點(diǎn)使用Hadoop啟動(dòng)用戶執(zhí)行(該啟動(dòng)用戶是在root中執(zhí)行的腳本里創(chuàng)建的，下面假設(shè)該用戶為hadoop2)：
- 在root用戶中直接進(jìn)入hadoop2用戶，執(zhí)行su - hadoop2
- 執(zhí)行以下操作：
- 執(zhí)行cd~，進(jìn)入/home/hadoop2目錄下
- 將hadoop腳本所在目錄打成tar包(假設(shè)打包后的文件名為hadoop_install.tar.gz)，執(zhí)行rz -y，上傳hadoop_install.tar.gz(若無法找到rz命令，執(zhí)行安裝命令：yum -y install lrzsz)
- 執(zhí)行tar -zxvf hadoop_install.tar.gz解壓
- 執(zhí)行cd hadoop_install，進(jìn)入到hadoop_install目錄中
- 執(zhí)行./installCluster.sh，開始配置并啟動(dòng)zookeeper、Hadoop，等待腳本執(zhí)行結(jié)束
- 檢查zookeeper、Hadoop啟動(dòng)日志，檢查是否安裝成功。通過Hadoop本身提供的監(jiān)控頁面來檢查Hadoop集群的狀態(tài)。
- 最后根據(jù)mountTable.xml中fs.viewfs.mounttable.hCluster.link./tmp的配置，執(zhí)行如下命令創(chuàng)建該name對應(yīng)的value目錄：
hdfs dfs -mkdir hdfs://hadoop-cluster1/tmp
如果不創(chuàng)建，執(zhí)行hdfs dfs -ls /tmp時(shí)會(huì)提示找不到目錄。
5 總結(jié)
Hadoop2.6.0部署腳本仍有缺陷，比如配置文件中參數(shù)較多，有部分重復(fù)，腳本的編寫也有待改進(jìn)。權(quán)當(dāng)拋磚引玉。如有錯(cuò)誤請童鞋們指正，謝謝。
責(zé)任編輯：龐桂玉
來源：
神算子
