Hadoop3.1.1 集群安装

具体配置参照《大数据平台相关配置》一文

1. 环境初始化

• 修改主机名称，时区

systemctl stop firewalld.service #关闭防火墙
systemctl disable firewalld.service #关闭防火墙
timedatectl set-timezone Asia/Shanghai  #都要执行
hostnamectl set-hostname master   #master执行
hostnamectl set-hostname node1    #node1执行
hostnamectl set-hostname node2    #node2执行

• 修改Host

vi /etc/hosts

192.169.0.104 master
192.169.0.105 node1
192.169.0.106 node2

• 安装JDK1.8
  解压java 至 /opt/apps

export JAVA_HOME=/opt/apps/jdk1.8.0_181
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

• 编辑hadoop.sh[ 创建 /etc/profile.d/hadoop.sh]

cd /usr/lib &&ln -s /opt/apps/hadoop-3.1.1 hadoop-current

# set the hadoop home
export HADOOP_HOME="/usr/lib/hadoop-current"
export HADOOP_MAPRED_HOME="/usr/lib/hadoop-current"
export HADOOP_PID_DIR="${HADOOP_HOME}/pids"
export YARN_PID_DIR=${HADOOP_PID_DIR}

# set hadoop log dir
export HADOOP_LOG_DIR="/data/bigdata/log/hadoop-hdfs"
export YARN_LOG_DIR="/data/bigdata/log/hadoop-yarn"
export HADOOP_MAPRED_LOG_DIR="/data/bigdata/log/hadoop-mapred"

if [[ -n $HADOOP_HOME ]]; then
  export PATH=$HADOOP_HOME/bin:$PATH
  export PATH=$HADOOP_HOME/sbin:$PATH
fi

• 创建目录

mkdir -p /data/bigdata/log
chown hadoop:hadoop /data/bigdata/log -R

• 修改core-site.xml

<configuration>
   <property>
       <name>fs.defaultFS</name>
       <value>hdfs://master:9000</value>
   </property>
   <property>
     <name>hadoop.proxyuser.hadoop.hosts</name>
     <value>*</value>
   </property>
   <property>
     <name>hadoop.proxyuser.hadoop.groups</name>
    <value>hadoop</value>
   </property>
</configuration>

• 修改hdfs-site.xml

无需修改

• 修改mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=/opt/apps/hadoop-3.1.1</value>
    </property>
    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=/opt/apps/hadoop-3.1.1</value>
    </property>
    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=/opt/apps/hadoop-3.1.1</value>
    </property>
</configuration>

• 修改yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>1</value>
    </property>
    <property>
        <name>yarn.web-proxy.address</name>
        <value>master:9001</value>
    </property>
</configuration>

2.启动Hadoop集群

• 启动HDFS集群

[hdfs]$ $HADOOP_HOME/bin/hdfs namenode -format <cluster_name>

• Start the HDFS NameNode (主节点)

[hdfs]$ $HADOOP_HOME/bin/hdfs --daemon start namenode

• Start a HDFS DataNode （从节点）

[hdfs]$ $HADOOP_HOME/bin/hdfs --daemon start datanode

• Start the YARN (主节点)

[yarn]$ $HADOOP_HOME/bin/yarn --daemon start resourcemanager

• Run a script to start a NodeManager （从节点）

[yarn]$ $HADOOP_HOME/bin/yarn --daemon start nodemanager

• Start a standalone WebAppProxy server (主节点)

[yarn]$ $HADOOP_HOME/bin/yarn --daemon start proxyserver

• Start the MapReduce JobHistory Server (主节点)

[mapred]$ $HADOOP_HOME/bin/mapred --daemon start historyserver

访问：http://master:8088/cluster