本地YARN集群部署

发布于：2025-03-06 ⋅ 阅读:(17) ⋅ 点赞:(0)

请先完成HDFS的前置部署，部署方式可查看:本地部署HDFS集群https://blog.csdn.net/m0_73641796/article/details/145998092?spm=1001.2014.3001.5502

 部署说明

组件 配置文件 启动进程 备注
Hadoop HDFS 需修改

需启动:

NameNode作为主节点

DataNode作为从节点
SecondaryNameNode主节点辅助

 分布式文件系统
Hadoop YARN 需修改

需启动
ResourceManager作为集群资源管理者
NodeManager作为单机资源管理者

ProxyServer代理服务器提供安全性

JobHistoryServer记录历史信息和日志

 分布式资源调度
Hadoop MapReduce 需修改

无需启动任何进程

MapReduce程序运行在YARN容器内

 分布式数据计算

集群规划

主机 角色
node1

ResourceManager
NodeManager

ProxyServer

JobHistoryServer
node2 NodeManager
node3 NodeManager

修改MapReduce配置文件

在$HADOOP_HOME/etc/hadoop文件夹内，修改mapred-env.sh文件，添加如下环境变量

export JAVA_HOME=/export/server/jdk
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000
export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA

修改mapred-site.xml文件，添加如下配置信息

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    <description></description>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>node1:10020</value>
    <description></description>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>node1:19888</value>
    <description></description>
  </property>
  <property>
    <name>mapreduce.jobhistory.intermediate-done-dir</name>
    <value>/data/mr-history/tmp</value>
    <description></description>
  </property>
  <property>
    <name>mapreduce.jobhistory.done-dir</name>
    <value>/data/mr-history/done</value>
    <description></description>
  </property>
 <property>
   <name>yarn.app.mapreduce.am.env</name>
   <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
 </property>
 <property>
   <name>mapreduce.map.env</name>
   <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
 </property>
 <property>
   <name>mapreduce.reduce.env</name>
   <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
 </property>
</configuration>

修改YARN配置文件

在$HADOOP_HOME/etc/hadoop文件夹内，修改yarn-env.sh文件，添加如下环境变量

export JAVA_HOME=/export/server/jdk
export HADOOP_HOME=/export/server/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR=$HADOOP_HOME/logs

修改yarn-site.xml文件，添加如下内容：

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.log.server.url</name>
    <value>http://node1:19888/jobhistory/logs</value>
    <description></description>
</property>
  <property>
    <name>yarn.web-proxy.address</name>
    <value>node1:8089</value>
    <description>proxy server hostname and port</description>
  </property>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
    <description>Configuration to enable or disable log aggregation</description>
  </property>
  <property>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/tmp/logs</value>
    <description>Configuration to enable or disable log aggregation</description>
  </property>

<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>node1</value>
    <description></description>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    <description></description>
  </property>
  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/data/nm-local</value>
    <description>Comma-separated list of paths on the local filesystem where intermediate data is written.</description>
  </property>
  <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/data/nm-log</value>
    <description>Comma-separated list of paths on the local filesystem where logs are written.</description>
  </property>
  <property>
    <name>yarn.nodemanager.log.retain-seconds</name>
    <value>10800</value>
    <description>Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.</description>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
    <description>Shuffle service that needs to be set for Map Reduce applications.</description>
  </property>
</configuration>

 分发配置文件

scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node2:`pwd`/
scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node3:`pwd`/

 启动HDFS

start-dfs.sh

 启动YARN

# 一键启动YARN集群
start-yarn.sh

# 一键停止YARN集群
stop-yarn.sh

# 在当前机器，单独启动或停止进程
yarn --daemon start|stop resourcemanager|nodemanager|proxyserver

# 历史服务器启动和停止
mapred --daemon start|stop historyserver

 可通过jps命令查看当前运行的进程

 

查看YARN的WEB UI页面

打开 http://node1:8088 即可看到YARN集群的监控页面（ResourceManager的WEB UI）

网站公告

今日签到

点亮在社区的每一天
去签到

热门文章

最新发布