linux上安装spark_hadoop_java_scala

最新推荐文章于 2022-11-30 17:49:17 发布

Takoony

最新推荐文章于 2022-11-30 17:49:17 发布

阅读量316

点赞数

分类专栏： linux bigdata

linux 同时被 2 个专栏收录

99 篇文章

订阅专栏

bigdata

34 篇文章

订阅专栏

本文详细介绍了如何在Hadoop中配置YARN资源管理器，包括设置job tracker、local directory、资源管理器名称等关键参数，以及如何指定资源管理器的多个地址和服务。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

cp   /opt/hadoop/hadoop-2.8.0/etc/hadoop/mapred-site.xml.template     /opt/hadoop/hadoop-2.8.0/etc/hadoop/mapred-site.xml

修改这个新建的mapred-site.xml文件，在<configuration>节点内加入配置:

 <property>

   <name>mapred.job.tracker</name>

   <value>hserver1:49001</value>

</property>

<property>

      <name>mapred.local.dir</name>

       <value>/root/hadoop/var</value>

</property>


<property>

       <name>mapreduce.framework.name</name>

       <value>yarn</value>

</property>

<property>

        <name>yarn.resourcemanager.hostname</name>

        <value>hserver1</value>

   </property>

   <property>

        <description>The address of the applications manager interface in the RM.</description>

        <name>yarn.resourcemanager.address</name>

        <value>${yarn.resourcemanager.hostname}:8032</value>

   </property>

   <property>

        <description>The address of the scheduler interface.</description>

        <name>yarn.resourcemanager.scheduler.address</name>

        <value>${yarn.resourcemanager.hostname}:8030</value>

   </property>

   <property>

        <description>The http address of the RM web application.</description>

        <name>yarn.resourcemanager.webapp.address</name>

        <value>${yarn.resourcemanager.hostname}:8088</value>

   </property>

   <property>

        <description>The https adddress of the RM web application.</description>

        <name>yarn.resourcemanager.webapp.https.address</name>

        <value>${yarn.resourcemanager.hostname}:8090</value>

   </property>

   <property>

        <name>yarn.resourcemanager.resource-tracker.address</name>

        <value>${yarn.resourcemanager.hostname}:8031</value>

   </property>

   <property>

        <description>The address of the RM admin interface.</description>

        <name>yarn.resourcemanager.admin.address</name>

        <value>${yarn.resourcemanager.hostname}:8033</value>

   </property>

   <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

   </property>

   <property>

        <name>yarn.scheduler.maximum-allocation-mb</name>

        <value>2048</value>

        <discription>每个节点可用内存,单位MB,默认8182MB</discription>

   </property>

   <property>

        <name>yarn.nodemanager.vmem-pmem-ratio</name>

        <value>2.1</value>

   </property>

   <property>

        <name>yarn.nodemanager.resource.memory-mb</name>

        <value>2048</value>

</property>

   <property>

        <name>yarn.nodemanager.vmem-check-enabled</name>

        <value>false</value>

</property>

这里只是启动了Hadoop，没有用到spark