spark standalone work扩展

最新推荐文章于 2025-08-23 15:54:57 发布

原创最新推荐文章于 2025-08-23 15:54:57 发布 · 682 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #hadoop

Hadoop 同时被 2 个专栏收录

6 篇文章

订阅专栏

spark

1 篇文章

订阅专栏

本文档详细介绍了如何在已有的Spark Standalone集群上进行扩展，包括添加新的Worker节点。首先确保所有节点配置了Java环境和下载了Spark安装包，并更新hosts文件。接着创建名为'spark'的用户，并实现免密登录。然后进行目录权限设置。在master和node1节点上完成特定的Spark配置。最后，逐步指导如何将node2作为新的Worker节点加入到集群中，以扩展Spark的工作能力。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

所有节点配置Java环境以及下载spark安装包

所有节点配置hosts文件

192.168.2.28 master
192.168.2.29 node1
192.168.2.30 node2

1. 创建spark用户

[root@master ~]# useradd spark
[root@node1 ~]# useradd spark

2. 配置免秘钥登录

具体操作查看此处: http://blog.csdn.net/zxf_668899/article/details/53726226

测试：

[spark@master ~]$ ssh master    登录自己
Last login: Thu Feb  9 15:12:08 2017 from master
[spark@master ~]$ exit
logout
Connection to master closed.
[spark@master ~]$ ssh node1     登录node1
Last login: Thu Feb  9 14:55:46 2017
[spark@node1 ~]$ exit
logout
Connection to node1 closed.

3. 目录权限设置

master:
[spark@master ~]$ ll -d /opt/source/spark-2.0.2-bin-hadoop2.7
drwxr-xr-x 14 spark spark 4096 Feb  9 10:53 /opt/source/spark-2.0.2-bin-hadoop2.7

node1:
[spark@node1 ~]$ ll -d /opt/source/spark-2.0.2-bin-hadoop2.7
drwxr-xr-x 14 spark spark 4096 Feb  9 10:54 /opt/source/spark-2.0.2-bin-hadoop2.7

4. spark 配置(spark用户操作)

master节点操作：

[spark@master ~]$ cd /opt/spark/
[spark@master spark]$ cp conf/spark-env.sh.template conf/spark-env.sh
[spark@master spark]$ cp conf/slaves.template conf/slaves
[spark@master spark]$ vim conf/spark-env.sh
......
export JAVA_HOME=/opt/jdk
export SPARK_WORKER_CORES=1
export SPARK_WORKER_DIR=/home/spark/work
export SPARK_DAEMON_MEMORY=1G
export SHARK_MASTER_MEM=1G
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/lib/*:/opt/spark/lib/*
[spark@master spark]$ tail -n 2 /opt/spark/conf/slaves
master
node1

node1节点操作：

[spark@node1 spark]$ vim conf/spark-env.sh
......
export JAVA_HOME=/opt/jdk
export SPARK_WORKER_CORES=1
export SPARK_WORKER_DIR=/home/spark/work
export SPARK_DAEMON_MEMORY=1G
export SHARK_MASTER_MEM=1G
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/lib/*:/opt/spark/lib/*

启动spark

[spark@master spark]$ ./sbin/start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /opt/spark/logs/spark-spark-org.apache.spark.deploy.master.Master-1-master.out
node1: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-node1.out
master: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-master.out

这里写图片描述

扩展添加 work节点

添加node2 work 节点

配置Java环境、hosts文件、spark安装包、创建用户、免密登录(略)

[spark@node2 ~]$ ll -d /opt/source/spark-2.0.2-bin-hadoop2.7
drwxr-xr-x 14 spark spark 4096 Feb  9 10:54 /opt/source/spark-2.0.2-bin-hadoop2.7

[spark@node2 spark]$ vim conf/spark-env.sh
......
export JAVA_HOME=/opt/jdk
export SPARK_WORKER_CORES=1
export SPARK_WORKER_DIR=/home/spark/work
export SPARK_DAEMON_MEMORY=1G
export SHARK_MASTER_MEM=1G
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/lib/*:/opt/spark/lib/*

[spark@node2 spark]$ tail -n 2 conf/slaves
# A Spark Worker will be started on each of the machines listed below.
node2

[spark@node2 spark]$ ./sbin/start-slave.sh spark://master:7077
starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-node2.out