Spark运行自带例子与Hadoop常见问题解析-CSDN博客

本文链接：https://blog.csdn.net/u011569805/article/details/25216791

本文主要介绍了在运行Spark自带示例时遇到的问题及解决方法，包括运行wordcount示例时遇到的NN low错误。该错误是由于Namenode的容量不足导致Hadoop进入安全模式。通过释放磁盘空间，可以避免此错误。同时，文章还提到了在处理此类问题中涉及的一些Hadoop命令。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1. 运行spark自带例子时遇到几个问题

export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop/

# Submit Spark's ApplicationMaster to YARN's ResourceManager, and instruct Spark to run the SparkPi example
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0.jar \
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar ./assembly/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar \
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 2g \
      --worker-memory 2g \
      --worker-cores 1

直接运行时总报错，应该先启动 start-master.sh

2. 运行wordcount

export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop/

# Submit Spark's ApplicationMaster to YARN's ResourceManager, and instruct Spark to run the SparkPi example
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0.jar \
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar ./assembly/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar \
      --class org.apache.spark.examples.JavaWordCount \
      --args 'yarn-standalone' \
      --args 'hdfs://newslave3:9000/user/hadoop/input/test' \
      --num-workers 3 \
      --master-memory 2g \
      --worker-memory 2g \
      --worker-cores 2

输入参数个数应该正确，hdfs文件路径应该确定没有输错

3. 出现NN low错误总是hadoop自动进入safe mode(hadoop dfsadmin safemode -leave也不管用 )

原因namenode容量不够，空出一些容量后就不会出现该错误

4.涉及的hadoop命令

hadoop dfsadmin -safemode leave
hadoop fs -ls 
hadoop fs  -cat
hadoop fs -rmr
hadoop fs -put
hadoop fs -mkdir
hadoop dfsadmin -report

Spark 运行自带例子