使用IDEA工具运行SparkStreaming的WordCount时,运行报错,报错信息如下:
Exception in thread "main" java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
at org.apache.spark.streaming.Duration.<init>(Duration.scala:22)
at org.apache.spark.streaming.Seconds$.apply(Duration.scala:90)
at com.aaa.streaming.WordCount$.main(WordCount.scala:18)
at com.aaa.streaming.WordCount.main(WordCount.scala)
经查阅资料,参考https://www.cnblogs.com/zengsong-restService/p/10599007.html初步确认原因为:当前idea引入的scala运行环境版本与idea默认的scala版本不一样
查看本项目的Project Structure,点击Global Libraries选项查看,显示版本一致
打开pom.xml文件,检查Spark对应的Scala版本,发现添加的SparkStreaming依赖的信息如下:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.12</artifactId>
<version>2.4.4</version>
<scope>compile</scope>
</dependency>
修改如下:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.1.1</version>
<scope>compile</scope>
</dependency>
待依赖下载完成后运行程序,即可正常运行。