Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndSer

原创

已于 2022-03-24 10:51:33 修改 · 1.1w 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#Spark

于 2022-03-24 10:50:48 首次发布

本文报道了一次Spark作业因Python Worker中出现NameError：'memoryview'未定义的错误，同时提到可能的原因是重复执行或变量名冲突，并提供了故障排查和解决建议。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

在这里插入图片描述
报错如下：

: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 19, 192.168.10.103, executor 1): org.apache.spark.SparkException: 
Error from python worker:
  Traceback (most recent call last):
    File "/usr/lib64/python2.6/runpy.py", line 104, in _run_module_as_main
      loader, code, fname = _get_module_details(mod_name)
    File "/usr/lib64/python2.6/runpy.py", line 79, in _get_module_details
      loader = get_loader(mod_name)
    File "/usr/lib64/python2.6/pkgutil.py", line 456, in get_loader
      return find_loader(fullname)
    File "/usr/lib64/python2.6/pkgutil.py", line 466, in find_loader
      for importer in iter_importers(fullname):
    File "/usr/lib64/python2.6/pkgutil.py", line 422, in iter_importers
      __import__(pkg)
    File "/opt/module/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 51, in <module>
    File "/opt/module/spark/python/lib/pyspark.zip/pyspark/context.py", line 31, in <module>
    File "/opt/module/spark/python/lib/pyspark.zip/pyspark/accumulators.py", line 97, in <module>
    File "/opt/module/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 71, in <module>
    File "/opt/module/spark/python/lib/pyspark.zip/pysp