pyspark.RDD.saveAsPickleFile

RDD.saveAsPickleFile(path, batchSize=10)

Save this RDD as a SequenceFile of serialized objects. The serializer used is pyspark.serializers.CPickleSerializer, default batch size is 10.

Added in version 1.1.0.

Parameters:

pathstr

path to pickled file

batchSizeint, optional, default 10

the number of Python objects represented as a single Java object.

See also

SparkContext.pickleFile()

Examples

>>> import os
>>> import tempfile
>>> with tempfile.TemporaryDirectory(prefix="saveAsPickleFile") as d:
...     path = os.path.join(d, "pickle_file")
...
...     # Write a temporary pickled file
...     sc.parallelize(range(10)).saveAsPickleFile(path, 3)
...
...     # Load picked file as an RDD
...     sorted(sc.pickleFile(path, 3).collect())
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]