MongoSamplePartitioner

最新推荐文章于 2024-02-05 20:27:16 发布

Bol5261

最新推荐文章于 2024-02-05 20:27:16 发布

阅读量380

点赞数

CC 4.0 BY-SA版权

分类专栏： MongoDB(Collenction-oriented) 文章标签： mongodb

原文链接：https://www.javadoc.io/doc/

MongoDB(Collenction-oriented) 专栏收录该内容

192 篇文章

订阅专栏

本文深入探讨了MongoSamplePartitioner的实现与应用，这是一种用于确定MongoDB集合合适分区的采样分区器，通过平均文档大小和集合随机采样来工作。配置属性包括分区键、分区大小（MB）和每个分区的样本数。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Class/Object

com.mongodb.spark.rdd.partitioner
MongoSamplePartitioner
Related Docs: object MongoSamplePartitioner | package partitioner
class MongoSamplePartitioner extends Logging with MongoPartitioner

The Sample Partitioner.

Uses the average document size and random sampling of the collection to determine suitable partitions for the collection.
Configuration Properties

The prefix when using sparkConf is: spark.mongodb.input.partitionerOptions followed by the property name:

partitionKey, the field to partition the collection by. The field should be indexed and contain unique values. Defaults to _id.
partitionSizeMB, the size (in MB) for each partition. Defaults to 64.
samplesPerPartition, the number of samples for each partition. Defaults to 10.

Note: Requires MongoDB 3.2+ Note: Does not support views. Use MongoPaginateByCountPartitioner or create a custom partitioner.

Since

1.0

Linear Supertypes
Known Subclasses
Ordering

Alphabetic
By Inheritance

Inherited

MongoSamplePartitioner
MongoPartitioner
Serializable
Serializable
Logging
LoggingTrait
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Instance Constructors

new MongoSamplePartitioner()

Value Members

val partitionKeyProperty: String

The partition key property
val partitionSizeMBProperty: String

The partition size MB property
def partitions(connector: MongoConnector, readConfig: ReadConfig, pipeline: Array[BsonDocument]): Array[MongoPartition]

Calculate the Partitions
val samplesPerPartitionProperty: String

The number of samples for each partition