Related:
Is your feature request related to a problem or challenge?
Each operator that opts into range satisfaction currently re-tests the same shared config machinery (datafusion.optimizer.subset_repartition_threshold, datafusion.optimizer.preserve_file_partitions, datafusion.execution.target_partitions) through per-operator blocks in range_partitioning.slt. The machinery is shared, so the file grows the same permutations for every operator.
This was raised in the #23416 review (discussion): keep range_partitioning.slt lean per operator, and cover the config permutations once, colocated with the decision logic.
Describe the solution you'd like
A table-driven test in datafusion/core/tests/physical_optimizer/enforce_distribution.rs asserting reuse (no RepartitionExec inserted; the existing Range partitioning satisfies the requirement) vs. hash repartition (RepartitionExec: Hash inserted) across combinations of:
- key match: exact / subset / incompatible
subset_repartition_threshold: met / not metpreserve_file_partitions: met / not mettarget_partitions: equal to / greater than input partition count
Behavior verified while developing #23416 that the matrix must pin:
- exact key, threshold not met, equal partition count → reuse (repartitioning gains no parallelism)
- exact key, threshold not met, target > count, preserve not met → hash repartition
- exact key, threshold not met, target > count, preserve met → reuse (the threshold and preserve conditions are OR'd in
ensure_distribution)
- exact key, threshold met, preserve not met, target > count → reuse
- subset key, threshold not met, preserve not met → hash repartition
- incompatible key → hash repartition in all configurations
Describe alternatives you've considered
None.
Additional context
- Once the matrix lands, the equivalent per-operator slt permutations in the aggregate section of
range_partitioning.slt can also be slimmed, and future operator opt-ins only need operator-specific slt cases.
Related:
Partitioning::Rangeto satisfy windowDistribution::KeyPartitionedrequirements #23289 (PR feat: allow Partitioning::Range to satisfy window Distribution::KeyPartitioned requirements #23416, where this follow-up was requested)Partitioning::Range#23239Partitioning::Rangeto satisfy PartitionedTopKDistribution::KeyPartitionedrequirements #23290 (PR Allow Range partitioned inputs to PartitionedTopK #23355)Is your feature request related to a problem or challenge?
Each operator that opts into range satisfaction currently re-tests the same shared config machinery (
datafusion.optimizer.subset_repartition_threshold,datafusion.optimizer.preserve_file_partitions,datafusion.execution.target_partitions) through per-operator blocks inrange_partitioning.slt. The machinery is shared, so the file grows the same permutations for every operator.This was raised in the #23416 review (discussion): keep
range_partitioning.sltlean per operator, and cover the config permutations once, colocated with the decision logic.Describe the solution you'd like
A table-driven test in
datafusion/core/tests/physical_optimizer/enforce_distribution.rsasserting reuse (noRepartitionExecinserted; the existing Range partitioning satisfies the requirement) vs. hash repartition (RepartitionExec: Hashinserted) across combinations of:subset_repartition_threshold: met / not metpreserve_file_partitions: met / not mettarget_partitions: equal to / greater than input partition countBehavior verified while developing #23416 that the matrix must pin:
ensure_distribution)Describe alternatives you've considered
None.
Additional context
range_partitioning.sltcan also be slimmed, and future operator opt-ins only need operator-specific slt cases.