Skip to content

[core] Improve vector capacity growth for huge batches#8660

Open
huangxiaopingRD wants to merge 2 commits into
apache:masterfrom
huangxiaopingRD:improve-vector-capacity-growth
Open

[core] Improve vector capacity growth for huge batches#8660
huangxiaopingRD wants to merge 2 commits into
apache:masterfrom
huangxiaopingRD:improve-vector-capacity-growth

Conversation

@huangxiaopingRD

Copy link
Copy Markdown
Contributor

Purpose

Improve writable column vector capacity growth for very large batches.
Previously, AbstractWritableVector.reserve always expanded to requiredCapacity * 2. For deeply nested or large collection columns, this can cause a single reserve call to allocate much more memory than actually needed and trigger OutOfMemoryError.
This change keeps the existing doubling behavior for normal-sized vectors, but uses more conservative growth for huge vectors. It also releases oversized heap vector arrays on reset() by restoring capacity to the initial capacity, so one unusually large batch does not make the vector retain a huge backing array forever.

Tests

Added unit tests for normal and huge vector capacity growth, reset behavior, max capacity rejection, and heap vector backing array release. The new tests pass locally.

@JingsongLi

JingsongLi commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Aren't all vectors the same size? Why do we need to scale them back down?

@huangxiaopingRD

huangxiaopingRD commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Aren't all vectors the same size? Why do we need to scale them back down?

No. Only the top-level vectors are usually created with the same batch capacity.

For nested types, child vectors are sized by the number of nested elements, not by the number of top-level rows. For example, an array<int> column may have 4K top-level rows, so the parent array vector has capacity around 4K. But if those rows contain large arrays, the child int vector may need to hold hundreds of thousands or millions of elements.

So vectors in the same vector tree are not necessarily the same size:
parent array vector: one entry per row, stores nulls/offsets/lengths
child value vector: one entry per array element, stores actual values

That is why scaling down can matter. A child vector can grow much larger than the parent after one unusually large batch. Since the vector tree is reused across batches and AbstractStructVector.reset() also resets writable children, each vector should be able to release oversized backing arrays when it has crossed the huge-vector threshold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants