[core] Improve vector capacity growth for huge batches#8660
[core] Improve vector capacity growth for huge batches#8660huangxiaopingRD wants to merge 2 commits into
Conversation
|
Aren't all vectors the same size? Why do we need to scale them back down? |
No. Only the top-level vectors are usually created with the same batch capacity. For nested types, child vectors are sized by the number of nested elements, not by the number of top-level rows. For example, an So vectors in the same vector tree are not necessarily the same size: That is why scaling down can matter. A child vector can grow much larger than the parent after one unusually large batch. Since the vector tree is reused across batches and |
Purpose
Improve writable column vector capacity growth for very large batches.
Previously,
AbstractWritableVector.reservealways expanded torequiredCapacity * 2. For deeply nested or large collection columns, this can cause a single reserve call to allocate much more memory than actually needed and triggerOutOfMemoryError.This change keeps the existing doubling behavior for normal-sized vectors, but uses more conservative growth for huge vectors. It also releases oversized heap vector arrays on
reset()by restoring capacity to the initial capacity, so one unusually large batch does not make the vector retain a huge backing array forever.Tests
Added unit tests for normal and huge vector capacity growth, reset behavior, max capacity rejection, and heap vector backing array release. The new tests pass locally.