Skip to content

Cache directory children for read-only Path.iterdir() - #167

Closed
mdeck wants to merge 2 commits into
jaraco:mainfrom
mdeck:optimize-directory-index
Closed

mdeck wants to merge 2 commits into
jaraco:mainfrom
mdeck:optimize-directory-index

Conversation

@mdeck

@mdeck mdeck commented Oct 30, 2026 •

Copy link
Copy Markdown

Recursive Path.iterdir() traversal currently scans every archive entry and
constructs a Path for each candidate at every directory. Cache immediate child
names by parent on the first read-only listing, then construct paths only for
that directory's children. A complete traversal avoids repeated archive-wide
scans.

The index is owned by the archive and released on ZipFile.close(). Nonlisting
operations do not build it. Writable and closed archives scan without caching;
Path subclasses retain the original implementation to preserve their hooks.
Entry order, duplicates, and unusual-name behavior are preserved. Closing clears
the cached index without invalidating existing iterators.

Performance and trade-off

CPython main at 4a9b98115fec4d93c32c554aa541d6c3fc4c3b27
(3.16.0a1+dev), GCC 13.3, -O3, without PGO/LTO. Both implementations use
this interpreter; the baseline is its unmodified zipfile.Path.
Medians of five rotating runs, with ZIP opening excluded. The recursive case
has 500 directories and 2,500 files. The first traversal includes name
preparation and index construction; the repeated traversal follows a complete
walk.

Operation Before After
First recursive traversal 1,213 ms 6.99 ms
Repeated recursive traversal 1,113 ms 1.29 ms
First child, 5,000 flat files, names already prepared 0.015 ms 0.771 ms

The first listing builds the entire index even if only one child is requested.
The extra retained allocations were approximately 41 KiB for the flat archive
and 99 KiB for the recursive case. Directory-heavy layouts can have
proportionally larger overhead. The improvement favors repeated or recursive
listings at the expense of first-listing work and additional metadata storage.

Validation

On the same CPython main build:

  • 87 functional/doctest tests passed, including all 73 path tests.
  • 306 resource/metadata integration tests ran with the candidate installed in
    process; two skipped.
  • 1,000 randomized archives: 26,211 ordered listings matched upstream.
  • Tests cover writable archives, subclass hooks, ordering, duplicates, unusual
    names, pickling, concurrent construction, and cleanup during and after close.

Ruff, formatting, mypy, and git diff --check passed.

@mdeck

mdeck commented Nov 1, 2026

Copy link
Copy Markdown
Author

Closing in favor of the earlier #164, which proposes the same core directory index. I’ve shared focused parsing and close-cleanup follow-ups there, with the subclass compatibility question kept separate.

@mdeck mdeck closed this Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant