Skip to content

Make recursive iterdir linear on read-only archives - #164

Open
jonbaldie wants to merge 4 commits into
jaraco:mainfrom
jonbaldie:perf-iterdir-children
Open

jonbaldie wants to merge 4 commits into
jaraco:mainfrom
jonbaldie:perf-iterdir-children

Conversation

@jonbaldie

@jonbaldie jonbaldie commented Oct 5, 2026 •

Copy link
Copy Markdown

Summary

Path.iterdir() builds a Path for every name in the archive and then filters down to direct children. If you walk the whole tree, you rescan the archive once per directory, which makes the walk O(dirs × entries). importlib.resources.as_file() on a zipped package does exactly this walk.

Read-only archives already go through FastLookup, which caches the name list. This adds a cached parent → children index next to it:

 class FastLookup(CompleteDirs):
     namelist()      # cached
     _name_set()     # cached
+    _children       # cached: {parent_dir: [names]}

 Path.iterdir()
   if not is_dir(): raise
+  if open FastLookup and both hooks are unchanged:
+    return map(_next, root._children[at])
   return filter(_is_child, map(_next, root.namelist()))

Same names, same order, same parent rule (dirname(name.rstrip('/'))) as _is_child. Direct parent splitting preserves unusual slash-only parents and duplicate entries. Ordinary inherited subclasses retain the index. Overridden, replaced, or rebound _next and _is_child hooks use the original scan so their listing behavior is preserved. Write-mode archives also keep the original scan.

Closing an archive releases its child index. Iterators created before close remain usable, and metadata listings after close do not retain a new index. The two parent-parsing and close-lifecycle follow-ups from @mdeck are included with their authorship preserved.

Evidence

Current update on Python 3.14.5:

  • Configured tox -e py314 suite passes with 126 tests.
  • Nine close and custom-hook regressions fail against the original PR head and pass with this update.
  • A deterministic verifier passes 29,890 ordered-listing comparisons across 1,000 generated archives, including duplicates and unusual names.
  • Changed-line coverage against upstream main is 100% across 97 lines.

The timing and CPython results below are the initial PR measurements. They were not rerun for this update.

Full tree walk with zipp on Python 3.14, median of 7, macOS arm64. n is the number of directories, each holding one file:

case before after
walk, n=4 0.024 ms 0.014 ms
walk, n=128 11.8 ms 0.27 ms
walk, n=1024 728 ms 2.1 ms

Same patch applied to CPython's zipfile._path (3.16 dev build):

case before after
as_file() on a zipped package, n=1024 1,055 ms 128 ms
as_file(), n=4 (100 runs, alternating, 3 batches) p50 0.86 / 0.82 / 0.78 ms p50 0.80 / 0.78 / 0.75 ms

New test test_recursive_iterdir_work counts Path objects built while walking a 32-directory archive:

  • Before: AssertionError: assert 2113 <= (4 * 32)
  • After: 65 built, passes

Initial tox -e py314: 115 passed, ruff clean. In CPython, test_zipfile + test_importlib with the patch: 952 run, 42 skipped, all pass.

Merge Danger

Door: two-way

The index is private, cached per open FastLookup instance, and built on first eligible iterdir(). Custom hooks retain the scan path. The indexed path still assumes that a subclass inheriting both hooks preserves the supplied at value in its constructor.

Blast Radius: memory

Each open read-only archive using the index keeps one extra dict until close. In the initial 1,024-directory walk measurement (CPython build), retained traversal memory went from 79 KB to 262 KB and peak from 178 KB to 364 KB. Archives that never use the index don't pay this cost.

AI disclosure: the profiling and the patch were done with AI tools (Codex, Claude).

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mdeck

mdeck commented Nov 1, 2026

Copy link
Copy Markdown

I opened #167 before noticing this earlier PR. Both use the same directory index, so I’d prefer to consolidate here.

I’ve prepared two focused follow-ups against your branch:

  • Faster parent parsing: use direct string splitting instead of dirname, preserving its parent semantics. On CPython main at 4a9b981, index construction for 5,000 flat names took 1.78 → 1.08 ms; the 500-directory / 2,500-file case took 2.07 → 0.92 ms. These are medians of 11 alternating runs, with names already prepared and ZIP opening excluded.
  • Close-time cleanup: clear the cached index on close, without invalidating existing iterators or rebuilding the index for later metadata listings. Includes tests for closure during construction.

The combined follow-up passes 77 functional/doctest tests, 306 resource/metadata integration tests (two skipped), and 25,270 ordered-listing comparisons over 1,000 randomized archives on that CPython build. Lint and type checks pass.

One separate compatibility question: the indexed path bypasses _is_child overrides. For example, a subclass whose _is_child always returns True lists all three completed names for an archive containing a/one and top upstream, but only the two root children with this PR. Restricting the fast path preserves that behavior but also loses acceleration for ordinary subclasses. The parsing and cleanup follow-ups leave your subclass policy unchanged.

Happy to contribute whichever parts you and the maintainer find useful. I’ll close #167 in favor of this PR.

@jonbaldie

Copy link
Copy Markdown
Author

Thanks, Mike. Both follow-ups are now included with your authorship preserved in 58d12aa and e108b4c.

9fc3f77 addresses the hook compatibility issue. The index is used only when _next and _is_child are the original methods bound to the current path. Overrides, instance replacements, and hooks rebound to another path retain the original scan. Subclasses that inherit both hooks still use the index.

The configured Python 3.14 tox suite passes with 126 tests. All nine close and custom-hook regressions fail against the previous PR head and pass with this update. A deterministic verifier also passed 29,890 ordered-listing comparisons across 1,000 generated archives. Changed-line coverage is 100% across 97 lines.

I updated the PR description to describe the close lifecycle and compatibility policy. The earlier timing and CPython results are explicitly marked as initial measurements, since they were not rerun for this update. The GitHub test workflow currently reports action_required and has not run its jobs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants