Repository navigation
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I opened #167 before noticing this earlier PR. Both use the same directory index, so I’d prefer to consolidate here. I’ve prepared two focused follow-ups against your branch:
The combined follow-up passes 77 functional/doctest tests, 306 resource/metadata integration tests (two skipped), and 25,270 ordered-listing comparisons over 1,000 randomized archives on that CPython build. Lint and type checks pass. One separate compatibility question: the indexed path bypasses Happy to contribute whichever parts you and the maintainer find useful. I’ll close #167 in favor of this PR. |
|
Thanks, Mike. Both follow-ups are now included with your authorship preserved in 58d12aa and e108b4c. 9fc3f77 addresses the hook compatibility issue. The index is used only when The configured Python 3.14 tox suite passes with 126 tests. All nine close and custom-hook regressions fail against the previous PR head and pass with this update. A deterministic verifier also passed 29,890 ordered-listing comparisons across 1,000 generated archives. Changed-line coverage is 100% across 97 lines. I updated the PR description to describe the close lifecycle and compatibility policy. The earlier timing and CPython results are explicitly marked as initial measurements, since they were not rerun for this update. The GitHub test workflow currently reports |
Summary
Path.iterdir()builds aPathfor every name in the archive and then filters down to direct children. If you walk the whole tree, you rescan the archive once per directory, which makes the walk O(dirs × entries).importlib.resources.as_file()on a zipped package does exactly this walk.Read-only archives already go through
FastLookup, which caches the name list. This adds a cached parent → children index next to it:class FastLookup(CompleteDirs): namelist() # cached _name_set() # cached + _children # cached: {parent_dir: [names]} Path.iterdir() if not is_dir(): raise + if open FastLookup and both hooks are unchanged: + return map(_next, root._children[at]) return filter(_is_child, map(_next, root.namelist()))Same names, same order, same parent rule (
dirname(name.rstrip('/'))) as_is_child. Direct parent splitting preserves unusual slash-only parents and duplicate entries. Ordinary inherited subclasses retain the index. Overridden, replaced, or rebound_nextand_is_childhooks use the original scan so their listing behavior is preserved. Write-mode archives also keep the original scan.Closing an archive releases its child index. Iterators created before close remain usable, and metadata listings after close do not retain a new index. The two parent-parsing and close-lifecycle follow-ups from @mdeck are included with their authorship preserved.
Evidence
Current update on Python 3.14.5:
tox -e py314suite passes with 126 tests.mainis 100% across 97 lines.The timing and CPython results below are the initial PR measurements. They were not rerun for this update.
Full tree walk with zipp on Python 3.14, median of 7, macOS arm64.
nis the number of directories, each holding one file:Same patch applied to CPython's
zipfile._path(3.16 dev build):as_file()on a zipped package, n=1024as_file(), n=4 (100 runs, alternating, 3 batches)New test
test_recursive_iterdir_workcountsPathobjects built while walking a 32-directory archive:AssertionError: assert 2113 <= (4 * 32)Initial
tox -e py314: 115 passed, ruff clean. In CPython,test_zipfile+test_importlibwith the patch: 952 run, 42 skipped, all pass.Merge Danger
Door: two-way
The index is private, cached per open
FastLookupinstance, and built on first eligibleiterdir(). Custom hooks retain the scan path. The indexed path still assumes that a subclass inheriting both hooks preserves the suppliedatvalue in its constructor.Blast Radius: memory
Each open read-only archive using the index keeps one extra dict until close. In the initial 1,024-directory walk measurement (CPython build), retained traversal memory went from 79 KB to 262 KB and peak from 178 KB to 364 KB. Archives that never use the index don't pay this cost.
AI disclosure: the profiling and the patch were done with AI tools (Codex, Claude).
🤖 Generated with Claude Code