Add host_read_async interfaces to datasource
#18018
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test |
3 tasks
host_read_async interfaces to datasource
PointKernel
reviewed
Feb 19, 2025
|
Turns out I forgot to implement |
kingcrimsontianyu
approved these changes
Feb 20, 2025
Co-authored-by: Tianyu Liu <[email protected]>
|
Found another spot where the new APIs need to be implemented - |
rapids-bot bot
pushed a commit
that referenced
this pull request
Feb 24, 2025
) Depends on #18018 When reading multiple files, all data(i.e. pages) IO is performed in the same "batch", allowing parallel IO operations (provided by kvikIO). However, footers are read serially, leading to poor performance when reading many files. This is especially pronounced for IO that benefits from high level of parallelism. This PR performs footer reading/parsing asynchronously using an internal thread pool. The pool size can be controlled with an environment variable `LIBCUDF_NUM_HOST_WORKERS`. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Nghia Truong (https://github.com/ttnghia) - Paul Mattione (https://github.com/pmattione-nvidia) - Bradley Dice (https://github.com/bdice) URL: #17957
|
Now that #17957 is merged, should this PR be merged as well? |
|
I'll proceed with merging this PR; these APIs will be required to eliminate redundant copies when host compression is used. |
2 tasks
|
/merge |
1420ef2
into
rapidsai:branch-25.04
105 of 106 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
kvikIO supports asynchronous host reads, but we don't utilize them to optimize host reads such as metadata access.
This PR adds the async versions of the
host_readAPIs to allow efficient use of the kvikIO pool for host reads. Thedatasources that are not backed by kvikIO implement these as deferred calls to the synchronous versions.Checklist