Skip to content

fix: Fix Event loop is closed error on dynamodb test #5480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 30, 2025
Merged

fix: Fix Event loop is closed error on dynamodb test #5480

merged 1 commit into from
Jun 30, 2025

Conversation

ntkathole
Copy link
Member

@ntkathole ntkathole commented Jun 30, 2025 •
edited
Loading

What this PR does / why we need it:

We are seeing botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: Event loop is closed on integration tests intermittently and happening frequently.

https://github.com/feast-dev/feast/actions/workflows/master_only.yml

This PR may fix it. When the client is closed, the global variables are not reset to None, which can cause issues with subsequent operations trying to use a closed client.

The fix is to reset the global client and session variables to None after closing them.

=================================== FAILURES ===================================
_ test_async_online_retrieval_with_event_timestamps_dynamo[ParameterSet(values=(LOCAL:File:dynamodb:python_fs:False,), marks=[], id=None)] _
[gw7] linux -- Python 3.10.18 /opt/hostedtoolcache/Python/3.10.18/x64/bin/python
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/httpsession.py:222: in send
    response = await session.request(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiohttp/client.py:770: in _request
    resp = await handler(req)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiohttp/client.py:725: in _connect_and_send_request
    conn = await self._connector.connect(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiohttp/connector.py:622: in connect
    proto = await self._create_connection(req, traces, timeout)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiohttp/connector.py:1189: in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiohttp/connector.py:1512: in _create_direct_connection
    hosts = await self._resolve_host(host, port, traces=traces)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiohttp/connector.py:1128: in _resolve_host
    return await asyncio.shield(resolved_host_task)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiohttp/connector.py:1159: in _resolve_host_with_throttle
    addrs = await self._resolver.resolve(host, port, family=self._family)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiohttp/resolver.py:40: in resolve
    infos = await self._loop.getaddrinfo(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/asyncio/base_events.py:863: in getaddrinfo
    return await self.run_in_executor(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/asyncio/base_events.py:808: in run_in_executor
    self._check_closed()
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/asyncio/base_events.py:515: in _check_closed
    raise RuntimeError('Event loop is closed')
E   RuntimeError: Event loop is closed

During handling of the above exception, another exception occurred:
sdk/python/tests/integration/online_store/test_universal_online.py:525: in test_async_online_retrieval_with_event_timestamps_dynamo
    await _do_async_retrieval_test(environment, universal_data_sources)
sdk/python/tests/integration/online_store/test_universal_online.py:495: in _do_async_retrieval_test
    response = await fs.get_online_features_async(
sdk/python/feast/feature_store.py:1928: in get_online_features_async
    return await provider.get_online_features_async(
sdk/python/feast/infra/passthrough_provider.py:272: in get_online_features_async
    return await self.online_store.get_online_features_async(
sdk/python/feast/infra/online_stores/online_store.py:296: in get_online_features_async
    all_responses = await asyncio.gather(
sdk/python/feast/infra/online_stores/online_store.py:287: in query_table
    read_rows = await self.online_read_async(
sdk/python/feast/infra/online_stores/dynamodb.py:455: in online_read_async
    response_batches = await asyncio.gather(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/context.py:36: in wrapper
    return await func(*args, **kwargs)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/client.py:397: in _make_api_call
    http, parsed_response = await self._make_request(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/client.py:423: in _make_request
    return await self._endpoint.make_request(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/endpoint.py:121: in _send_request
    while await self._needs_retry(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/endpoint.py:281: in _needs_retry
    responses = await self._event_emitter.emit(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/hooks.py:68: in _emit
    response = await resolve_awaitable(handler(**kwargs))
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/_helpers.py:6: in resolve_awaitable
    return await obj
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/retryhandler.py:107: in _call
    if await resolve_awaitable(self._checker(**checker_kwargs)):
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/_helpers.py:6: in resolve_awaitable
    return await obj
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/retryhandler.py:126: in _call
    should_retry = await self._should_retry(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/retryhandler.py:152: in _should_retry
    return await resolve_awaitable(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/_helpers.py:6: in resolve_awaitable
    return await obj
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/retryhandler.py:174: in _call
    checker(attempt_number, response, caught_exception)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/botocore/retryhandler.py:247: in __call__
    return self._check_caught_exception(
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/botocore/retryhandler.py:416: in _check_caught_exception
    raise caught_exception
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/endpoint.py:202: in _do_get_response
    http_response = await self._send(request)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/endpoint.py:304: in _send
    return await self.http_session.send(request)
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/aiobotocore/httpsession.py:282: in send
    raise HTTPClientError(error=e)
E   botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: Event loop is closed

@ntkathole ntkathole self-assigned this Jun 30, 2025
@ntkathole ntkathole requested a review from a team as a code owner June 30, 2025 15:28
franciscojavierarceo
franciscojavierarceo approved these changes Jun 30, 2025
View reviewed changes
Copy link
Member

@franciscojavierarceo franciscojavierarceo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@robhowley mind taking a look? I think this lgtm

@franciscojavierarceo franciscojavierarceo merged commit fe0f671 into feast-dev:master Jun 30, 2025
17 of 18 checks passed
franciscojavierarceo pushed a commit that referenced this pull request Jun 30, 2025
@feast-ci-bot
chore(release): release 0.50.0
3f30a1b 
# [0.50.0](v0.49.0...v0.50.0) (2025-06-30)

### Bug Fixes

* Add asyncio to integration test ([#5418](#5418)) ([6765515](6765515))
* Add clickhouse to OFFLINE_STORE_CLASS_FOR_TYPE map ([#5251](#5251)) ([9ed2ffa](9ed2ffa))
* Add missing conn.commit() in SnowflakeOnlineStore.online_write_batch ([#5432](#5432)) ([a83dd85](a83dd85))
* Add transformers in required dependencies ([8cde460](8cde460))
* Allow custom annotations on Operator installed objects ([#5339](#5339)) ([44c7a76](44c7a76))
* Dask pulling of latest data ([#5229](#5229)) ([571d81f](571d81f))
* **dask:** preserve remote URIs (e.g. s3://) in DaskOfflineStore path resolution ([2561cfc](2561cfc))
* Fix Event loop is closed error on dynamodb test ([#5480](#5480)) ([fe0f671](fe0f671))
* Fix lineage entity filtering ([#5321](#5321)) ([0d05701](0d05701))
* Fix list saved dataset api ([833696c](833696c))
* Fix NumPy - PyArrow array type mapping in Trino offline store ([#5393](#5393)) ([9ba9ded](9ba9ded))
* Fix pandas 2.x compatibility issue of Trino offline store caused by removed Series.iteritems() method ([#5345](#5345)) ([61e3e02](61e3e02))
* Fix polling mechanism for TestApplyAndMaterialize ([#5451](#5451)) ([b512a74](b512a74))
* Fix remote rbac integration tests ([#5473](#5473)) ([10879ec](10879ec))
* Fix Trino offline store SQL in Jinja template ([#5346](#5346)) ([648c53d](648c53d))
* Fixed CurlGeneratorTab github theme type ([#5425](#5425)) ([5f15329](5f15329))
* Increase the Operator Manager memory limits and requests ([#5441](#5441)) ([6c94dbf](6c94dbf))
* Method signature for push_async is out of date ([#5413](#5413)) ([28c3379](28c3379)), closes [#5410](#5410) [#006BB4](https://github.com/feast-dev/feast/issues/006BB4)
* Operator - support securityContext override at Pod level ([#5325](#5325)) ([33ea0f5](33ea0f5))
* Pybuild-deps throws errors w/ latest pip version ([#5311](#5311)) ([f2d6a67](f2d6a67))
* Reopen for integration test about add s3 storage-based registry store in Go feature server ([#5352](#5352)) ([ef75f61](ef75f61))
* resolve Python logger warnings ([#5361](#5361)) ([37d5c19](37d5c19))
* The ignore_paths not taking effect duration feast apply ([#5353](#5353)) ([e4917ca](e4917ca))
* Update generate_answer function to provide correct parameter format to retrieve function ([dc5b2af](dc5b2af))
* Update milvus connect function to work with remote instance ([#5382](#5382)) ([7e5e7d5](7e5e7d5))
* Updating milvus connect function to work with remote instance ([#5401](#5401)) ([b89fadd](b89fadd))
* Upperbound limit for protobuf generation ([#5309](#5309)) ([a114aae](a114aae))

### Features

* Add CLI, SDK, and API documentation page to Feast UI ([#5337](#5337)) ([203e888](203e888))
* Add dark mode toggle to Feast UI ([#5314](#5314)) ([ad02e46](ad02e46))
* Add data labeling tabs to UI ([#5410](#5410)) ([389ceb7](389ceb7)), closes [#006BB4](https://github.com/feast-dev/feast/issues/006BB4)
* Add Decimal to allowed python scalar types ([#5367](#5367)) ([4777c03](4777c03))
* Add feast rag retriver functionality ([#5405](#5405)) ([0173033](0173033))
* Add feature view curl generator ([#5415](#5415)) ([7a5b48f](7a5b48f))
* Add feature view lineage tab and filtering to home page lineage ([#5308](#5308)) ([308255d](308255d))
* Add feature view tags to dynamo tags ([#5291](#5291)) ([3a787ac](3a787ac))
* Add HybridOnlineStore for multi-backend online store routing ([#5423](#5423)) ([ebd67d1](ebd67d1))
* Add max_file_size to Snowflake config ([#5377](#5377)) ([e8cdf5d](e8cdf5d))
* Add MCP (Model Context Protocol) support for Feast feature server ([#5406](#5406)) ([de650de](de650de)), closes [#5398](#5398) [#5382](#5382) [#5389](#5389) [#5401](#5401)
* Add rag project to default dev UI ([#5323](#5323)) ([3b3e1c8](3b3e1c8))
* Add s3 storage-based registry store in Go feature server ([#5336](#5336)) ([abe18df](abe18df))
* Add support for data labeling in UI ([#5409](#5409)) ([d183c4b](d183c4b)), closes [#27](#27)
* Added Lineage APIs to get registry objects relationships ([#5472](#5472)) ([be004ef](be004ef))
* Added rest-apis serving option for registry server ([#5342](#5342)) ([9740fd1](9740fd1))
* Added torch.Tensor as option for online and offline retrieval ([#5381](#5381)) ([0b4ae95](0b4ae95))
* Adding feast delete to CLI ([#5344](#5344)) ([19fe3ac](19fe3ac))
* Adding permissions to UI and refactoring some things ([#5320](#5320)) ([6f1b0cc](6f1b0cc))
* Allow to set registry server rest/grpc mode in operator ([#5364](#5364)) ([99afd6d](99afd6d))
* Allow to use env variable FEAST_FS_YAML_FILE_PATH and FEATURE_REPO_DIR ([#5420](#5420)) ([6a1b33a](6a1b33a))
* Enable materialization for ODFV Transform on Write ([#5459](#5459)) ([3d17892](3d17892))
* Improve search results formatting ([#5326](#5326)) ([18cbd7f](18cbd7f))
* Improvements to Lambda materialization engine ([#5379](#5379)) ([b486f29](b486f29))
* Make batch_source optional in PushSource ([#5440](#5440)) ([#5454](#5454)) ([ae7e20e](ae7e20e))
* Refactor materialization engine ([#5354](#5354)) ([f5c5360](f5c5360))
* Remote Write to Online Store completes client / server architecture ([#5422](#5422)) ([2368f42](2368f42))
* Serialization version 2 and below removed ([#5435](#5435)) ([9e50e18](9e50e18))
* SQLite online retrieval. Add timezone info into timestamp. ([#5386](#5386)) ([6b05153](6b05153))
* Support dual-mode REST and gRPC for Feast Registry Server ([#5396](#5396)) ([fd1f448](fd1f448))
* Support DynamoDB as online store in Go feature server ([#5464](#5464)) ([40d25c6](40d25c6))
* Update Spark Compute read source node to be able to use other data sources ([#5445](#5445)) ([a93d300](a93d300))

### Reverts

* Feat: Add CLI, SDK, and API documentation page to Feast UI" ([#5341](#5341)) ([b492f14](b492f14)), closes [#5337](#5337)
* Revert "feat: Add s3 storage-based registry store in Go feature server" ([#5351](#5351)) ([d5d6766](d5d6766)), closes [#5336](#5336)
* Revert "fix: Update milvus connect function to work with remote instance" ([#5398](#5398)) ([434dd92](434dd92)), closes [#5382](#5382)
franciscojavierarceo pushed a commit that referenced this pull request Jul 1, 2025
@feast-ci-bot
chore(release): release 0.50.0
a9dfac3 
# [0.50.0](v0.49.0...v0.50.0) (2025-07-01)

### Bug Fixes

* Add asyncio to integration test ([#5418](#5418)) ([6765515](6765515))
* Add clickhouse to OFFLINE_STORE_CLASS_FOR_TYPE map ([#5251](#5251)) ([9ed2ffa](9ed2ffa))
* Add missing conn.commit() in SnowflakeOnlineStore.online_write_batch ([#5432](#5432)) ([a83dd85](a83dd85))
* Add transformers in required dependencies ([8cde460](8cde460))
* Allow custom annotations on Operator installed objects ([#5339](#5339)) ([44c7a76](44c7a76))
* Dask pulling of latest data ([#5229](#5229)) ([571d81f](571d81f))
* **dask:** preserve remote URIs (e.g. s3://) in DaskOfflineStore path resolution ([2561cfc](2561cfc))
* Fix Event loop is closed error on dynamodb test ([#5480](#5480)) ([fe0f671](fe0f671))
* Fix lineage entity filtering ([#5321](#5321)) ([0d05701](0d05701))
* Fix list saved dataset api ([833696c](833696c))
* Fix NumPy - PyArrow array type mapping in Trino offline store ([#5393](#5393)) ([9ba9ded](9ba9ded))
* Fix pandas 2.x compatibility issue of Trino offline store caused by removed Series.iteritems() method ([#5345](#5345)) ([61e3e02](61e3e02))
* Fix polling mechanism for TestApplyAndMaterialize ([#5451](#5451)) ([b512a74](b512a74))
* Fix remote rbac integration tests ([#5473](#5473)) ([10879ec](10879ec))
* Fix Trino offline store SQL in Jinja template ([#5346](#5346)) ([648c53d](648c53d))
* Fixed CurlGeneratorTab github theme type ([#5425](#5425)) ([5f15329](5f15329))
* Increase the Operator Manager memory limits and requests ([#5441](#5441)) ([6c94dbf](6c94dbf))
* Method signature for push_async is out of date ([#5413](#5413)) ([28c3379](28c3379)), closes [#5410](#5410) [#006BB4](https://github.com/feast-dev/feast/issues/006BB4)
* Operator - support securityContext override at Pod level ([#5325](#5325)) ([33ea0f5](33ea0f5))
* Pybuild-deps throws errors w/ latest pip version ([#5311](#5311)) ([f2d6a67](f2d6a67))
* Reopen for integration test about add s3 storage-based registry store in Go feature server ([#5352](#5352)) ([ef75f61](ef75f61))
* resolve Python logger warnings ([#5361](#5361)) ([37d5c19](37d5c19))
* The ignore_paths not taking effect duration feast apply ([#5353](#5353)) ([e4917ca](e4917ca))
* Update generate_answer function to provide correct parameter format to retrieve function ([dc5b2af](dc5b2af))
* Update milvus connect function to work with remote instance ([#5382](#5382)) ([7e5e7d5](7e5e7d5))
* Updating milvus connect function to work with remote instance ([#5401](#5401)) ([b89fadd](b89fadd))
* Upperbound limit for protobuf generation ([#5309](#5309)) ([a114aae](a114aae))

### Features

* Add CLI, SDK, and API documentation page to Feast UI ([#5337](#5337)) ([203e888](203e888))
* Add dark mode toggle to Feast UI ([#5314](#5314)) ([ad02e46](ad02e46))
* Add data labeling tabs to UI ([#5410](#5410)) ([389ceb7](389ceb7)), closes [#006BB4](https://github.com/feast-dev/feast/issues/006BB4)
* Add Decimal to allowed python scalar types ([#5367](#5367)) ([4777c03](4777c03))
* Add feast rag retriver functionality ([#5405](#5405)) ([0173033](0173033))
* Add feature view curl generator ([#5415](#5415)) ([7a5b48f](7a5b48f))
* Add feature view lineage tab and filtering to home page lineage ([#5308](#5308)) ([308255d](308255d))
* Add feature view tags to dynamo tags ([#5291](#5291)) ([3a787ac](3a787ac))
* Add HybridOnlineStore for multi-backend online store routing ([#5423](#5423)) ([ebd67d1](ebd67d1))
* Add max_file_size to Snowflake config ([#5377](#5377)) ([e8cdf5d](e8cdf5d))
* Add MCP (Model Context Protocol) support for Feast feature server ([#5406](#5406)) ([de650de](de650de)), closes [#5398](#5398) [#5382](#5382) [#5389](#5389) [#5401](#5401)
* Add rag project to default dev UI ([#5323](#5323)) ([3b3e1c8](3b3e1c8))
* Add s3 storage-based registry store in Go feature server ([#5336](#5336)) ([abe18df](abe18df))
* Add support for data labeling in UI ([#5409](#5409)) ([d183c4b](d183c4b)), closes [#27](#27)
* Added Lineage APIs to get registry objects relationships ([#5472](#5472)) ([be004ef](be004ef))
* Added rest-apis serving option for registry server ([#5342](#5342)) ([9740fd1](9740fd1))
* Added torch.Tensor as option for online and offline retrieval ([#5381](#5381)) ([0b4ae95](0b4ae95))
* Adding feast delete to CLI ([#5344](#5344)) ([19fe3ac](19fe3ac))
* Adding permissions to UI and refactoring some things ([#5320](#5320)) ([6f1b0cc](6f1b0cc))
* Allow to set registry server rest/grpc mode in operator ([#5364](#5364)) ([99afd6d](99afd6d))
* Allow to use env variable FEAST_FS_YAML_FILE_PATH and FEATURE_REPO_DIR ([#5420](#5420)) ([6a1b33a](6a1b33a))
* Enable materialization for ODFV Transform on Write ([#5459](#5459)) ([3d17892](3d17892))
* Improve search results formatting ([#5326](#5326)) ([18cbd7f](18cbd7f))
* Improvements to Lambda materialization engine ([#5379](#5379)) ([b486f29](b486f29))
* Make batch_source optional in PushSource ([#5440](#5440)) ([#5454](#5454)) ([ae7e20e](ae7e20e))
* Refactor materialization engine ([#5354](#5354)) ([f5c5360](f5c5360))
* Remote Write to Online Store completes client / server architecture ([#5422](#5422)) ([2368f42](2368f42))
* Serialization version 2 and below removed ([#5435](#5435)) ([9e50e18](9e50e18))
* SQLite online retrieval. Add timezone info into timestamp. ([#5386](#5386)) ([6b05153](6b05153))
* Support dual-mode REST and gRPC for Feast Registry Server ([#5396](#5396)) ([fd1f448](fd1f448))
* Support DynamoDB as online store in Go feature server ([#5464](#5464)) ([40d25c6](40d25c6))
* Update Spark Compute read source node to be able to use other data sources ([#5445](#5445)) ([a93d300](a93d300))

### Reverts

* Chore Release "chore(release): release 0.50.0" ([#5483](#5483)) ([0eef391](0eef391))
* Feat: Add CLI, SDK, and API documentation page to Feast UI" ([#5341](#5341)) ([b492f14](b492f14)), closes [#5337](#5337)
* Revert "feat: Add s3 storage-based registry store in Go feature server" ([#5351](#5351)) ([d5d6766](d5d6766)), closes [#5336](#5336)
* Revert "fix: Update milvus connect function to work with remote instance" ([#5398](#5398)) ([434dd92](434dd92)), closes [#5382](#5382)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@franciscojavierarceo franciscojavierarceo franciscojavierarceo approved these changes

Assignees

@ntkathole ntkathole

Labels
ok-to-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ntkathole @franciscojavierarceo