feat: Offline store update pull_all_from_table_or_query to make timestampfield optional #5281

HaoXuAI · 2025-04-18T06:00:10Z

What this PR does / why we need it:

The major change in this PR is making the start_date and end_date in pull_all_from_table_or_query optional:

    def pull_all_from_table_or_query(
        config: RepoConfig,
        data_source: DataSource,
        join_key_columns: List[str],
        feature_name_columns: List[str],
        timestamp_field: str,
        created_timestamp_field: Optional[str] = None,
        start_date: Optional[datetime] = None,
        end_date: Optional[datetime] = None,
    ) -> RetrievalJob:

The reason is the API should be able to let the user skip the time range selection, and return all entities with features.
This will also make the current materialization and get_historical_features in Compute Engine to be able to use this API.
The change is non-breaking, and the API currently is only used in retrieve_feature_service_logs, and retrieve_saved_dataset.

Which issue(s) this PR fixes:

Misc

Signed-off-by: HaoXuAI <[email protected]>

HaoXuAI · 2025-04-18T06:48:07Z

sdk/python/feast/infra/compute_engines/local/feature_builder.py

@@ -95,25 +94,3 @@ def build_output_nodes(self, input_node):
        node = LocalOutputNode("output")
        node.add_input(input_node)
        self.nodes.append(node)
-
-    def build(self) -> ExecutionPlan:


we can remove it as it is the same with the Parent build

HaoXuAI · 2025-04-18T06:48:30Z

sdk/python/feast/infra/common/retrieval_task.py

@@ -15,5 +15,5 @@ class HistoricalRetrievalTask:
    feature_view: Union[BatchFeatureView, StreamFeatureView]
    full_feature_name: bool
    registry: Registry
-    start_time: datetime
-    end_time: datetime
+    start_time: Optional[datetime] = None


retrieval job doesn't have to provide start or end date

HaoXuAI · 2025-04-18T06:48:55Z

sdk/python/feast/infra/compute_engines/feature_builder.py

-        if isinstance(self.task, HistoricalRetrievalTask):
-            last_node = self.build_join_node(last_node)
+        # Join entity_df with source if needed
+        last_node = self.build_join_node(last_node)


without entity_df provided, the join node is just a pass through node

HaoXuAI · 2025-04-18T06:49:33Z

sdk/python/feast/infra/compute_engines/local/nodes.py

+        ) = context.column_info
+
+        # 📥 Reuse Feast's robust query resolver
+        retrieval_job = offline_store.pull_all_from_table_or_query(


reuse the pull_all_from_table_or_query to read data from all feast offline_store now

HaoXuAI · 2025-04-18T06:50:17Z

sdk/python/feast/infra/offline_stores/bigquery.py

@@ -206,10 +207,13 @@ def pull_all_from_table_or_query(
            + BigQueryOfflineStore._escape_query_columns(feature_name_columns)
            + [timestamp_field]
        )
+        timestamp_filter = get_timestamp_filter_sql(


major change: get the timestamp filter from get_timestamp_filter_sql

Signed-off-by: HaoXuAI <[email protected]>

# [0.49.0](v0.48.0...v0.49.0) (2025-04-29) ### Bug Fixes * Adding brackets to unit tests ([c46fea3](c46fea3)) * Adding logic back for a step ([2bb240b](2bb240b)) * Adjustment for unit test action ([a6f78ae](a6f78ae)) * Allow get_historical_features with only On Demand Feature View ([#5256](#5256)) ([0752795](0752795)) * CI adjustment ([3850643](3850643)) * Embed Query configuration breaks when switching between DataFrame and SQL ([#5257](#5257)) ([32375a5](32375a5)) * Fix for proto issue in utils ([1b291b2](1b291b2)) * Fix milvus online_read ([#5233](#5233)) ([4b91f26](4b91f26)) * Fix tests ([431d9b8](431d9b8)) * Fixed Permissions object parameter in example ([#5259](#5259)) ([045c100](045c100)) * Java CI [#12](#12) ([d7e44ac](d7e44ac)) * Java PR [#15](#15) ([a5da3bb](a5da3bb)) * Java PR [#16](#16) ([e0320fe](e0320fe)) * Java PR [#17](#17) ([49da810](49da810)) * Materialization logs ([#5243](#5243)) ([4aa2f49](4aa2f49)) * Moving to custom github action for checking skip tests ([caf312e](caf312e)) * Operator - remove default replicas setting from Feast Deployment ([#5294](#5294)) ([e416d01](e416d01)) * Patch java pr [#14](#14) ([592526c](592526c)) * Patch update for test ([a3e8967](a3e8967)) * Remove conditional from steps ([995307f](995307f)) * Remove misleading HTTP prefix from gRPC endpoints in logs and doc ([#5280](#5280)) ([0ee3a1e](0ee3a1e)) * removing id ([268ade2](268ade2)) * Renaming workflow file ([5f46279](5f46279)) * Resolve `no pq wrapper` import issue ([#5240](#5240)) ([d5906f1](d5906f1)) * Update actions to remove check skip tests ([#5275](#5275)) ([b976f27](b976f27)) * Update docling demo ([446efea](446efea)) * Update java pr [#13](#13) ([fda7db7](fda7db7)) * Update java_pr ([fa138f4](fa138f4)) * Update repo_config.py ([6a59815](6a59815)) * Update unit tests workflow ([06486a0](06486a0)) * Updated docs for docling demo ([768e6cc](768e6cc)) * Updating action for unit tests ([0996c28](0996c28)) * Updating github actions to filter at job level ([0a09622](0a09622)) * Updating Java CI ([c7c3a3c](c7c3a3c)) * Updating java pr to skip tests ([e997dd9](e997dd9)) * Updating workflows ([c66bcd2](c66bcd2)) ### Features * Add date_partition_column_format for spark source ([#5273](#5273)) ([7a61d6f](7a61d6f)) * Add Milvus tutorial with Feast integration ([#5292](#5292)) ([a1388a5](a1388a5)) * Add pgvector tutorial with PostgreSQL integration ([#5290](#5290)) ([bb1cbea](bb1cbea)) * Add ReactFlow visualization for Feast registry metadata ([#5297](#5297)) ([9768970](9768970)) * Add retrieve online documents v2 method into pgvector ([#5253](#5253)) ([6770ee6](6770ee6)) * Compute Engine Initial Implementation ([#5223](#5223)) ([64bdafd](64bdafd)) * Enable write node for compute engine ([#5287](#5287)) ([f9baf97](f9baf97)) * Local compute engine ([#5278](#5278)) ([8e06dfe](8e06dfe)) * Make transform on writes configurable for ingestion ([#5283](#5283)) ([ecad170](ecad170)) * Offline store update pull_all_from_table_or_query to make timestampfield optional ([#5281](#5281)) ([4b94608](4b94608)) * Serialization version 2 deprecation notice ([#5248](#5248)) ([327d99d](327d99d)) * Vector length definition moved to Feature View from Config ([#5289](#5289)) ([d8f1c97](d8f1c97))

…tampfield optional (feast-dev#5281) * Update offline store pull all API date field optional Signed-off-by: HaoXuAI <[email protected]> * Update offline store pull all API date field optional Signed-off-by: HaoXuAI <[email protected]> * update Signed-off-by: HaoXuAI <[email protected]> * update Signed-off-by: HaoXuAI <[email protected]> * update Signed-off-by: HaoXuAI <[email protected]> * fix test Signed-off-by: HaoXuAI <[email protected]> * update source read node Signed-off-by: HaoXuAI <[email protected]> * fix linting Signed-off-by: HaoXuAI <[email protected]> * fix linting Signed-off-by: HaoXuAI <[email protected]> * fix linting Signed-off-by: HaoXuAI <[email protected]> * fix linting Signed-off-by: HaoXuAI <[email protected]> * fix linting Signed-off-by: HaoXuAI <[email protected]> * fix test Signed-off-by: HaoXuAI <[email protected]> * fix test Signed-off-by: HaoXuAI <[email protected]> * fix test Signed-off-by: HaoXuAI <[email protected]> * fix test Signed-off-by: HaoXuAI <[email protected]> * fix test Signed-off-by: HaoXuAI <[email protected]> * fix test Signed-off-by: HaoXuAI <[email protected]> * fix test Signed-off-by: HaoXuAI <[email protected]> * fix test Signed-off-by: HaoXuAI <[email protected]> --------- Signed-off-by: HaoXuAI <[email protected]> Signed-off-by: Jacob Weinhold <[email protected]>

# [0.49.0](feast-dev/feast@v0.48.0...v0.49.0) (2025-04-29) ### Bug Fixes * Adding brackets to unit tests ([c46fea3](feast-dev@c46fea3)) * Adding logic back for a step ([2bb240b](feast-dev@2bb240b)) * Adjustment for unit test action ([a6f78ae](feast-dev@a6f78ae)) * Allow get_historical_features with only On Demand Feature View ([feast-dev#5256](feast-dev#5256)) ([0752795](feast-dev@0752795)) * CI adjustment ([3850643](feast-dev@3850643)) * Embed Query configuration breaks when switching between DataFrame and SQL ([feast-dev#5257](feast-dev#5257)) ([32375a5](feast-dev@32375a5)) * Fix for proto issue in utils ([1b291b2](feast-dev@1b291b2)) * Fix milvus online_read ([feast-dev#5233](feast-dev#5233)) ([4b91f26](feast-dev@4b91f26)) * Fix tests ([431d9b8](feast-dev@431d9b8)) * Fixed Permissions object parameter in example ([feast-dev#5259](feast-dev#5259)) ([045c100](feast-dev@045c100)) * Java CI [feast-dev#12](feast-dev#12) ([d7e44ac](feast-dev@d7e44ac)) * Java PR [feast-dev#15](feast-dev#15) ([a5da3bb](feast-dev@a5da3bb)) * Java PR [feast-dev#16](feast-dev#16) ([e0320fe](feast-dev@e0320fe)) * Java PR [feast-dev#17](feast-dev#17) ([49da810](feast-dev@49da810)) * Materialization logs ([feast-dev#5243](feast-dev#5243)) ([4aa2f49](feast-dev@4aa2f49)) * Moving to custom github action for checking skip tests ([caf312e](feast-dev@caf312e)) * Operator - remove default replicas setting from Feast Deployment ([feast-dev#5294](feast-dev#5294)) ([e416d01](feast-dev@e416d01)) * Patch java pr [feast-dev#14](feast-dev#14) ([592526c](feast-dev@592526c)) * Patch update for test ([a3e8967](feast-dev@a3e8967)) * Remove conditional from steps ([995307f](feast-dev@995307f)) * Remove misleading HTTP prefix from gRPC endpoints in logs and doc ([feast-dev#5280](feast-dev#5280)) ([0ee3a1e](feast-dev@0ee3a1e)) * removing id ([268ade2](feast-dev@268ade2)) * Renaming workflow file ([5f46279](feast-dev@5f46279)) * Resolve `no pq wrapper` import issue ([feast-dev#5240](feast-dev#5240)) ([d5906f1](feast-dev@d5906f1)) * Update actions to remove check skip tests ([feast-dev#5275](feast-dev#5275)) ([b976f27](feast-dev@b976f27)) * Update docling demo ([446efea](feast-dev@446efea)) * Update java pr [feast-dev#13](feast-dev#13) ([fda7db7](feast-dev@fda7db7)) * Update java_pr ([fa138f4](feast-dev@fa138f4)) * Update repo_config.py ([6a59815](feast-dev@6a59815)) * Update unit tests workflow ([06486a0](feast-dev@06486a0)) * Updated docs for docling demo ([768e6cc](feast-dev@768e6cc)) * Updating action for unit tests ([0996c28](feast-dev@0996c28)) * Updating github actions to filter at job level ([0a09622](feast-dev@0a09622)) * Updating Java CI ([c7c3a3c](feast-dev@c7c3a3c)) * Updating java pr to skip tests ([e997dd9](feast-dev@e997dd9)) * Updating workflows ([c66bcd2](feast-dev@c66bcd2)) ### Features * Add date_partition_column_format for spark source ([feast-dev#5273](feast-dev#5273)) ([7a61d6f](feast-dev@7a61d6f)) * Add Milvus tutorial with Feast integration ([feast-dev#5292](feast-dev#5292)) ([a1388a5](feast-dev@a1388a5)) * Add pgvector tutorial with PostgreSQL integration ([feast-dev#5290](feast-dev#5290)) ([bb1cbea](feast-dev@bb1cbea)) * Add ReactFlow visualization for Feast registry metadata ([feast-dev#5297](feast-dev#5297)) ([9768970](feast-dev@9768970)) * Add retrieve online documents v2 method into pgvector ([feast-dev#5253](feast-dev#5253)) ([6770ee6](feast-dev@6770ee6)) * Compute Engine Initial Implementation ([feast-dev#5223](feast-dev#5223)) ([64bdafd](feast-dev@64bdafd)) * Enable write node for compute engine ([feast-dev#5287](feast-dev#5287)) ([f9baf97](feast-dev@f9baf97)) * Local compute engine ([feast-dev#5278](feast-dev#5278)) ([8e06dfe](feast-dev@8e06dfe)) * Make transform on writes configurable for ingestion ([feast-dev#5283](feast-dev#5283)) ([ecad170](feast-dev@ecad170)) * Offline store update pull_all_from_table_or_query to make timestampfield optional ([feast-dev#5281](feast-dev#5281)) ([4b94608](feast-dev@4b94608)) * Serialization version 2 deprecation notice ([feast-dev#5248](feast-dev#5248)) ([327d99d](feast-dev@327d99d)) * Vector length definition moved to Feature View from Config ([feast-dev#5289](feast-dev#5289)) ([d8f1c97](feast-dev@d8f1c97)) Signed-off-by: Jacob Weinhold <[email protected]>

HaoXuAI added 2 commits April 17, 2025 22:41

Update offline store pull all API date field optional

112014e

Signed-off-by: HaoXuAI <[email protected]>

Update offline store pull all API date field optional

c75c264

Signed-off-by: HaoXuAI <[email protected]>

HaoXuAI requested review from sudohainguyen and a team as code owners April 18, 2025 06:00

HaoXuAI requested review from shuchu, ejscribner and tokoko April 18, 2025 06:00

HaoXuAI added the ok-to-test label Apr 18, 2025

HaoXuAI added 5 commits April 17, 2025 23:10

update

b62fd1a

Signed-off-by: HaoXuAI <[email protected]>

update

8b5bdae

Signed-off-by: HaoXuAI <[email protected]>

update

e92d015

Signed-off-by: HaoXuAI <[email protected]>

fix test

4765d8f

Signed-off-by: HaoXuAI <[email protected]>

update source read node

0bbdbbc

Signed-off-by: HaoXuAI <[email protected]>

HaoXuAI commented Apr 18, 2025

View reviewed changes

HaoXuAI added 11 commits April 17, 2025 23:51

fix linting

0336318

Signed-off-by: HaoXuAI <[email protected]>

fix linting

ac03a75

Signed-off-by: HaoXuAI <[email protected]>

fix linting

0033b9c

Signed-off-by: HaoXuAI <[email protected]>

fix linting

743a490

Signed-off-by: HaoXuAI <[email protected]>

fix linting

4285b5f

Signed-off-by: HaoXuAI <[email protected]>

fix test

618c4b8

Signed-off-by: HaoXuAI <[email protected]>

fix test

afcaf9c

Signed-off-by: HaoXuAI <[email protected]>

fix test

8e02747

Signed-off-by: HaoXuAI <[email protected]>

fix test

71daec2

Signed-off-by: HaoXuAI <[email protected]>

fix test

23a0fb4

Signed-off-by: HaoXuAI <[email protected]>

fix test

bcf1d7a

Signed-off-by: HaoXuAI <[email protected]>

HaoXuAI added 2 commits April 18, 2025 13:48

fix test

89ad0da

Signed-off-by: HaoXuAI <[email protected]>

fix test

cdae95b

Signed-off-by: HaoXuAI <[email protected]>

franciscojavierarceo approved these changes Apr 20, 2025

View reviewed changes

franciscojavierarceo merged commit 4b94608 into master Apr 20, 2025
23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Offline store update pull_all_from_table_or_query to make timestampfield optional #5281

feat: Offline store update pull_all_from_table_or_query to make timestampfield optional #5281

Uh oh!

HaoXuAI commented Apr 18, 2025 •

edited

Loading

Uh oh!

HaoXuAI Apr 18, 2025

Uh oh!

HaoXuAI Apr 18, 2025

Uh oh!

HaoXuAI Apr 18, 2025

Uh oh!

HaoXuAI Apr 18, 2025

Uh oh!

HaoXuAI Apr 18, 2025

Uh oh!

Uh oh!

Uh oh!

feat: Offline store update pull_all_from_table_or_query to make timestampfield optional #5281

feat: Offline store update pull_all_from_table_or_query to make timestampfield optional #5281

Uh oh!

Conversation

HaoXuAI commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Misc

Uh oh!

HaoXuAI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

HaoXuAI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

HaoXuAI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

HaoXuAI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

HaoXuAI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HaoXuAI commented Apr 18, 2025 •

edited

Loading