Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Fixing transformations on writes #5127

Merged
merged 28 commits into from
Mar 9, 2025
Merged

fix: Fixing transformations on writes #5127

merged 28 commits into from
Mar 9, 2025

Conversation

franciscojavierarceo
Copy link
Member

@franciscojavierarceo franciscojavierarceo commented Mar 9, 2025 •
edited
Loading

What this PR does / why we need it:

Fixes transformations on writes. They did not actually work before. 🤦‍♂️ 😭

  • sdk/python/feast/feature_store.py

    • Import new transformation modules: PandasTransformation and PythonTransformation
    • Apply transformations for OnDemandFeatureViews with write_to_online_store set to True
    • Update retrieve_online_documents_v2 method to handle OnDemandFeatureViews

  • sdk/python/feast/feature_view.py

    • Update features field to be a list of features instead of fields in the to_proto method

  • sdk/python/feast/infra/online_stores/milvus_online_store/milvus.py

    • Update condition to check for vector fields in the schema

  • sdk/python/feast/infra/online_stores/sqlite.py

    • Refactor online_write_batch to handle vector types properly
    • Optimize online_read by pre-serializing entity keys

  • sdk/python/feast/infra/passthrough_provider.py

    • Update type hinting for feature_views to include OnDemandFeatureView

  • sdk/python/feast/infra/provider.py

    • Update type hinting for feature_views to include OnDemandFeatureView

  • sdk/python/feast/on_demand_feature_view.py

    • Add fields vector_index and vector_search_metric to features
    • Update infer_features method to handle singleton and array types
    • Add explode parameter to on_demand_feature_view decorator

  • sdk/python/feast/transformation/pandas_transformation.py

    • Update infer_features method to accept singleton parameter

  • sdk/python/feast/transformation/python_transformation.py

    • Update infer_features method to accept singleton parameter

  • sdk/python/feast/transformation/substrait_transformation.py

    • Update infer_features method to accept singleton parameter

  • sdk/python/feast/utils.py

    • Ensure join keys are included in proto values when converting Arrow ODFV to proto
    • Update _group_feature_refs to handle OnDemandFeatureViews with write_to_online_store
    • Update _augment_response_with_on_demand_transforms to skip ODFVs with write_to_online_store
    • Update _prepare_entities_to_read_from_online_store to handle entities properly

  • sdk/python/tests/unit/infra/test_inference_unit_tests.py

    • Remove invalid test case for Python mode in OnDemandFeatureView

  • sdk/python/tests/unit/test_on_demand_python_transformation.py

    • Add tests for OnDemandFeatureView with write_to_online_store and explode parameter

Which issue(s) this PR fixes:

#5128

Misc

@franciscojavierarceo
feat: Adding Docling RAG demo
e38a7c0 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updated demo
f9c7db4 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
cleaned up notebook
4ba9fce 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
adding chunk id
6c3ea35 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
adding quickstart demo that is WIP and updating docling-demo to expor…
410ded0 
…t unique chunk-id

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
adding current tentative exmaple repo
8f0c663 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
adding current temporary work
f098674 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updating to handle missed edge case
99dc5f3 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
linter
29ca46f 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updating demo script to rename things
3f23d25 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updated quickstart
d366988 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
added comment
f917183 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
checking in progress
e8b9e4a 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updating with progress...found large bug
521abb5 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
almost have something working...very close
731d6f1 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
have writes behaving, now need reads
f0d396d 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
fixed writes and reads and unit test
86b09c5 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
got write test working (not explode though)
cbcbe70 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
checking in progress for now, still have some issues with vector retr…
48fba5f 
…ieval

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
okay think i have most things working
fef8666 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
unit tests are passing...still need to lint but checking in for tonight
8626762 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
fixed unit test and linter...i believe
3cd0093 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
removing commenting and unnecessary code
ffac095 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
forgot to add file for unit tests
2e9d2d5 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
skipping test for some
94c60d6 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo franciscojavierarceo requested review from a team as code owners March 9, 2025 02:30
@franciscojavierarceo
removing rag docling demo from this branch
fabd394 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
addding skip test back in for milvus
e545e4b 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
removed print statement
548e0ce 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo franciscojavierarceo merged commit 95ac34a into master Mar 9, 2025
34 checks passed
franciscojavierarceo pushed a commit that referenced this pull request Mar 10, 2025
@feast-ci-bot
chore(release): release 0.47.0
3e3afa9 
# [0.47.0](v0.46.0...v0.47.0) (2025-03-10)

* feat!: Include PUBLIC_URL in defaultProjectListPromise URL in /ui ([2f0f7b3](2f0f7b3))

### Bug Fixes

* Add transformation_service_endpoit to support Go feature server. ([#5071](#5071)) ([5627d7c](5627d7c))
* Adding extra space on the VM to kind cluster to see if this solves the issue with memory not available with operator e2e tests. ([#5102](#5102)) ([e6e928c](e6e928c))
* Allow unencrypted Snowflake key ([#5097](#5097)) ([87a7c23](87a7c23))
* Cant add different type of list types ([#5118](#5118)) ([bebd7be](bebd7be))
* Fixing transformations on writes ([#5127](#5127)) ([95ac34a](95ac34a))
* Identify s3/remote uri path correctly ([#5076](#5076)) ([93becff](93becff))
* Increase available action VM storage and reduce dev feature-server image size ([#5112](#5112)) ([75f5a90](75f5a90))
* Move Feast to pyproject.toml instead of setup.py ([#5067](#5067)) ([4231274](4231274))
* Skip refresh if already in progress or if lock is already held ([#5068](#5068)) ([f3a24de](f3a24de))

### Features

* Add an OOTB Chat uI to the Feature Server to support RAG demo ([#5106](#5106)) ([40ea7a9](40ea7a9))
* Add Couchbase Columnar as an Offline Store ([#5025](#5025)) ([4373cbf](4373cbf))
* Add Feast Operator RBAC example with Kubernetes Authentication … ([#5077](#5077)) ([2179fbe](2179fbe))
* Added docling and pytorch as add on ([#5089](#5089)) ([135342b](135342b))
* Feast Operator example with Postgres in TLS mode. ([#5028](#5028)) ([2c46f6a](2c46f6a))
* Operator - Add feastProjectDir section to CR with git & init options ([#5079](#5079)) ([d64f01e](d64f01e))
* Override the udf name when provided as input to an on demand transformation ([#5094](#5094)) ([8a714bb](8a714bb))
* Set value_type of entity directly in from_proto ([#5092](#5092)) ([90e7498](90e7498))
* Updating retrieve online documents v2 to work for other fields for sq… ([#5082](#5082)) ([fc121c3](fc121c3))

### BREAKING CHANGES

* The PUBLIC_URL environment variable is now taken into account by default
when fetching the projects list. This is a breaking change only if all
these points apply:

1. You're using Feast UI as a module

2. You're serving the UI files from a non-root path via the PUBLIC_URL
   environment variable

3. You're serving the project list from the root path

4. You're not passing the `feastUIConfigs.projectListPromise` prop to
   the FeastUI component

In this case, you need to explicitly fetch the project list from the
root path via the `feastUIConfigs.projectListPromise` prop:

```diff
 const root = createRoot(document.getElementById("root")!);
 root.render(
   <React.StrictMode>
-    <FeastUI />
+    <FeastUI
+      feastUIConfigs={{
+        projectListPromise: fetch("/projects-list.json", {
+            headers: {
+              "Content-Type": "application/json",
+            },
+          }).then((res) => res.json())
+      }}
+    />
   </React.StrictMode>
 );
```

Signed-off-by: Harri Lehtola <peruukki@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shuchu shuchu shuchu approved these changes

@tokoko tokoko Awaiting requested review from tokoko tokoko was automatically assigned from feast-dev/maintainers

@HaoXuAI HaoXuAI Awaiting requested review from HaoXuAI HaoXuAI was automatically assigned from feast-dev/maintainers

Assignees
No one assigned
Labels
ok-to-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@franciscojavierarceo @shuchu