Skip to content

Commit 31afd99

Browse files
dmartinolfranciscojavierarceo
dmartinol
and
franciscojavierarceo
authored
fix: Updated python-helm-demo example to use MinIO instead of GS (feast-dev#4691)
* Updated python-helm-demo example to use MinIO instead of GS Signed-off-by: Daniele Martinoli <dmartino@redhat.com> * Update examples/python-helm-demo/README.md Co-authored-by: Francisco Arceo <farceo@redhat.com> Signed-off-by: Daniele Martinoli <dmartino@redhat.com> * Adding explicit wait to container to validate CI failures Signed-off-by: Daniele Martinoli <dmartino@redhat.com> * restored original conftest Signed-off-by: Daniele Martinoli <dmartino@redhat.com> --------- Signed-off-by: Daniele Martinoli <dmartino@redhat.com> Co-authored-by: Francisco Arceo <farceo@redhat.com>
1 parent 132ce2a commit 31afd99

9 files changed

+287
-50
lines changed

‎examples/python-helm-demo/README.md

+120-39
Original file line numberDiff line numberDiff line change
@@ -3,87 +3,168 @@
33

44
For this tutorial, we set up Feast with Redis.
55

6-
We use the Feast CLI to register and materialize features, and then retrieving via a Feast Python feature server deployed in Kubernetes
6+
We use the Feast CLI to register and materialize features from the current machine, and then retrieving via a
7+
Feast Python feature server deployed in Kubernetes
78

89
## First, let's set up a Redis cluster
910
1. Start minikube (`minikube start`)
10-
2. Use helm to install a default Redis cluster
11+
1. Use helm to install a default Redis cluster
1112
```bash
1213
helm repo add bitnami https://charts.bitnami.com/bitnami
1314
helm repo update
1415
helm install my-redis bitnami/redis
1516
```
1617
![](redis-screenshot.png)
17-
3. Port forward Redis so we can materialize features to it
18+
1. Port forward Redis so we can materialize features to it
1819

1920
```bash
2021
kubectl port-forward --namespace default svc/my-redis-master 6379:6379
2122
```
22-
4. Get your Redis password using the command (pasted below for convenience). We'll need this to tell Feast how to communicate with the cluster.
23+
1. Get your Redis password using the command (pasted below for convenience). We'll need this to tell Feast how to communicate with the cluster.
2324
2425
```bash
2526
export REDIS_PASSWORD=$(kubectl get secret --namespace default my-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
2627
echo $REDIS_PASSWORD
2728
```
2829
30+
## Then, let's set up a MinIO S3 store
31+
Manifests have been taken from [Deploy Minio in your project](https://ai-on-openshift.io/tools-and-applications/minio/minio/#deploy-minio-in-your-project).
32+
33+
1. Deploy MinIO instance:
34+
```
35+
kubectl apply -f minio-dev.yaml
36+
```
37+
38+
1. Forward the UI port:
39+
```console
40+
kubectl port-forward svc/minio-service 9090:9090
41+
```
42+
1. Login to (localhost:9090)[http://localhost:9090] as `minio`/`minio123` and create bucket called `feast-demo`.
43+
1. Stop previous port forwarding and forward the API port instead:
44+
```console
45+
kubectl port-forward svc/minio-service 9000:9000
46+
```
47+
2948
## Next, we setup a local Feast repo
30-
1. Install Feast with Redis dependencies `pip install "feast[redis]"`
31-
2. Make a bucket in GCS (or S3)
32-
3. The feature repo is already setup here, so you just need to swap in your GCS bucket and Redis credentials.
33-
We need to modify the `feature_store.yaml`, which has two fields for you to replace:
49+
1. Install Feast with Redis dependencies `pip install "feast[redis,aws]"`
50+
1. The feature repo is already setup here, so you just need to swap in your Redis credentials.
51+
We need to modify the `feature_store.yaml`, which has one field for you to replace:
52+
```console
53+
sed "s/_REDIS_PASSWORD_/${REDIS_PASSWORD}/" feature_repo/feature_store.yaml.template > feature_repo/feature_store.yaml
54+
cat feature_repo/feature_store.yaml
55+
```
56+
57+
Example repo:
3458
```yaml
35-
registry: gs://[YOUR GCS BUCKET]/demo-repo/registry.db
59+
registry: s3://localhost:9000/feast-demo/registry.db
3660
project: feast_python_demo
37-
provider: gcp
61+
provider: local
3862
online_store:
3963
type: redis
40-
# Note: this would normally be using instance URL's to access Redis
41-
connection_string: localhost:6379,password=[YOUR PASSWORD]
64+
connection_string: localhost:6379,password=****
4265
offline_store:
4366
type: file
4467
entity_key_serialization_version: 2
4568
```
46-
4. Run `feast apply` from within the `feature_repo` directory to apply your local features to the remote registry
47-
- Note: you may need to authenticate to gcloud first with `gcloud auth login`
48-
5. Materialize features to the online store:
69+
1. To run `feast apply` from the current machine we need to define the AWS credentials to connect the MinIO S3 store, which
70+
are defined in [minio.env](./minio.env):
71+
```console
72+
source minio.env
73+
cd feature_repo
74+
feast apply
75+
```
76+
1. Let's validate the setup by running some queries
77+
```console
78+
feast entities list
79+
feast feature-views list
80+
```
81+
1. Materialize features to the online store:
4982
```bash
83+
cd feature_repo
5084
CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
5185
feast materialize-incremental $CURRENT_TIME
5286
```
5387
5488
## Now let's setup the Feast Server
55-
1. Add the gcp-auth addon to mount GCP credentials:
56-
```bash
57-
minikube addons enable gcp-auth
58-
```
59-
2. Add Feast's Python/Go feature server chart repo
89+
1. Add Feast's Python feature server chart repo
6090
```bash
6191
helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
6292
helm repo update
6393
```
64-
3. For this tutorial, because we don't have a direct hosted endpoint into Redis, we need to change `feature_store.yaml` to talk to the Kubernetes Redis service
65-
```bash
66-
sed -i '' 's/localhost:6379/my-redis-master:6379/g' feature_store.yaml
67-
```
68-
4. Install the Feast helm chart: `helm install feast-release feast-charts/feast-feature-server --set feature_store_yaml_base64=$(base64 feature_store.yaml)`
69-
> **Dev instructions**: if you're changing the java logic or chart, you can do
70-
1. `eval $(minikube docker-env)`
71-
2. `make build-feature-server-dev`
72-
3. `helm install feast-release ../../../infra/charts/feast-feature-server --set image.tag=dev --set feature_store_yaml_base64=$(base64 feature_store.yaml)`
73-
5. (Optional): check logs of the server to make sure it’s working
94+
1. For this tutorial, we'll use a predefined configuration where we just needs to inject the Redis service password:
95+
```console
96+
sed "s/_REDIS_PASSWORD_/$REDIS_PASSWORD/" online_feature_store.yaml.template > online_feature_store.yaml
97+
cat online_feature_store.yaml
98+
```
99+
As you see, the connection points to `my-redis-master:6379` instead of `localhost:6379`.
100+
101+
1. Install the Feast helm chart:
102+
```console
103+
helm upgrade --install feast-online feast-charts/feast-feature-server \
104+
--set fullnameOverride=online-server --set feast_mode=online \
105+
--set feature_store_yaml_base64=$(base64 -i 'online_feature_store.yaml')
106+
```
107+
1. Patch the deployment to include MinIO settings:
108+
```console
109+
kubectl patch deployment online-server --type='json' -p='[
110+
{
111+
"op": "add",
112+
"path": "/spec/template/spec/containers/0/env/-",
113+
"value": {
114+
"name": "AWS_ACCESS_KEY_ID",
115+
"value": "minio"
116+
}
117+
},
118+
{
119+
"op": "add",
120+
"path": "/spec/template/spec/containers/0/env/-",
121+
"value": {
122+
"name": "AWS_SECRET_ACCESS_KEY",
123+
"value": "minio123"
124+
}
125+
},
126+
{
127+
"op": "add",
128+
"path": "/spec/template/spec/containers/0/env/-",
129+
"value": {
130+
"name": "AWS_DEFAULT_REGION",
131+
"value": "default"
132+
}
133+
},
134+
{
135+
"op": "add",
136+
"path": "/spec/template/spec/containers/0/env/-",
137+
"value": {
138+
"name": "FEAST_S3_ENDPOINT_URL",
139+
"value": "http://minio-service:9000"
140+
}
141+
}
142+
]'
143+
kubectl wait --for=condition=available deployment/online-server --timeout=2m
144+
```
145+
1. (Optional): check logs of the server to make sure it’s working
74146
```bash
75-
kubectl logs svc/feast-release-feast-feature-server
147+
kubectl logs svc/online-server
76148
```
77-
6. Port forward to expose the grpc endpoint:
149+
1. Port forward to expose the grpc endpoint:
78150
```bash
79-
kubectl port-forward svc/feast-release-feast-feature-server 6566:80
151+
kubectl port-forward svc/online-server 6566:80
80152
```
81-
7. Run test fetches for online features:8.
82-
- First: change back the Redis connection string to allow localhost connections to Redis
153+
1. Run test fetches for online features:8.
83154
```bash
84-
sed -i '' 's/my-redis-master:6379/localhost:6379/g' feature_store.yaml
155+
source minio.env
156+
cd test
157+
python test_python_fetch.py
85158
```
86-
- Then run the included fetch script, which fetches both via the HTTP endpoint and for comparison, via the Python SDK
87-
```bash
88-
python test_python_fetch.py
159+
160+
Output example:
161+
```console
162+
--- Online features with SDK ---
163+
WARNING:root:_list_feature_views will make breaking changes. Please use _list_batch_feature_views instead. _list_feature_views will behave like _list_all_feature_views in the future.
164+
conv_rate : [0.6799587607383728, 0.9761165976524353]
165+
driver_id : [1001, 1002]
166+
167+
--- Online features with HTTP endpoint ---
168+
conv_rate : [0.67995876 0.9761166 ]
169+
driver_id : [1001 1002]
89170
```

‎examples/python-helm-demo/feature_repo/data/driver_stats_with_string.parquet

383 Bytes
Binary file not shown.

‎examples/python-helm-demo/feature_repo/feature_store.yaml

-10
This file was deleted.

‎examples/python-helm-demo/feature_repo/feature_store.yaml.template

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
registry: s3://feast-demo/registry.db
2+
project: feast_python_demo
3+
provider: local
4+
online_store:
5+
type: redis
6+
connection_string: localhost:6379,password=_REDIS_PASSWORD_
7+
offline_store:
8+
type: file
9+
entity_key_serialization_version: 2

‎examples/python-helm-demo/minio-dev.yaml

+128
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
---
2+
kind: PersistentVolumeClaim
3+
apiVersion: v1
4+
metadata:
5+
name: minio-pvc
6+
spec:
7+
accessModes:
8+
- ReadWriteOnce
9+
resources:
10+
requests:
11+
storage: 20Gi
12+
volumeMode: Filesystem
13+
---
14+
kind: Secret
15+
apiVersion: v1
16+
metadata:
17+
name: minio-secret
18+
stringData:
19+
# change the username and password to your own values.
20+
# ensure that the user is at least 3 characters long and the password at least 8
21+
minio_root_user: minio
22+
minio_root_password: minio123
23+
---
24+
kind: Deployment
25+
apiVersion: apps/v1
26+
metadata:
27+
name: minio
28+
spec:
29+
replicas: 1
30+
selector:
31+
matchLabels:
32+
app: minio
33+
template:
34+
metadata:
35+
labels:
36+
app: minio
37+
spec:
38+
volumes:
39+
- name: data
40+
persistentVolumeClaim:
41+
claimName: minio-pvc
42+
containers:
43+
- resources:
44+
limits:
45+
cpu: 250m
46+
memory: 1Gi
47+
requests:
48+
cpu: 20m
49+
memory: 100Mi
50+
readinessProbe:
51+
tcpSocket:
52+
port: 9000
53+
initialDelaySeconds: 5
54+
timeoutSeconds: 1
55+
periodSeconds: 5
56+
successThreshold: 1
57+
failureThreshold: 3
58+
terminationMessagePath: /dev/termination-log
59+
name: minio
60+
livenessProbe:
61+
tcpSocket:
62+
port: 9000
63+
initialDelaySeconds: 30
64+
timeoutSeconds: 1
65+
periodSeconds: 5
66+
successThreshold: 1
67+
failureThreshold: 3
68+
env:
69+
- name: MINIO_ROOT_USER
70+
valueFrom:
71+
secretKeyRef:
72+
name: minio-secret
73+
key: minio_root_user
74+
- name: MINIO_ROOT_PASSWORD
75+
valueFrom:
76+
secretKeyRef:
77+
name: minio-secret
78+
key: minio_root_password
79+
ports:
80+
- containerPort: 9000
81+
protocol: TCP
82+
- containerPort: 9090
83+
protocol: TCP
84+
imagePullPolicy: IfNotPresent
85+
volumeMounts:
86+
- name: data
87+
mountPath: /data
88+
subPath: minio
89+
terminationMessagePolicy: File
90+
image: >-
91+
quay.io/minio/minio:RELEASE.2023-06-19T19-52-50Z
92+
args:
93+
- server
94+
- /data
95+
- --console-address
96+
- :9090
97+
restartPolicy: Always
98+
terminationGracePeriodSeconds: 30
99+
dnsPolicy: ClusterFirst
100+
securityContext: {}
101+
schedulerName: default-scheduler
102+
strategy:
103+
type: Recreate
104+
revisionHistoryLimit: 10
105+
progressDeadlineSeconds: 600
106+
---
107+
kind: Service
108+
apiVersion: v1
109+
metadata:
110+
name: minio-service
111+
spec:
112+
ipFamilies:
113+
- IPv4
114+
ports:
115+
- name: api
116+
protocol: TCP
117+
port: 9000
118+
targetPort: 9000
119+
- name: ui
120+
protocol: TCP
121+
port: 9090
122+
targetPort: 9090
123+
internalTrafficPolicy: Cluster
124+
type: ClusterIP
125+
ipFamilyPolicy: SingleStack
126+
sessionAffinity: None
127+
selector:
128+
app: minio

‎examples/python-helm-demo/minio.env

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
export AWS_ACCESS_KEY_ID=minio
2+
export AWS_DEFAULT_REGION=default
3+
#export AWS_S3_BUCKET=feast-demo
4+
#export AWS_S3_ENDPOINT=http://localhost:9000
5+
export FEAST_S3_ENDPOINT_URL=http://localhost:9000
6+
export AWS_SECRET_ACCESS_KEY=minio123
7+

‎examples/python-helm-demo/online_feature_store.yaml.template

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
project: feast_python_demo
2+
provider: local
3+
registry: s3://feast-demo/registry.db
4+
online_store:
5+
type: redis
6+
connection_string: my-redis-master:6379,password=_REDIS_PASSWORD_
7+
entity_key_serialization_version: 2

‎examples/python-helm-demo/test/feature_store.yaml

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
registry: s3://feast-demo/registry.db
2+
project: feast_python_demo
3+
provider: local
4+
online_store:
5+
path: http://localhost:6566
6+
type: remote
7+
entity_key_serialization_version: 2

0 commit comments

Comments
 (0)