Skip to content

Commit f036670

Browse files
dependabot[bot]tsisodia10
authored andcommitted
# This is a combination of 7 commits.
# This is the 1st commit message: chore: Bump ws from 7.5.6 to 7.5.10 in /ui (feast-dev#4292) # This is the commit message feast-dev#2: Remove metric checks Signed-off-by: Twinkll Sisodia <tsisodia@redhat.com> # This is the commit message feast-dev#3: Draft different md file # This is the commit message feast-dev#4: Refine opentelemetry.md # This is the commit message feast-dev#5: Refine opentelemetry.md # This is the commit message feast-dev#6: Refine opentelemetry.md # This is the commit message feast-dev#7: Refine opentelemetry.md
1 parent 5744d3e commit f036670

File tree

9 files changed

+113
-12
lines changed

9 files changed

+113
-12
lines changed
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
## Adding Monitoring
2+
To add monitoring to the Feast Feature Server, follow these steps:
3+
4+
### Workflow
5+
6+
Feast instrumentation Using OpenTelemetry and Prometheus -
7+
![Workflow](samples/workflow.png)
8+
9+
### Deploy Prometheus Operator
10+
Navigate to OperatorHub and install the stable version of the Prometheus Operator
11+
12+
### Deploy OpenTelemetry Operator
13+
Before installing OTEL Operator, install `cert-manager` and validate the `pods` should spin up --
14+
```
15+
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
16+
```
17+
18+
Then navigate to OperatorHub and install the stable version of the community OpenTelemetry Operator
19+
20+
21+
### Configure OpenTelemetry Collector
22+
Add the OpenTelemetry Collector configuration under the metrics section in your values.yaml file.
23+
24+
Example values.yaml:
25+
26+
```
27+
metrics:
28+
enabled: true
29+
otelCollector:
30+
endpoint: "otel-collector.default.svc.cluster.local:4317" #sample
31+
headers:
32+
api-key: "your-api-key"
33+
```
34+
35+
### Add instrumentation annotation and environment variables in the deployment.yaml
36+
37+
```
38+
template:
39+
metadata:
40+
{{- with .Values.podAnnotations }}
41+
annotations:
42+
{{- toYaml . | nindent 8 }}
43+
instrumentation.opentelemetry.io/inject-python: "true"
44+
```
45+
46+
```
47+
- name: OTEL_EXPORTER_OTLP_ENDPOINT
48+
value: http://{{ .Values.service.name }}-collector.{{ .Release.namespace }}.svc.cluster.local:{{ .Values.metrics.endpoint.port}}
49+
- name: OTEL_EXPORTER_OTLP_INSECURE
50+
value: "true"
51+
```
52+
53+
### Add checks
54+
Add metric checks to all manifests and deployment file -
55+
56+
```
57+
{{ if .Values.metrics.enabled }}
58+
apiVersion: opentelemetry.io/v1alpha1
59+
kind: Instrumentation
60+
metadata:
61+
name: feast-instrumentation
62+
spec:
63+
exporter:
64+
endpoint: http://{{ .Values.service.name }}-collector.{{ .Release.Namespace }}.svc.cluster.local:4318 # This is the default port for the OpenTelemetry Collector
65+
env:
66+
propagators:
67+
- tracecontext
68+
- baggage
69+
python:
70+
env:
71+
- name: OTEL_METRICS_EXPORTER
72+
value: console,otlp_proto_http
73+
- name: OTEL_LOGS_EXPORTER
74+
value: otlp_proto_http
75+
- name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
76+
value: "true"
77+
{{end}}
78+
```
79+
80+
### Add manifests to the chart
81+
Add Instrumentation, OpenTelemetryCollector, ServiceMonitors, Prometheus Instance and RBAC rules as provided in the samples/ directory.
82+
83+
For latest updates please refer the official repository - https://github.com/open-telemetry/opentelemetry-operator
84+
85+
### Deploy Feast
86+
Deploy Feast and set `metrics` value to `true`.
87+
88+
Example -
89+
```
90+
helm install feast-release infra/charts/feast-feature-server --set metric=true --set feature_store_yaml_base64=""
91+
```
92+
93+
## See logs
94+
Once the opentelemetry is deployed, you can search the logs to see the required metrics -
95+
96+
```
97+
oc logs otelcol-collector-0 | grep "Name: feast_feature_server_memory_usage\|Value: 0.*"
98+
oc logs otelcol-collector-0 | grep "Name: feast_feature_server_cpu_usage\|Value: 0.*"
99+
```
100+
```
101+
-> Name: feast_feature_server_memory_usage
102+
Value: 0.579426
103+
```
104+
```
105+
-> Name: feast_feature_server_cpu_usage
106+
Value: 0.000000
107+
```

infra/charts/feast-feature-server/samples/instrumentation.yml

-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
{{- if .Values.metrics }}
21
apiVersion: opentelemetry.io/v1alpha1
32
kind: Instrumentation
43
metadata:

infra/charts/feast-feature-server/samples/otel-collector.yml

-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
# API reference https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md
22
# Refs for v1beta1 config: https://github.com/open-telemetry/opentelemetry-operator/issues/3011#issuecomment-2154118998
3-
{{- if .Values.metrics }}
43
apiVersion: opentelemetry.io/v1beta1
54
kind: OpenTelemetryCollector
65
metadata:

infra/charts/feast-feature-server/samples/otel-sm.yml

-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
{{- if .Values.metrics }}
21
apiVersion: monitoring.coreos.com/v1
32
kind: ServiceMonitor
43
metadata:

infra/charts/feast-feature-server/samples/prometheus.yaml

-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
{{- if .Values.metrics }}
21
apiVersion: monitoring.coreos.com/v1
32
kind: Prometheus
43
metadata:

infra/charts/feast-feature-server/samples/rbac.yml

-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
{{- if .Values.metrics }}
21
apiVersion: v1
32
kind: ServiceAccount
43
metadata:

infra/charts/feast-feature-server/samples/service-monitor.yml

-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
{{- if .Values.metrics }}
21
apiVersion: monitoring.coreos.com/v1
32
kind: ServiceMonitor
43
metadata:
Loading

ui/yarn.lock

+6-6
Original file line numberDiff line numberDiff line change
@@ -11640,14 +11640,14 @@ write-file-atomic@^3.0.0:
1164011640
typedarray-to-buffer "^3.1.5"
1164111641

1164211642
ws@^7.4.6:
11643-
version "7.5.6"
11644-
resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.6.tgz#e59fc509fb15ddfb65487ee9765c5a51dec5fe7b"
11645-
integrity sha512-6GLgCqo2cy2A2rjCNFlxQS6ZljG/coZfZXclldI8FB/1G3CCI36Zd8xy2HrFVACi8tfk5XrgLQEk+P0Tnz9UcA==
11643+
version "7.5.10"
11644+
resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.10.tgz#58b5c20dc281633f6c19113f39b349bd8bd558d9"
11645+
integrity sha512-+dbF1tHwZpXcbOJdVOkzLDxZP1ailvSxM6ZweXTegylPny803bFhA+vqBYw4s31NSAk4S2Qz+AKXK9a4wkdjcQ==
1164611646

1164711647
ws@^8.1.0:
11648-
version "8.4.2"
11649-
resolved "https://registry.yarnpkg.com/ws/-/ws-8.4.2.tgz#18e749868d8439f2268368829042894b6907aa0b"
11650-
integrity sha512-Kbk4Nxyq7/ZWqr/tarI9yIt/+iNNFOjBXEWgTb4ydaNHBNGgvf2QHbS9fdfsndfjFlFwEd4Al+mw83YkaD10ZA==
11648+
version "8.17.1"
11649+
resolved "https://registry.yarnpkg.com/ws/-/ws-8.17.1.tgz#9293da530bb548febc95371d90f9c878727d919b"
11650+
integrity sha512-6XQFvXTkbfUOZOKKILFG1PDK2NDQs4azKQl26T0YS5CxqWLgXajbPZ+h4gZekJyRqFU8pvnbAbbs/3TgRPy+GQ==
1165111651

1165211652
xml-name-validator@^3.0.0:
1165311653
version "3.0.0"

0 commit comments

Comments
 (0)