Bring your own Prometheus
Big picture​
Scrape Calico Enterprise metrics for Bring Your Own (BYO) Prometheus.
Value​
Calico Enterprise uses the Prometheus monitoring tool to scrape metrics from instrumented jobs, and displays time-series data in a visualizer such as Grafana. You can scrape the following time-series metrics for Calico Enterprise components to your own Prometheus:
- elasticsearch
- fluentd
- calico-node
- kube-controllers
- felix
- typha
To use BYO Prometheus, you must create your own:
- Service monitors
- Alerts
With BYO Prometheus, Calico Enterprise metrics and alerts are not visible in the web console.
Before you begin​
Supported
For the supported version of Prometheus in this release, see the Release Notes (coreos-prometheus
).
How to​
Scrape metrics​
- elasticsearch
- fluentd
- calico node
- kube-controllers
- Felix
- Typha
Configure TLS certificates
-
Copy the required secret and configmap to your namespace.
-
Save the manifest of the required TLS secret and CA configmap.
kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml
kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
-
Edit
calico-node-prometheus-client-tls.yaml
andtigera-ca-bundle.yaml
by changing the namespace to the namespace where your prometheus instance is running. -
Apply the manifests to your cluster.
kubectl apply -f calico-node-prometheus-client-tls.yaml
kubectl apply -f tigera-ca-bundle.yaml
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.18.6/manifests/prometheus/elasticsearch-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Configure TLS certificates
-
Copy the required secret and configmap to your namespace.
-
Save the manifest of the required TLS secret and CA configmap.
kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml
kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
-
Edit
calico-node-prometheus-client-tls.yaml
andtigera-ca-bundle.yaml
and change the namespace to the namespace where your prometheus instance is running. -
Apply the manifests to your cluster.
kubectl apply -f calico-node-prometheus-client-tls.yaml
kubectl apply -f tigera-ca-bundle.yaml
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.18.6/manifests/prometheus/fluentd-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Configure TLS certificates
-
Copy the required secret and configmap to your namespace.
-
Save the manifest of the required TLS secret and CA configmap.
kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml
kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
-
Edit
calico-node-prometheus-client-tls.yaml
andtigera-ca-bundle.yaml
by changing the namespace to the namespace where your prometheus instance is running. -
Apply the manifests to your cluster.
kubectl apply -f calico-node-prometheus-client-tls.yaml
kubectl apply -f tigera-ca-bundle.yaml
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.18.6/manifests/prometheus/calico-node-monitor-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in $NAMESPACE.
Configure TLS certificates
-
Copy the required secret and configmap to your namespace.
-
Save the manifest of the required TLS secret and CA configmap.
kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml
kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
-
Edit
calico-node-prometheus-client-tls.yaml
andtigera-ca-bundle.yaml
by changing the namespace to the namespace where your prometheus instance is running. -
Apply the manifests to your cluster.
kubectl apply -f calico-node-prometheus-client-tls.yaml
kubectl apply -f tigera-ca-bundle.yaml
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.18.6/manifests/prometheus/kube-controller-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Enable metrics
Felix metrics are not enabled by default.
By default, Felix uses port 9091 TCP to publish metrics.
Use the following command to enable Felix metrics.
kubectl patch felixconfiguration default --type merge --patch '{"spec":{"prometheusMetricsEnabled": true}}'
You should see a result similar to:
felixconfiguration.projectcalico.org/default patched
For all Felix configuration values, see Felix configuration.
For all Prometheus Felix configuration values, see Felix Prometheus.
Create a service to expose Felix metrics
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: felix-metrics-svc
namespace: calico-system
labels:
k8s-app: felix-metrics
spec:
selector:
k8s-app: calico-node
ports:
- port: 9091
targetPort: 9091
EOF
If running Calico Enterprise for Windows, also create a service for Windows nodes:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: felix-windows-metrics-svc
namespace: calico-system
labels:
k8s-app: felix-metrics
spec:
clusterIP: None
selector:
k8s-app: calico-node-windows
ports:
- port: 9091
targetPort: 9091
EOF
By default, the Windows firewall blocks listening on ports. For Calico Enterprise to manage the Prometheus metrics ports Windows firewall rules, enable the windowsManageFirewallRules
setting in FelixConfiguration:
kubectl patch felixConfiguration default --type merge --patch '{"spec":{"windowsManageFirewallRules": "Enabled"}}'
See the FelixConfiguration reference for more details. You can also add a Windows firewall rule that allows listening on the Prometheus port(s) instead of having Calico Enterprise manage it.
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.18.6/manifests/prometheus/felix-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Enable metrics
Typha metrics are not enabled by default.
By default, Typha uses port 9091 TCP to publish metrics. However, if Calico Enterprise is installed using the Amazon yaml file, this port will be 9093 because it is set manually using the TYPHA_PROMETHEUSMETRICSPORT environment variable.
Use the following command to enable Typha metrics.
kubectl patch installation default --type=merge -p '{"spec": {"typhaMetricsPort":9093}}'
You should see a result similar to:
installation.operator.tigera.io/default patched
Create a service to expose Typha metrics
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: typha-metrics-svc
namespace: calico-system
labels:
k8s-app: typha-metrics
spec:
selector:
k8s-app: calico-typha
ports:
- port: 9093
targetPort: 9093
EOF
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.18.6/manifests/prometheus/typha-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Verify BYO Prometheus​
Verify metrics in the Prometheus console
-
Access the Prometheus dashboard using the port-forwarding feature.
kubectl port-forward pod/byo-prometheus-pod 9090:9090 -n $NAMESPACE
-
Browse to the Prometheus dashboard: http://localhost:9090.
-
In the Expression text box, enter your metric name and click the Execute button.
The Console table is populated with all of your nodes with the number of endpoints.
Verify endpoint authentication
-
Use the following command to retrieve the tls.key and tls.cert.
export NAMESPACE=<my-prometheus-namespace>
kubectl get secret -n $NAMESPACE calico-node-prometheus-client-tls -o yaml
-
Save the tls.key and tls.cert content into key and cert after base64 decode.
$:tls_key=<tls.key content>
$:echo $tls_key|base64 -d >key.pem
$:tls_cert=<tls.crt content>
$:echo $cert|base64 -d>cert.pem -
Get the ca-bundle certificate using this command:
kubectl get cm -n $NAMESPACE tigera-ca-bundle -o yaml
-
Open a new file (bundle.pem) in your favorite editor, and paste the content from "BEGIN CERTIFICATE" to "END CERTIFICATE".
-
Port-forward the prometheus pods and run this command with the forwarded port.
curl --cacert bundle.pem --key key.pem --cert cert.pem https://localhost:8080/metrics
You should be able to see the metrics.
Create policy to secure traffic between pods​
To support zero trust, we recommend that you create Calico Enterprise network policy to allow the traffic between BYO Prometheus pods, and the respective metrics pods. For samples of ingress and egress policies, see Get started with Calico network policy.