Table of Contents
Prometheus is really trending. It is the second largest project in CNCF. It’s much familiar because of its wide use cases and the simplicity in operations. We have one sub category for the Prometheus, if you guys are interested please have a look here: https://www.crybit.com/category/devops/prom/
Prometheus has a lot of advantages over other monitoring platforms. The main and important advantages of Prometheus are:
- Prometheus is TSDB
- Prometheus is Pull based tool
- Centralised control
- In built Alerting facility
- Easy for monitoring teams
- Data visualisation
- Service discovery (sd)
- Scalability
- PromQL
You can see a detailed explanations about all these points here Advantages of Prometheus
Kubernetes is one of the very matured cluster orchestration tools available now. Almost all big companies now moved their containerised workloads to Kubernetes (different flavours of Kubernetes) because of its ease in use and scalability. Monitoring the Kubernetes cluster is also important and this can be easily done by the help of Prometheus.
In current Kubernetes world everything can be deployed by the use of operators. Most of the advanced and well matured Kuberenetes providers (like OpenShift) includes Prometheus operators by default as part of cluster operators or it’s away from a single click. In this article, I will explain how we can simply deploy a Prometheus setup along with Grafana for monitoring your cluster resources and Kubernetes workloads.
As we all know, monitoring is crucial. Lets go with Prometheus.
Prerequisites
- A running Kuberenetes cluster
- kubectl
- helm 3
Why Helm?
To deploy Prometheus easily. Helm is the standard package manager for Kubernetes workloads. It’s easy to deploy, maintain and manage the deployments using helm charts. Read more about helm from the official site: https://helm.sh/docs/topics/charts/
Deploy Prometheus and Grafana on Kubernetes
You can test this setup on your minicube setup as well. Follow the below steps to deploy Prometheus along with Grafana for your Kubernetes cluster.
Prometheus
Create a namespace for Prometheus
It’s a best practice to deploy Prometheus or your monitoring components to a different namespace. You can allow access to your production or staging namespaces by the help of Networking Policies later, if you face issues or in case you need to add more restrictions. It also help the cluster admin to limit resources for your prometheus components by applying limits in namespace level as well.
$ kubectl create namespace prometheus
You can check the namespace by using get call.
$ kubectl get namespaces | grep prometheus
prometheus Active 35m
Now we are good to deploy prometheus and it’s dependent components using helm. Hope you are familiar with helm and its use cases.
Make sure you have prometheus repo added in your helm repo
To see currently available repos you can use helm repo list from your CLI.
$ helm repo list
NAME URL
stable https://charts.helm.sh/stable
dandydev https://dandydeveloper.github.io/charts
bitnami https://charts.bitnami.com/bitnami
Add Prometheus community helm repo
Future Note: This article is published on July 2021. Please make sure any changes in repo/chart and add the repo accordingly.
The stable/prometheus-operator chart is deprecated and they moved it to https://github.com/prometheus-community/helm-charts
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts/
If you see SSL error, please make sure that your client (kubectl) configurations are properly configured. Otherwise pass –insecure-skip-tls-verify parameter
Error: looks like "https://prometheus-community.github.io/helm-charts/" is not a valid chart repository or cannot be reached: Get "https://prometheus-community.github.io/helm-charts/index.yaml": x509: certificate signed by unknown authority
Fix
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts/ --insecure-skip-tls-verify
Now you have added the Prometheus community repo to your helm.
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts/ --insecure-skip-tls-verify
"prometheus-community" has been added to your repositories
$ helm repo list
NAME URL
stable https://charts.helm.sh/stable
dandydev https://dandydeveloper.github.io/charts
bitnami https://charts.bitnami.com/bitnami
prometheus-community https://prometheus-community.github.io/helm-charts/
Now we are good to install / deploy Prometheus components to your Kubernetes cluster.
Install Prometheus using helm
$ helm repo update Hang tight while we grab the latest from your chart repositories… …Successfully got an update from the "dandydev" chart repository …Successfully got an update from the "prometheus-community" chart repository …Successfully got an update from the "stable" chart repository …Successfully got an update from the "bitnami" chart repository Update Complete. ⎈Happy Helming!⎈
Install by running
$ helm install prometheus prometheus-community/kube-prometheus-stack --namespace prometheus
That’s it. You can see all pods by running the get call.
$ kubectl --namespace prometheus get pods
Sample output
$ kctl --namespace prometheus get pods -l "release=prometheus" NAME READY STATUS RESTARTS AGE prometheus-kube-prometheus-operator-59c5dcf5bc-g4ctl 1/1 Running 0 34s prometheus-prometheus-node-exporter-4bvmw 1/1 Running 0 34s prometheus-prometheus-node-exporter-ddw5g 1/1 Running 0 34s prometheus-prometheus-node-exporter-rgn5q 1/1 Running 0 34s $ kubectl --namespace prometheus get pods NAME READY STATUS RESTARTS AGE alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 46s prometheus-grafana-6576787696-qrkqq 2/2 Running 0 71s prometheus-kube-prometheus-operator-59c5dcf5bc-g4ctl 1/1 Running 0 71s prometheus-kube-state-metrics-fcd578d77-bhxs5 1/1 Running 0 71s prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 1 46s prometheus-prometheus-node-exporter-4bvmw 1/1 Running 0 71s prometheus-prometheus-node-exporter-ddw5g 1/1 Running 0 71s prometheus-prometheus-node-exporter-rgn5q 1/1 Running 0 71s
To see all available charts
$ helm search repo prometheus-community
Now you can port forward your Prometheus service or direct pod by using the Kubernetes port forwarding technique and test your prometheus installation. Read more about port forwarding from official documentation.
$ kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090 -n prometheus Forwarding from 127.0.0.1:44829 -> 9090 Forwarding from [::1]:44829 -> 9090
It should be accessible from your local browser after connecting to your cluster using SSH port forwarding. Example command is added below:
$ ssh -t -L 9090:localhost:9090 [email protected] -i "kub_key.pem"
Now you can access prometheus using localhost:44829/graph It should looks like
Now time to explore Prometheus. Refer: https://prometheus.io/
Grafana
Similar way you can access Grafana as well. Identify svc or pod for Grafana and expose & access it. As the Prometheus installation is managed by the operator, it install Grafana as well.
$ kubectl port-forward svc/prometheus-grafana 80 -n prometheus Forwarding from 127.0.0.1:44829 -> 3000 Forwarding from [::1]:44829 -> 3000
Now access it http://localhost:3000/login after forwarding local port.
Where is Grafana credentials?
This can be found from the secrets.
$ kubectl get secrets prometheus-grafana -n prometheus
NAME TYPE DATA AGE
prometheus-grafana Opaque 3 57m
$ kubectl get secrets prometheus-grafana -n prometheus -o yaml
apiVersion: v1
data:
admin-password: cHJvbS1vcGVyYXRvcg==
admin-user: YWRtaW4=
ldap-toml: ""
kind: Secret
Secret data encoded with base64 you can decode it by the following command:
- For getting username
$ openssl base64 -d YWRtaW4= admin
- For getting password
$ openssl base64 -d cHJvbS1vcGVyYXRvcg== prom-operator
Now you are in to Grafana
Explore Grafana. It comes with a lot of default dashboards which helps to monitor your Kubernetes cluster.
Will explain the details about each dashboards and the way to troubleshoot your cluster in an efficient way will include in upcoming blog posts. Stay tuned.
We just exposed the Prometheus and Grafana using port forwarding. In case of real production cluster you can create ingress for accessing the endpoints. This also will
Please let me know if you have any questions or suggestions. I will try to reach out to you.
Thanks!!
