Alerting with Prometheus in Kubernetes – A Beginners Guide
Alertmanager is an open-source alerting system. The prime purpose is to manage alerts sent by client applications such as the Prometheus server. Moreover, it takes care of deduplicating, grouping, and routing them to the correct receiver integration such as Slack, Email, PagerDuty, OpsGenie, etc. It also manages silencing and inhibition of alerts.
This blog will cover the Alert Manager setup and its integration with Prometheus deployed on a kubernetes cluster. This architecture will be using slack, as a notification channel.
I will be showing you the installation of Prometheus and Alertmanager via helm as initially, when I was trying to do so, I struggled a lot as there were only a few resources available on the internet. Hence this gave me motivation to write up this blog. Have a look-
The main steps involved in setting up the alerting and notifications are:
- Setup and configure the Alertmanager
Configure the alertmanager with different notification channels.
- Create alerting rules in Prometheus
(Like if any pod is not running)
Below is the basic architecture of alertmanager
If you install a stable/prometheus helm chart, alertmanager is also installed with it.
Setting up Slack Alerts
The first thing you need to do-
Be a part of the Slack workspace.
Go to Slack -> Administration -> Manage apps.
The Alertmanager uses the Incoming Webhooks feature of Slack, so we need to set that up first.
Manage -> Custom Integrations -> Incoming Webhooks
Here you need to add the channel for receiving the alerts. Once you confirm and add Incoming WebHooks integration, the webhook URL (which is your Slack api_url) is displayed.
Now copy the webhook url, this is your Slack api_url which will be used in the configmap of alertmanager.
Changes are to be made in the configmaps of alertmanager to configure it with slack. We need to edit the alertmanger.yml file.
Step 1: View the alertmanager configmap.
Step 2: Store the alertmanager configmap on local.
We can even directly edit the configmap file or the best approach is to save the cm(configmap) on local and then edit it using vs code or any text editor.
Step 3: Edit the alertmanager.yml file
- Various alertmanager receivers can be integrated such as email, slack, PagerDuty, OpsGenie, etc.
- Below is the setup for slack, edit the configmap accordingly
Step 4: Apply the configmap manifest file.
Step 5: Validate the configmap
Alerting Rules file in Prometheus
Alerting rules can be set in the configmap of prometheus-server.
Step 1: Store the cm on your local
Step 2: Add the rules in alerting_rules.yml file
Step 3: Apply the prometheus-server configmap manifest file
Relax and check your Slack Channel
Here you go, now you are all set to get notified via the firing alerts in slack, till then relax !!
Get In Touch
How Can We Help ?
We make your product happen. Our dynamic, robust and scalable solutions help you drive value at the greatest speed in the market