🚨 Alerts

An alert warns the team when something is out of the ordinary (a pod restarting nonstop, a node down), before users notice.

Kubernetes alone does not send alerts. For that, use Prometheus (which evaluates the rules) with Alertmanager (which sends the message by email, Slack or another channel).

info

Alerts are configured with kubectl and YAML files. The Nuvion Panel does not have a screen for this feature.

---

🚀 Before you start

• Have cluster access configured.

🚀 1. Check whether Prometheus is installed

kubectl get pods -A | grep -i -E "prometheus|alertmanager"
kubectl get crd | grep monitoring.coreos.com

• If Prometheus and Alertmanager pods appear along with the prometheusrules.monitoring.coreos.com CRD, go to step 2.
• If nothing appears, monitoring is not installed yet. Use the quick check (last section) or contact support.

🚀 2. Create the alert rules

Create the file alertas-basicos.yaml. Replace NAMESPACE-DO-PROMETHEUS and NOME-DO-RELEASE with the values from your environment:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: alertas-basicos
  namespace: NAMESPACE-DO-PROMETHEUS
  labels:
    release: NOME-DO-RELEASE
spec:
  groups:
    - name: kubernetes-basico
      rules:
        - alert: PodReiniciandoMuito
          expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} restarted more than 3 times in 15 minutes"
        - alert: PodEmCrashLoop
          expr: kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} in CrashLoopBackOff"
        - alert: NodeNaoPronto
          expr: kube_node_status_condition{condition="Ready",status="true"} == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Node {{ $labels.node }} is not in Ready state"

kubectl apply -f alertas-basicos.yaml
kubectl get prometheusrule -A

The for: 5m rule avoids false alarms: the problem must last 5 minutes before firing.

🚀 3. Define where the alerts go

In Alertmanager, configure a destination (receiver). Email example:

route:
  receiver: equipe
receivers:
  - name: equipe
    email_configs:
      - to: "[email protected]"
        from: "[email protected]"
        smarthost: "smtp.example.com:587"
        auth_username: "[email protected]"
        auth_password: "USE-A-SECRET-FOR-THE-PASSWORD"

warning

Do not write the password directly in the file. Store it in a Secret and reference it.

🚀 4. Test the alert

Trigger an error that fires the PodEmCrashLoop rule:

kubectl run pod-crash --image=busybox -- sh -c 'exit 1'

To see the alerts in Prometheus, open a tunnel to it:

kubectl port-forward -n NAMESPACE-DO-PROMETHEUS svc/NOME-DO-SERVICE-PROMETHEUS 9090:9090

Open http://localhost:9090/alerts in your browser. After about 5 minutes, the alert appears as Firing and the message reaches the configured destination.

⚡ Quick check (without Prometheus)

While monitoring is not installed, use these commands to check cluster health:

kubectl get nodes
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get events -A --field-selector type=Warning

This does not replace an automatic alert, since it only shows the situation at the moment you run it.

✅ How to know it worked

The test alert appears as Firing at /alerts and the notification reaches the destination.

❗ Common problems

Symptom	Cause	Solution
no matches for kind "PrometheusRule"	Prometheus Operator is not installed	See step 1
The rule does not appear in Prometheus	The release label does not match	Use the same release as your Prometheus
The alert fires but nothing arrives	Receiver misconfigured	Review step 3 and the Alertmanager logs
The query returns no data	kube-state-metrics is missing	Check for a kube-state-metrics pod in kubectl get pods -A

🧹 Cleanup

kubectl delete pod pod-crash
kubectl delete prometheusrule alertas-basicos -n NAMESPACE-DO-PROMETHEUS

🧠 Questions?

Talk to our SaveinCloud technical support team!