Prometheus

Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

TL;DR;

$ helm install stable/prometheus

Introduction

This chart bootstraps a Prometheus deployment on a Kubernetes cluster using the Helm package manager.

Prerequisites

• Kubernetes 1.3+ with Beta APIs enabled

Installing the Chart

To install the chart with the release name my-release:

$ helm install --name my-release stable/prometheus

The command deploys Prometheus on the Kubernetes cluster in the default configuration. The configuration section lists the parameters that can be configured during installation.

Tip: List all releases using helm list

Uninstalling the Chart

To uninstall/delete the my-release deployment:

$ helm delete my-release

The command removes all the Kubernetes components associated with the chart and deletes the release.

Prometheus 2.x

Prometheus version 2.x has made changes to alertmanager, storage and recording rules. Check out the migration guide here

Users of this chart will need to update their alerting rules to the new format before they can upgrade.

Upgrading from previous chart versions.

Version 9.0 adds a new option to enable or disable the Prometheus Server. This supports the use case of running a Prometheus server in one k8s cluster and scraping exporters in another cluster while using the same chart for each deployment. To install the server server.enabled must be set to true.

As of version 5.0, this chart uses Prometheus 2.x. This version of prometheus introduces a new data format and is not compatible with prometheus 1.x. It is recommended to install this as a new release, as updating existing releases will not work. See the prometheus docs for instructions on retaining your old data.

Example migration

Assuming you have an existing release of the prometheus chart, named prometheus-old. In order to update to prometheus 2.x while keeping your old data do the following:

1. Update the prometheus-old release. Disable scraping on every component besides the prometheus server, similar to the configuration below:

   	alertmanager:
   	  enabled: false
   	alertmanagerFiles:
   	  alertmanager.yml: ""
   	kubeStateMetrics:
   	  enabled: false
   	nodeExporter:
   	  enabled: false
   	pushgateway:
   	  enabled: false
   	server:
   	  extraArgs:
   	    storage.local.retention: 720h
   	serverFiles:
   	  alerts: ""
   	  prometheus.yml: ""
   	  rules: ""
   

2. Deploy a new release of the chart with version 5.0+ using prometheus 2.x. In the values.yaml set the scrape config as usual, and also add the prometheus-old instance as a remote-read target.

   	  prometheus.yml:
   	    ...
   	    remote_read:
   	    - url: http://prometheus-old/api/v1/read
   	    ...
   

   Old data will be available when you query the new prometheus instance.

Scraping Pod Metrics via Annotations

This chart uses a default configuration that causes prometheus to scrape a variety of kubernetes resource types, provided they have the correct annotations. In this section we describe how to configure pods to be scraped; for information on how other resource types can be scraped you can do a helm template to get the kubernetes resource definitions, and then reference the prometheus configuration in the ConfigMap against the prometheus documentation for relabel_config and kubernetes_sd_config.

In order to get prometheus to scrape pods, you must add annotations to the the pods as below:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: /metrics
    prometheus.io/port: "8080"
spec:
...

You should adjust prometheus.io/path based on the URL that your pod serves metrics from. prometheus.io/port should be set to the port that your pod serves metrics from. Note that the values for prometheus.io/scrape and prometheus.io/port must be enclosed in double quotes.

Configuration

The following table lists the configurable parameters of the Prometheus chart and their default values.

Specify each parameter using the --set key=value[,key=value] argument to helm install. For example,

$ helm install stable/prometheus --name my-release \
    --set server.terminationGracePeriodSeconds=360

Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example,

$ helm install stable/prometheus --name my-release -f values.yaml

Tip: You can use the default values.yaml

Note that you have multiple yaml files. This is particularly useful when you have alerts belonging to multiple services in the cluster. For example,

# values.yaml
# ...

# service1-alert.yaml
serverFiles:
  alerts:
    service1:
      - alert: anAlert
      # ...

# service2-alert.yaml
serverFiles:
  alerts:
    service2:
      - alert: anAlert
      # ...

$ helm install stable/prometheus --name my-release -f values.yaml -f service1-alert.yaml -f service2-alert.yaml

RBAC Configuration

Roles and RoleBindings resources will be created automatically for server service.

To manually setup RBAC you need to set the parameter rbac.create=false and specify the service account to be used for each service by setting the parameters: serviceAccounts.{{ component }}.create to false and serviceAccounts.{{ component }}.name to the name of a pre-existing service account.

Tip: You can refer to the default *-clusterrole.yaml and *-clusterrolebinding.yaml files in templates to customize your own.

ConfigMap Files

AlertManager is configured through alertmanager.yml. This file (and any others listed in alertmanagerFiles) will be mounted into the alertmanager pod.

Prometheus is configured through prometheus.yml. This file (and any others listed in serverFiles) will be mounted into the server pod.

Ingress TLS

If your cluster allows automatic creation/retrieval of TLS certificates (e.g. kube-lego), please refer to the documentation for that mechanism.

To manually configure TLS, first create/retrieve a key & certificate pair for the address(es) you wish to protect. Then create a TLS secret in the namespace:

kubectl create secret tls prometheus-server-tls --cert=path/to/tls.cert --key=path/to/tls.key

Include the secret's name, along with the desired hostnames, in the alertmanager/server Ingress TLS section of your custom values.yaml file:

server:
  ingress:
    ## If true, Prometheus server Ingress will be created
    ##
    enabled: true

    ## Prometheus server Ingress hostnames
    ## Must be provided if Ingress is enabled
    ##
    hosts:
      - prometheus.domain.com

    ## Prometheus server Ingress TLS configuration
    ## Secrets must be manually created in the namespace
    ##
    tls:
      - secretName: prometheus-server-tls
        hosts:
          - prometheus.domain.com

NetworkPolicy

Enabling Network Policy for Prometheus will secure connections to Alert Manager and Kube State Metrics by only accepting connections from Prometheus Server. All inbound connections to Prometheus Server are still allowed.

To enable network policy for Prometheus, install a networking plugin that implements the Kubernetes NetworkPolicy spec, and set networkPolicy.enabled to true.

If NetworkPolicy is enabled for Prometheus' scrape targets, you may also need to manually create a networkpolicy which allows it.