Building a Prometheus, Grafana and pushgateway cluster with Kubernates

Building a Prometheus, Grafana and pushgateway cluster with Kubernates
Page content

This article is my notes on practicing building a cluster of Prometheus, Grafana and pushgateway in Kubernates environment.

The repository is located at soudegesu/prometheus-pushgw-practice .

Goals

For now, I will aim to build a cluster on a local machine. These goals outline is as shown in the following figure.

purpose

Environment

  • Mac OS X 11.1
  • Docker Desktop 3.0.0
    • Docker 20.1.0
    • Kubernates 1.19.3

Example of Kubernates configuration file

The procedure is based on this article and expanding the parts I want personally.

Specify the Kubernates Context

This time, specify docker-desktop.

1kubectl config use-context docker-desktop

Also, set the namespace to monitoring.

1kubectl create namespace monitoring

Create ClusterRole and ClusterRoleBinding

Create ClusterRole and ClusterRoleBinding. You can change kinds to Role and RoleBinding and use them.

 1apiVersion: rbac.authorization.k8s.io/v1
 2kind: ClusterRole
 3metadata:
 4  name: prometheus
 5rules:
 6- apiGroups: [""]
 7  resources:
 8  - nodes
 9  - nodes/proxy
10  - services
11  - endpoints
12  - pods
13  verbs: ["get", "list", "watch"]
14- apiGroups:
15  - extensions
16  resources:
17  - ingresses
18  verbs: ["get", "list", "watch"]
19- nonResourceURLs: ["/metrics"]
20  verbs: ["get"]
21---
22apiVersion: rbac.authorization.k8s.io/v1
23kind: ClusterRoleBinding
24metadata:
25  name: prometheus
26roleRef:
27  apiGroup: rbac.authorization.k8s.io
28  kind: ClusterRole
29  name: prometheus
30subjects:
31- kind: ServiceAccount
32  name: default
33  namespace: monitoring

Create a pushgateway server

Next, create a definition for pushgateway. A Deployment and a Service definition are in the following example. Other Pods can access to pushgateway service with pushgateway-service:8120.

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: pushgateway-deployment
 5  namespace: monitoring
 6  labels:
 7    app: pushgateway-server
 8spec:
 9  replicas: 1
10  selector:
11    matchLabels:
12      app: pushgateway-server
13  template:
14    metadata:
15      labels:
16        app: pushgateway-server
17    spec:
18      containers:
19        - name: pushgateway
20          image: prom/pushgateway:latest
21          ports:
22            - containerPort: 9091
23---
24apiVersion: v1
25kind: Service
26metadata:
27  name: pushgateway-service
28spec:
29  selector: 
30    app: pushgateway-server
31  type: NodePort
32  ports:
33    - port: 8120
34      targetPort: 9091
35      nodePort: 30040

Create a Prometheus server

Then, create definition of Prometheus. Create a ConfigMap in addition to Deployment and Service.

Define a configuration file (prometheus.yml) for Prometheus in ConfigMap and mount it.

Other Pods can access to Prometheus service with prometheus-service:8080.

 1apiVersion: v1
 2kind: ConfigMap
 3metadata:
 4  name: prometheus-server-conf
 5  labels:
 6    name: prometheus-server-conf
 7  namespace: monitoring
 8data:
 9  prometheus.yml: |-
10    # my global config
11    global:
12      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
13      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
14      # scrape_timeout is set to the global default (10s).
15    # Alertmanager configuration
16    alerting:
17      alertmanagers:
18      - static_configs:
19        - targets:
20          # - alertmanager:9093
21    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
22    rule_files:
23      # - "first_rules.yml"
24      # - "second_rules.yml"
25    # A scrape configuration containing exactly one endpoint to scrape:
26    # Here it's Prometheus itself.
27    scrape_configs:
28      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
29      - job_name: 'prometheus'
30        static_configs:
31        - targets: ['localhost:9090']
32      
33      - job_name: 'pushgateway'
34        honor_labels: true
35        metrics_path: /metrics
36        static_configs:
37        - targets: ['pushgateway-service:8120']
38---
39apiVersion: apps/v1
40kind: Deployment
41metadata:
42  name: prometheus-deployment
43  namespace: monitoring
44  labels:
45    app: prometheus-server
46spec:
47  replicas: 1
48  selector:
49    matchLabels:
50      app: prometheus-server  
51  template:
52    metadata:
53      labels:
54        app: prometheus-server
55    spec:
56      containers:
57        - name: prometheus
58          image: prom/prometheus:latest
59          args:
60            - "--config.file=/etc/config/prometheus.yml"
61            - "--storage.tsdb.path=/prometheus/"
62            - "--storage.tsdb.retention=3d"
63          ports:
64            - containerPort: 9090
65          volumeMounts:
66            - name: config-vol
67              mountPath: /etc/config
68      volumes:
69        - name: config-vol
70          configMap:
71            name: prometheus-server-conf
72---
73apiVersion: v1
74kind: Service
75metadata:
76  name: prometheus-service
77spec:
78  selector: 
79    app: prometheus-server
80  type: NodePort
81  ports:
82    - port: 8080
83      targetPort: 9090 
84      nodePort: 30000            

Create a Grafana server

Then, Create Grafana definition, In following sample, Deployment and Service and ConfigMap are defined.

The definition file is a little bit long, but I create a configuration file here as well with ConfigMap. The ConfigMap definition includes data source definition, dashboard definition, and dashboard configuration.

As for the dashboard settings, editing them from a configuration file is time-consuming, so it is better to export the definitions JSON file from the GUI and paste it.

  1apiVersion: v1
  2kind: ConfigMap
  3metadata:
  4  name: grafana-datasources-conf
  5  labels:
  6    name: grafana-datasources-conf
  7  namespace: monitoring
  8data:
  9  datasources.yml: |-
 10    apiVersion: 1
 11    datasources:
 12    - name: Prometheus
 13      type: prometheus
 14      access: proxy
 15      orgId: 1
 16      uid: 1
 17      url: http://prometheus-service:8080/
 18      basicAuth: false
 19      editable: true
 20      version: 1
 21      isDefault: true
 22---
 23apiVersion: v1
 24kind: ConfigMap
 25metadata:
 26  name: grafana-dashboards-conf
 27  labels:
 28    name: grafana-dashboards-conf
 29  namespace: monitoring
 30data:
 31  dashboards.yml: |-
 32    apiVersion: 1
 33    providers:
 34    - name: 'hoge'
 35      orgId: 1
 36      folder: ''
 37      type: file
 38      disableDeletion: true
 39      editable: true
 40      options:
 41        path: /var/lib/grafana/dashboards
 42---
 43apiVersion: v1
 44kind: ConfigMap
 45metadata:
 46  name: grafana-dashboards-settings
 47  labels:
 48    name: grafana-dashboards-settings
 49  namespace: monitoring
 50data:
 51  hoge.json: |
 52    {
 53      "annotations": {
 54        "list": [
 55          {
 56            "builtIn": 1,
 57            "datasource": "-- Grafana --",
 58            "enable": true,
 59            "hide": true,
 60            "iconColor": "rgba(0, 211, 255, 1)",
 61            "name": "Annotations & Alerts",
 62            "type": "dashboard"
 63          }
 64        ]
 65      },
 66      "editable": true,
 67      "gnetId": null,
 68      "graphTooltip": 0,
 69      "id": 1,
 70      "links": [],
 71      "panels": [
 72        {
 73          "aliasColors": {},
 74          "bars": false,
 75          "dashLength": 10,
 76          "dashes": false,
 77          "datasource": "Prometheus",
 78          "fieldConfig": {
 79            "defaults": {
 80              "custom": {}
 81            },
 82            "overrides": []
 83          },
 84          "fill": 1,
 85          "fillGradient": 0,
 86          "gridPos": {
 87            "h": 9,
 88            "w": 12,
 89            "x": 0,
 90            "y": 0
 91          },
 92          "hiddenSeries": false,
 93          "id": 2,
 94          "legend": {
 95            "avg": false,
 96            "current": false,
 97            "max": false,
 98            "min": false,
 99            "show": true,
100            "total": false,
101            "values": false
102          },
103          "lines": true,
104          "linewidth": 1,
105          "nullPointMode": "null",
106          "options": {
107            "alertThreshold": true
108          },
109          "percentage": false,
110          "pluginVersion": "7.3.7",
111          "pointradius": 2,
112          "points": false,
113          "renderer": "flot",
114          "seriesOverrides": [],
115          "spaceLength": 10,
116          "stack": false,
117          "steppedLine": false,
118          "targets": [
119            {
120              "expr": "go_gc_duration_seconds",
121              "interval": "",
122              "legendFormat": "",
123              "refId": "A"
124            }
125          ],
126          "thresholds": [],
127          "timeFrom": null,
128          "timeRegions": [],
129          "timeShift": null,
130          "title": "Panel Title",
131          "tooltip": {
132            "shared": true,
133            "sort": 0,
134            "value_type": "individual"
135          },
136          "type": "graph",
137          "xaxis": {
138            "buckets": null,
139            "mode": "time",
140            "name": null,
141            "show": true,
142            "values": []
143          },
144          "yaxes": [
145            {
146              "format": "short",
147              "label": null,
148              "logBase": 1,
149              "max": null,
150              "min": null,
151              "show": true
152            },
153            {
154              "format": "short",
155              "label": null,
156              "logBase": 1,
157              "max": null,
158              "min": null,
159              "show": true
160            }
161          ],
162          "yaxis": {
163            "align": false,
164            "alignLevel": null
165          }
166        }
167      ],
168      "schemaVersion": 26,
169      "style": "dark",
170      "tags": [],
171      "templating": {
172        "list": []
173      },
174      "time": {
175        "from": "now-6h",
176        "to": "now"
177      },
178      "timepicker": {},
179      "timezone": "",
180      "title": "sample",
181      "uid": "UpBk6EfGz",
182      "version": 1
183---
184apiVersion: apps/v1
185kind: Deployment
186metadata:
187  name: grafana-deployment
188  namespace: monitoring
189  labels:
190    app: grafana-server
191spec:
192  replicas: 1
193  selector:
194    matchLabels:
195      app: grafana-server
196  template:
197    metadata:
198      labels:
199        app: grafana-server
200    spec:
201      containers:
202        - name: grafana
203          image: grafana/grafana:latest
204          # args:
205          #  - "--config=/etc/config/datasources/datasources.yml"
206          ports:
207            - containerPort: 3000
208          volumeMounts:
209            - name: datasources-config-vol
210              mountPath: /etc/grafana/provisioning/datasources
211            - name: dashboards-config-vol
212              mountPath: /etc/grafana/provisioning/dashboards
213            - name: dashboards-settings-vol
214              mountPath: /var/lib/grafana/dashboards
215      volumes:
216        - name: datasources-config-vol
217          configMap:
218            name: grafana-datasources-conf
219        - name: dashboards-config-vol
220          configMap:
221            name: grafana-dashboards-conf
222        - name: dashboards-settings-vol
223          configMap:
224            name: grafana-dashboards-settings
225---
226apiVersion: v1
227kind: Service
228metadata:
229  name: grafana-service
230spec:
231  selector: 
232    app: grafana-server
233  type: NodePort
234  ports:
235    - port: 8100
236      targetPort: 3000
237      nodePort: 30020

apply definitions

Finally, run the kubectl apply command to deploy the container.

1kubectl apply -f ${name of pushgateway k8s file}.yml -n monitoring
2kubectl apply -f ${name of prometheus k8s file}.yml -n monitoring
3kubectl apply -f ${name of grafana k8s file}.yml -n monitoring

Open the dashboard of Docker Desktop and you can see the deployment container has been created successfully. After that, open your browser and access the following URL to make sure that the service console is displayed.

  • Prometheus: http://localhost:30000/
  • grafana: http://localhost:30020/

Conclusion

Created a configuration to build Prometheus, Grafana and pushgateway in the Kubernates In the scope of this work, the storage volume of Prometheus is not mounted externally, so if you need it, please configure it accordingly.