使用 Prometheus 監控 Kubernetes Cluster

Kubernetes monitoring by Prometheus and Grafana
Kevin Jen
Div X/ICL
2018/04/27

Copyright 2016 ITRI 工業技術研究院
個人簡介
• 任博煜 Kevin Jen
• 工研院資通所
• 資料中心與雲端應用軟體組
• 協助整合以及維運由Kubernetes
搭建的 ITRI DNN Farm

Tools
Prometheus
This is the central piece, it contains the time series database and the logic of scraping stats from
exporters (see below) as well as alerts.
Node-exporter
A Node Exporter to export system metrics in a Prometheus-compatible format.
There are many other types of expoter ,like container expoter or snmp expoter.
Nvidia-smi-exporter
nvidia-smi metrics exporter for Prometheus
Grafana
A web-based graphical dashboard builder that supports Prometheus

What is Prometheus
• Prometheus is an open-source systems monitoring and alerting toolkit originally built
at SoundCloud.
• Since its inception in 2012, many companies and organizations have adopted Prometheus,
and the project has a very active developer and user community. It is now a standalone open
source project and maintained independently of any company.
• To emphasize this and clarify the project's governance structure, Prometheus joined
the Cloud Native Computing Foundation in 2016 as the second hosted project
after Kubernetes.

Features
• A multi-dimensional data model (time series identified by metric
name and key/value pairs)
• A flexible query language to leverage this dimensionality
• Targets are discovered via service discovery or static configuration

Grafana
• Grafana is an open source metric analytics & visualization suite.
• Prometheus delegates the heavy lifting of proper graphical displays and dashboards to
Grafana.
• Grafana features pluggable panels and data sources allowing easy extensibility and a
variety of panels, including fully featured graph panels with rich visualization options.
• Easily create alert rules from within the UI and have them be continually evaluated by the
Grafana backend.

部署
kubectl create -f node-exporter.yaml
kubectl create -f rbac-setup.yaml.yaml
kubectl create -f prometheus-config-map.yaml
kubectl create -f prometheus-deploy.yaml
kubectl create -f grafana-deploy.yaml
git clone https://github.com/kevin7674/prometheus.git
cd Prometheus
下載
部署
檢查
kubectl get pod

使用RBAC創建Cluster Role並設置訪問權限
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: ["", "extensions", "apps"]
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
- deployments
- services
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: default

node-exporter.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: node-exporter
spec:
template:
metadata:
labels:
app: node-exporter
name: node-exporter
spec:
containers:
- image: prom/node-exporter:latest
name: node-exporter
ports:
- containerPort: 9100
hostPort: 9100
name: scrape
hostNetwork: true
hostPID: true
restartPolicy: Always
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: 'true'
labels:
app: node-exporter
name: node-exporter
name: node-exporter
spec:
clusterIP: None
ports:
- name: scrape
port: 9100
protocol: TCP
selector:
app: node-exporter
type: ClusterIP

prometheus-ConfigMap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
namespace: default
data:
prometheus.yml: |-
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https

- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics

- job_name: 'kubernetes-pods'
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::d+)?;(d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name

- job_name: 'kubernetes-service-endpoints'
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::d+)?;(d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name

prometheus-deploy.yaml
kind: Deployment
metadata:
name: prometheus-deployment
spec:
replicas: 1
template:
metadata:
labels:
app: prometheus-server
spec:
serviceAccountName: prometheus
containers:
- name: prometheus
image: prom/prometheus:v1.8.2
args:
- "-config.file=/etc/prometheus/prometheus.yml"
- "-storage.local.path=/prometheus/"
- "-storage.local.retention=240h"
ports:
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
- name: prometheus-storage-volume
nfs:
server: 10.241.1.11
path: "/nfs/prometheus“
nodeSelector:
release: production
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: 'true'
labels:
name: prometheus
name: prometheus
spec:
selector:
type: NodePort
ports:
- name: prometheus
protocol: TCP
port: 9090
nodePort: 30008

nvidia_smi_exporter
到每台GPU-node做以下步驟
1.下載並執行nvidia_smi_exporter
git clone https://github.com/kevin7674/nvidia_smi_exporter.git
cd nvidia_smi_exporter
./nvidia_smi_exporter 9101 &
2.修改 vi /etc/rc.local
./nvidia_smi_exporter 9101 &
修改Prometheus-configmap.yaml
- job_name: "nvidia_smi_exporter"
static_configs:
- targets: [‘Node_IP:9101']
- targets: [‘Node_IP:9101']

grafana-deploy.yaml
kind: Deployment
metadata:
name: grafana
#namespace: prometheus
labels:
app: grafana
spec:
replicas: 1
template:
metadata:
labels:
app: grafana
spec:
containers:
- image: grafana/grafana:3.1.0
name: grafana
ports:
hostPort: 3000
env:
- name: GF_AUTH_BASIC_ENABLED
value: "false"
- name: GF_AUTH_ANONYMOUS_ENABLED
value: "true"
- name: GF_AUTH_ANONYMOUS_ORG_ROLE
value: Admin
- name: GF_SERVER_ROOT_URL
value: /api/v1/proxy/namespaces/prometheus/services/grafana/
volumeMounts:
- name: grafana-persistent-storage
mountPath: /var/lib/grafana
volumes:
- name: grafana-persistent-storage
nfs:
server: 10.241.1.11
path: "/nfs/grafana“
nodeSelector:
release: production
apiVersion: v1
kind: Service
metadata:
name: grafana
#namespace: prometheus
labels:
app: grafana
spec:
ports:
- port: 80
targetPort: 3000
nodePort: 30000
selector:
app: grafana
type: LoadBalancer

Query

Excute graph
Refer to the official website
https://prometheus.io/docs/querying/basics/

Slice & Dice
container_memory_usage_bytes{name="consul-server"}

Aggregation sum(container_memory_usage_bytes{name=~"^consul"})
avg(container_memory_usage_bytes{name=~"^consul"}) by (name)

sum by (pod_name)( rate(container_cpu_usage_seconds_total{image!="", pod_name!=""}[1m] ) )

Data source

Dashboard
Creating a new dashboard and add a graph panel.

Edit

Creating a Prometheus graph by querying metrics.

平台基礎資源監控
監控項目包含:
• Prometheus 透過node-exporter做為agent，收集node上的資源使用狀況:
(服務節點數量、CPU使用率、Memory使用量、Filesystem Space、Host
Temperature、Network I/O)
• Kubernetes和基礎服務是否存活: (kubelet、Registry、API Server、NFS
Server)

Container監控
• Prometheus透過kubernetes API得到container資源使用狀態。
• Container監控項目 : (Pod名稱、數量、使否存活、CPU使用率、
Memory使用量、Filesystem、I/O)

GPU監控
• 每一台GPU Node利用nvidia-smi exporter做為agent回報資訊，
再由prometheu統一收集、達到監控。
• GPU監控項目: (GPU使用率、GPU溫度)

kind: Deployment
metadata:
name: alertmanager-deployment
spec:
replicas: 1
template:
metadata:
labels:
spec:
containers:
- name: alertmanager
image: quay.io/prometheus/alertmanager:v0.10.0
args:
- "-config.file=/etc/prometheus/config.yml"
- "-storage.path=/alertmanager"
ports:
protocol: TCP
name: http
volumeMounts:
- name: alertmanager-config-volume
mountPath: /etc/prometheus
volumes:
- name: alertmanager-config-volume
configMap:
name: alertmanager
nodeSelector:
release: production
Alertmanager-deploy.yaml

apiVersion: v1
kind: ConfigMap
metadata:
labels:
namespace: default
data:
prometheus.yml: |-
global:
scrape_interval: 15s
rule_files:
- '/etc/prometheus-rules/alert.rules'
alerting:
alertmanagers:
- static_configs:
- targets: ["10.241.1.11:30009"]
修改prometheus-ConfigMap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-alert-rules
data:
alert.rules: |-
ALERT GPUHighTemperature
IF nvidia_temperature_gpu{instance="10.241.1.22:9101",gpu="Tesla V100-SXM2-16GB[0]"} > 25
FOR 1m
ANNOTATIONS {
summary = "{{$labels.instance}}: GPU Temperature High",
description = "{{$labels.instance}}: This GPU Temperature has really high for over 1m",
}
Alert-rules-configmap.yaml

Grafana Alert

Alert Mail設置
修改 grafana image 設定發信功能
docker run -d -p 3000:3000 monitoringartist/grafana-xxl
docker ps
docker exec -ti <container_ID> sh
apt-get update
apt-get install vim
vim /usr/share/grafana/conf/defaults.ini
SMTP / Emailing [smtp]
enabled = true
host = smtp.gmail.com:587
user = itriccmabin@gmail.com
password =“XXXX"
cert_file =
key_file =
skip_verify = false
from_address = itriccmabin@gmail.com
from_name = DNN_Monitor
[emails] welcome_email_on_sign_up = false templates_pattern = emails/*.html

使用 Prometheus 監控 Kubernetes Cluster

More Related Content

Similar to 使用 Prometheus 監控 Kubernetes Cluster (20)

More from inwin stack (20)

Recently uploaded (20)

使用 Prometheus 監控 Kubernetes Cluster

Editor's Notes