k8s配置Prometheus监控时，可以通过servicemonitor的方式增加job，以此来增加监控项，但这种方式进行监控配置，只能手工一个一个的增加，如果k8s集群规模较大的情况下，这种方式会很麻烦。

一种方式是采用consul注册中心的方式进行自动发现。

另外一种方式是基于kubernetes_sd_configs的自动发现的方式配置增加监控项，本文主要讲解此种配置方式。

基于consul的自动发现

在安装consul后，可以通过指定consul读取特定配置文件的方式发现并加载监控项

 但这种方式和Prometheus基于文件的动态发现没有本质区别，甚至还增加了系统的复杂度，并不可取。

可以通过调用consul的API的方式，让程序自动向consul进行注册，在Prometheus中配置consul的相关项，让其自动增加监控Target

#Prometheus主配置文件增加如下内容

#以便可以从consul中自动获取监控信息

- job_name: 'consul-prometheus'

consul_sd_configs:

- server: '10.0.12.8:8500'

services: []

# 注册服务

curl -X PUT -d '{"id": "consul-redis","name": "redis","address": "10.0.12.8","port": 6379,"tags": ["service"],"checks": [{"http": "http://10.0.12.8:6379/","interval": "5s"}]}' http://10.0.12.8:8500/v1/agent/service/register

# 查询指定节点以及指定的服务信息

[root@iZ2zejaz33icbod2k4cvy6Z ~]# curl http://10.0.12.8:8500/v1/catalog/service/consul-redis

#删除指定服务 redis为要删除服务的id

curl -X PUT http://10.0.12.8:8500/v1/agent/service/deregister/consul-redis

基于kubernetes_sd_configs的自动发现

准备Prometheus的自动发现的配置文件并加载

[root@VM-12-8-centos kube-prom]# cat prometheus-additional.yaml

- job_name: 'blackbox'

metrics_path: /probe

params:

module: [http_2xx]

static_configs:

- targets:

- http://10.1.226.250:6000

- http://10.1.38.97:3000/healthz/ready

- http://10.1.116.84:5000

- http://10.1.215.125:7000/healthz/ready

- http://10.1.111.235:8000/healthz/ready

relabel_configs:

- source_labels: [__address__]

target_label: __param_target

- source_labels: [__param_target]

target_label: instance

- target_label: __address__

replacement: blackbox-exporter:9115

- job_name: 'kubernetes-service-endpoints'

kubernetes_sd_configs:

- role: endpoints

relabel_configs:

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]

action: keep

regex: true

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]

action: replace

target_label: __scheme__

regex: (https?)

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]

action: replace

target_label: __metrics_path__

regex: (.+)

- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]

action: replace

target_label: __address__

regex: ([^:]+)(?::\d+)?;(\d+)

replacement: $1:$2

- action: labelmap

regex: __meta_kubernetes_service_label_(.+)

- source_labels: [__meta_kubernetes_namespace]

action: replace

target_label: namespace

- source_labels: [__meta_kubernetes_service_name]

action: replace

target_label: service

- source_labels: [__meta_kubernetes_pod_name]

target_label: pod

action: replace

- job_name: 'kubernetes-pods'

kubernetes_sd_configs:

- role: pod

relabel_configs:

- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]

action: keep

regex: true

- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]

action: replace

target_label: __metrics_path__

regex: (.+)

- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]

action: replace

regex: ([^:]+)(?::\d+)?;(\d+)

replacement: $1:$2

target_label: __address__

- action: labelmap

regex: __meta_kubernetes_pod_label_(.+)

- source_labels: [__meta_kubernetes_namespace]

action: replace

target_label: namespace

- source_labels: [__meta_kubernetes_pod_name]

action: replace

target_label: pod

运行生成secret文件

[root@VM-12-8-centos kube-prom]# kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run -oyaml > additional-scrape-configs.yaml

应用，配置进入Prometheus中

[root@VM-12-8-centos kube-prom]# kubectl apply -f additional-scrape-configs.yaml -n monitoring

secret/additional-scrape-configs configured

运行curl -XPOST http://10.0.12.8:30090/-/reload热加载一下，就可以在dashboard中看到增加的配置了

修改prometheus-k8s 的 ClusterRole权限 

 Prometheus 绑定了一个名为 prometheus-k8s 的 ServiceAccount 对象，而这个对象绑定的是一个名为 prometheus-k8s 的 ClusterRole，这个角色没有对 Service 或者 Pod 的 list 权限，所以需要进行修改

[root@VM-12-8-centos manifests]# kubectl edit clusterrole prometheus-k8s -n monitoring -o yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

creationTimestamp: "2022-11-13T14:21:08Z"

name: prometheus-k8s

resourceVersion: "16164985"

selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/prometheus-k8s

uid: 7a404fac-9462-486a-a109-65a1ef98e423

rules:

- apiGroups:

- ""

resources:

- nodes

- services

- endpoints

- pods

- nodes/proxy

verbs:

- get

- list

- watch

- apiGroups:

- ""

resources:

- configmaps

- nodes/metrics

verbs:

- get

- nonResourceURLs:

- /metrics

verbs:

- get

 pod配置自动发现

pod要自动发现，必须在annotations:增加prometheus.io/scrape: "true"

新建一个pod

[root@VM-12-8-centos k8s]# cat PODforheadlesssvr.yml

apiVersion: v1

kind: Pod

metadata:

name: ex-podforheadlesssvr

annotations:

prometheus.io/scrape: "true"

spec:

containers:

- name: testcontainer

image: docker.io/appropriate/curl

imagePullPolicy: IfNotPresent

command: ['sh', '-c']

args: ['echo "test pod for headless service";sleep 96000']

[root@VM-12-8-centos k8s]# kubectl apply -f ex6_1_4PODforheadlesssvr.yml

pod/ex-podforheadlesssvr created

[root@VM-12-8-centos k8s]# kubectl get po

NAME READY STATUS RESTARTS AGE

ex-podforheadlesssvr 1/1 Running 0 3s

过一会检查dashboard，已经在界面上了

 状态为down，因为这个pod对应的镜像并没有相关的metrics接口，我们主要是用来进行自动发现测试的

在服务发现界面

在target labels部分

如上操作，就可以基于k8s自动发现 在Prometheus中增加监控项了

 

精彩文章

评论可见，请评论后查看内容，谢谢！！！
 您阅读本篇文章共花了：