Prometheus

    This document demonstrates how to scrape metrics from edge node through Yurt-Tunnel’s DNS mode within an OpenYurt cluster.

    • OpenYurt v0.5.0+

    • CoreDNS v1.6.8+

    • prometheus-operator

    OpenYurt will create yurt-tunnel-nodes ConfigMap，which keeps track of nodename dns records of nodes.

    use hosts plugin to load dns records in yurt-tunnel-nodes configmap.

    1. $ kubectl edit configmap coredns -n kube-system
    2. ...........
    3.  Corefile: |
    4.     .:53 {
    5.         errors
    6.         health {
    7.            lameduck 5s
    8.         }
    9.         ready
    10.             reload 300ms
    11.             fallthrough
    12.         }
    13.         kubernetes cluster.local in-addr.arpa ip6.arpa {
    14.            pods insecure
    15.            fallthrough in-addr.arpa ip6.arpa
    16.            ttl 30
    18.         prometheus :9153
    19.         forward . /etc/resolv.conf {
    20.            max_concurrent 1000
    21.         }
    22.         cache 30
    23.         loop
    24.         reload
    25.         loadbalance
    26.     }

    By default, prometheus scrape node metrics with node ip. With relabel functionality provided by prometheus, we can change node ip to node hostname. You can config scrape behavior by modify ServiceMonitor CR.

    • More about relabel config, please refer to .

    Add relabel rule in kubelet ServiceMonitor，Use __meta_kubernetes_endpoint_address_target_name to replace node ip：

    1. $ kubectl edit serviceMonitor kubelet -n monitoring
    2. spec:
    3.   endpoint:
    4.     relabelings:
    5.     - action: replace  # add relabel rule
    6.       regex: (.*);.*:(.*)
    7.       replacement: $1:$2
    8.       sourceLabels:
    9.       - __meta_kubernetes_endpoint_address_target_name
    11.       targetLabel: __address__
    12.     ..........

    Yurt-tunnel will only do forward for port 10250 and 10255, if you want to add forward for other ports, you can modify yurt-tunnel-server-cfg ConfigMap. For node-exporter， you may need to add 9100 to https-proxy-ports。If you want to add http forward, just modify http-proxy-ports.

    modify yurt-tunnel-server-cfgConfigMap

    Add relabel rule in node-exporter ServiceMonitor，use __meta_kubernetes_pod_node_nameto replace node ip：

    1. $ kubectl edit servicemonitor  prom-kube-prometheus-stack-node-exporter
    2. spec:
    3.  endpoint:
    4.    ......
    5.    relabelings:
    6.     - action: replace #add relabel rule
    7.       regex: (.*);.*:(.*)
    8.       replacement: $1:$2
    9.       sourceLabels:
    10.       - __meta_kubernetes_pod_node_name
    11.       - __address__
    13.     ........

    Openyurt Yurt-Tunnel DNS Mode