How-To: Set up Fluentd, Elastic search and Kibana in Kubernetes

    Install Elastic search and Kibana

    1. Create namespace for monitoring tool and add Helm repo for Elastic Search

    2. Add Elastic helm repo

      2. helm repo update

    3. Install Elastic Search using Helm

    By default the chart creates 3 replicas which must be on different nodes. If your cluster has less than 3 nodes, specify a lower number of replicas. For example, this sets it to 1:

    1. helm install elasticsearch elastic/elasticsearch -n dapr-monitoring --set replicas=1

    Otherwise:

    1. helm install elasticsearch elastic/elasticsearch -n dapr-monitoring

    If you are using minikube or want to disable persistent volumes for development purposes, you can disable it by using the following command:

    1. Install Kibana

      1. helm install kibana elastic/kibana -n dapr-monitoring

    2. Validation

      Ensure Elastic Search and Kibana are running in your Kubernetes cluster.

      1. kubectl get pods -n dapr-monitoring
      2. NAME                            READY   STATUS    RESTARTS   AGE
      3. elasticsearch-master-0          1/1     Running   0          6m58s
      4. kibana-kibana-95bc54b89-zqdrk   1/1     Running   0          4m21s

    Apply the configurations to your cluster:

    1. kubectl apply -f ./fluentd-config-map.yaml
    2. kubectl apply -f ./fluentd-dapr-with-rbac.yaml
    1. Ensure that Fluentd is running as a daemonset; the number of instances should be the same as the number of cluster nodes. In the example below we only have 1 node.

    Install Dapr with JSON formatted logs

    1. Install Dapr with enabling JSON-formatted logs

      1. helm repo add dapr https://dapr.github.io/helm-charts/
      2. helm repo update
      3. helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true

    2. Enable JSON formatted log in Dapr sidecar

    Add dapr.io/log-as-json: "true" annotation to your deployment yaml.

    Example:

    1. apiVersion: apps/v1
    3. metadata:
    4.   name: pythonapp
    5.   namespace: default
    6.   labels:
    7.     app: python
    8. spec:
    9.   replicas: 1
    10.   selector:
    11.     matchLabels:
    12.       app: python
    13.   template:
    14.       labels:
    15.         app: python
    16.       annotations:
    18.         dapr.io/app-id: "pythonapp"
    19.         dapr.io/log-as-json: "true"
    20. ...
    1. Port-forward to svc/kibana-kibana
    1. $ kubectl port-forward svc/kibana-kibana 5601 -n dapr-monitoring
    2. Forwarding from 127.0.0.1:5601 -> 5601
    3. Forwarding from [::1]:5601 -> 5601
    4. Handling connection for 5601
    5. Handling connection for 5601

    1. Browse http://localhost:5601

    2. Click Management -> Index Management

    1. Wait until dapr-* is indexed.

    1. Once dapr-* indexed, click Kibana->Index Patterns and Create Index Pattern

    create index pattern

    1. Define index pattern - type dapr* in index pattern

    1. Select time stamp filed: @timestamp

    timestamp

    1. Confirm that scope, type, app_id, level, etc are being indexed.

    1. Click icon and search scope:*

    discover

    References