Collecting Logs with Fluentbit

    Setting up log collection consists of two pieces: running a log forwarding DaemonSet on each node, and running a collector somewhere in the cluster (in our example, we use a StatefulSet which stores logs on a Kubernetes PersistentVolumeClaim, but you could also use a HostPath).

    It’s useful to set up the collector before the forwarders, because you’ll need the address of the collector when configuring the forwarders, and the forwarders may queue logs until the collector is ready.

    The defines a StatefulSet as well as a Kubernetes Service which allows accessing and reading the logs from within the cluster. The supplied configuration will create the monitoring configuration in a namespace called logging. You can apply the configuration with:

    The default configuration will classify logs into Knative, apps (pods with an app= label which aren’t Knative), and the default to logging with the pod name; this can be changed by updating the log-collector-config ConfigMap before or after installation. Once the ConfigMap is updated, you’ll need to restart Fluent Bit (for example, by deleting the pod and letting the StatefulSet recreate it).

    To access the logs through your web browser:

      You can also open a shell in the nginx pod and search the logs using unix tools:

      For the most part, you can follow the . Those directions will set up a Fluent Bit DaemonSet which forwards logs to ElasticSearch by default; when the directions call for creating the ConfigMap, you’ll want to either replace the elasticsearch configuration with this fluent-bit-configmap.yaml or add the following block to the ConfigMap and update the @INCLUDE output-elasticsearch.conf to be @INCLUDE output-forward.conf.

      1. output-forward.conf: |
      2.   [OUTPUT]
      3.       Name            forward
      4.       Host            log-collector.logging
      6.       Require_ack_response  True

      If you are using a different log collection infrastructure (Splunk, for example), on how to configure your forwarders.

      NOTE: This describes a development environment setup, and is not appropriate for production.

      If you are using a local Kubernetes cluster for development (Kind, Docker Desktop, or Minikube), you can create a hostPath PersistentVolume to store the logs on your desktop OS. This will allow you to use all your normal desktop tools on the files without needing Kubernetes-specific tools.

      The PersistentVolumeClaim will look something like this, but the hostPath will vary based on your Kubernetes software and host operating system. Some example values are documented below.

      1.   metadata:
      2.     name: logs
      3.   spec:
      4.     accessModes: ["ReadWriteOnce"]
      5.     volumeName: shared-logs

      When creating your cluster, you’ll need to use a kind-config.yaml and specify for each node, like so:

      You can then use /shared/logs as the spec.hostPath.path in your PersistentVolume. Note that the directory path ./logs is relative to the directory that the Kind cluster was created in.

      Docker desktop automatically creates some shared mounts between the host and the guest operating systems, so you only need to know the path to your home directory. Here are some examples for different operating systems:

      Minikube requires an explicit command to mount a directory into the VM running Kubernetes. This command mounts the logs directory inside the current directory onto /mnt/logs in the VM:

      1. minikube mount ./logs:/mnt/logs

      You would then reference /mnt/logs as the hostPath.path in the PersistentVolume.

      Table of contents