Changes in Rancher v2.5

  • The now powers Rancher’s logging solution in place of the former, in-house solution.
  • Fluent Bit is now used to aggregate the logs, and is used for filtering the messages and routing them to the outputs. Previously, only Fluentd was used.
  • Logging can be configured with a Kubernetes manifest, because logging now uses a Kubernetes operator with Custom Resource Definitions.
  • We now support filtering logs.
  • We now support writing logs to multiple outputs.
  • We now always collect Control Plane and etcd logs.

The following figure from the Banzai documentation shows the new logging architecture:

How the Banzai Cloud Logging Operator Works with Fluentd and Fluent Bit

Enabling Logging for Rancher Managed Clusters

You can enable the logging for a Rancher managed cluster by going to the Apps page and installing the logging app.

  1. In the Rancher UI, go to the cluster where you want to install logging and click Cluster Explorer.
  2. Click Apps.
  3. Click the rancher-logging app.
  4. Scroll to the bottom of the Helm chart README and click Install.

Result: The logging app is deployed in the cattle-logging-system namespace.

Uninstall Logging

  1. From the Cluster Explorer, click Apps & Marketplace.
  2. Click Installed Apps.
  3. Go to the cattle-logging-system namespace and check the boxes for rancher-logging and rancher-logging-crd.
  4. Click Delete.
  5. Confirm Delete.

Result rancher-logging is uninstalled.

Role-based Access Control

Rancher logging has two roles, logging-admin and logging-view.

  • logging-admin gives users full access to namespaced flows and outputs
  • logging-view allows users to view namespaced flows and outputs, and cluster flows and outputs

In Rancher, the cluster administrator role is the only role with full access to all rancher-logging resources. Cluster members are not able to edit or read any logging resources. Project owners and members have the following privileges:

Both project owners and project members require at least one namespace in their project to use logging. If they do not, then they may not see the logging button in the top nav dropdown.

Configuring the Logging Application

To configure the logging application, go to the Cluster Explorer in the Rancher UI. In the upper left corner, click Cluster Explorer > Logging.

The following Custom Resource Definitions are used to configure logging:

Examples

Once logging is installed, you can use these examples to help craft your own logging pipeline.

Cluster Output to ElasticSearch

Let’s say you wanted to send all logs in your cluster to an elasticsearch cluster. First, we create a cluster output.

We have created this cluster output, without elasticsearch configuration, in the same namespace as our operator: cattle-logging-system. Any time we create a cluster flow or cluster output, we have to put it in the cattle-logging-system namespace.

Now that we have configured where we want the logs to go, let’s configure all logs to go to that output.

  1. apiVersion: logging.banzaicloud.io/v1beta1
  2. kind: ClusterFlow
  3. metadata:
  4.     name: "all-logs"
  5.     namespace: "cattle-logging-system"
  6. spec:
  7.   globalOutputRefs:
  8.     - "example-es"

We should now see our configured index with logs in it.

What if we have an application team who only wants logs from a specific namespaces sent to a splunk server? For this case, we can use namespaced outputs and flows.

Before we start, let’s set up that team’s application: coolapp.

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4.   name: devteam
  5. ---
  6. apiVersion: apps/v1
  7. kind: Deployment
  8. metadata:
  9.   namespace: devteam
  10.   labels:
  11.     app: coolapp
  12. spec:
  13.   replicas: 2
  14.   selector:
  15.     matchLabels:
  16.       app: coolapp
  17.   template:
  18.     metadata:
  19.       labels:
  20.         app: coolapp
  21.     spec:
  22.       containers:
  23.         - name: generator
  24.           image: paynejacob/loggenerator:latest

With coolapp running, we will follow a similar path as when we created a cluster output. However, unlike cluster outputs, we create our output in our application’s namespace.

Once again, let’s feed our output some logs.

  1. apiVersion: logging.banzaicloud.io/v1beta1
  2. kind: Flow
  3. metadata:
  4.     name: "devteam-logs"
  5.     namespace: "devteam"
  6. spec:
  7.   localOutputRefs:
  8.     - "devteam-splunk"

Unsupported Output

For the final example, we create an output to write logs to a destination that is not supported out of the box (e.g. syslog):

  2. kind: Secret
  3. metadata:
  4.   name: syslog-config
  5.   namespace: cattle-logging-system
  6. type: Opaque
  7. stringData:
  8.   fluent-bit.conf: |
  9.     [INPUT]
  10.         Name              forward
  11.         Port              24224
  13.     [OUTPUT]
  14.         Name              syslog
  15.         InstanceName      syslog-output
  16.         Match             *
  17.         Addr              syslog.example.com
  18.         Port              514
  19.         Cluster           ranchers
  21. ---
  22. apiVersion: apps/v1
  23. kind: Deployment
  24. metadata:
  25.   name: fluentbit-syslog-forwarder
  26.   namespace: cattle-logging-system
  27.   labels:
  28.     output: syslog
  29. spec:
  30.   selector:
  31.     matchLabels:
  32.       output: syslog
  33.   template:
  34.     metadata:
  35.         output: syslog
  36.     spec:
  37.       containers:
  38.       - name: fluentbit
  39.         image: paynejacob/fluent-bit-out-syslog:latest
  40.         ports:
  41.           - containerPort: 24224
  42.         volumeMounts:
  43.           - mountPath: "/fluent-bit/etc/"
  44.             name: configuration
  45.       volumes:
  46.       - name: configuration
  47.         secret:
  48.           secretName: syslog-config
  49. ---
  50. apiVersion: v1
  51. kind: Service
  52. metadata:
  53.   name: syslog-forwarder
  54.   namespace: cattle-logging-system
  55. spec:
  56.   selector:
  57.     output: syslog
  58.   ports:
  59.     - protocol: TCP
  60.       port: 24224
  62. ---
  63. apiVersion: logging.banzaicloud.io/v1beta1
  64. kind: ClusterFlow
  65. metadata:
  66.   name: all-logs
  67.   namespace: cattle-logging-system
  68. spec:
  69.   globalOutputRefs:
  70.     - syslog
  71. ---
  72. apiVersion: logging.banzaicloud.io/v1beta1
  73. kind: ClusterOutput
  74. metadata:
  75.   name: syslog
  76.   namespace: cattle-logging-system
  77. spec:
  78.   forward:
  79.     servers:
  80.       - host: "syslog-forwarder.cattle-logging-system"
  81.     require_ack_response: false
  82.     ignore_network_errors_at_startup: false

Let’s break down what is happening here. First, we create a deployment of a container that has the additional syslog plugin and accepts logs forwarded from another fluentd. Next we create an output configured as a forwarder to our deployment. The deployment fluentd will then forward all logs to the configured syslog destination.

Working with a Custom Docker Root Directory

Applies to v2.5.6+

If using a custom Docker root directory, you can set global.dockerRootDirectory in values.yaml. This will ensure that the Logging CRs created will use your specified path rather than the default Docker data-root location.

Working with Taints and Tolerations

“Tainting” a Kubernetes node causes pods to repel running on that node. Unless the pods have a toleration for that node’s taint, they will run on other nodes in the cluster. Taints and tolerations can work in conjunction with the nodeSelector within the PodSpec, which enables the opposite effect of a taint. Using nodeSelector gives pods an affinity towards certain nodes. Both provide choice for the what node(s) the pod will run on.

By default, Rancher taints all Linux nodes with cattle.io/os=linux, and does not taint Windows nodes. The logging stack pods have tolerations for this taint, which enables them to run on Linux nodes. Moreover, we can populate the nodeSelector to ensure that our pods only run on Linux nodes. Let’s look at an example pod YAML file with these settings…

In the above example, we ensure that our pod only runs on Linux nodes, and we add a toleration for the taint we have on all of our Linux nodes. You can do the same with Rancher’s existing taints, or with your own custom ones.

Windows Support

Clusters with Windows workers support exporting logs from Linux nodes, but Windows node logs are currently unable to be exported. Only Linux node logs are able to be exported.

If you would like to add your own nodeSelector settings, or if you would like to add tolerations for additional taints, you can pass the following to the chart’s values.

  1. tolerations:
  2.   # insert tolerations...
  3. nodeSelector:
  4.   # insert nodeSelector...

These values will add both settings to the fluentd, fluentbit, and logging-operator containers. Essentially, these are global settings for all pods in the logging stack.

However, if you would like to add tolerations for only the fluentbit container, you can add the following to the chart’s values.

  1. fluentbit_tolerations:
  2.   # insert tolerations list for fluentbit containers only...

Troubleshooting

The cattle-logging Namespace Being Recreated

If your cluster previously deployed logging from the Cluster Manager UI, you may encounter an issue where its cattle-logging namespace is continually being recreated.

The solution is to delete all clusterloggings.management.cattle.io and projectloggings.management.cattle.io custom resources from the cluster specific namespace in the management cluster. The existence of these custom resources causes Rancher to create the cattle-logging namespace in the downstream cluster if it does not exist.

The cluster namespace matches the cluster ID, so we need to find the cluster ID for each cluster.

  1. In your web browser, navigate to your cluster(s) in either the Cluster Manager UI or the Cluster Explorer UI.
  2. Copy the <cluster-id> portion from one of the URLs below. The <cluster-id> portion is the cluster namespace name.
  1. kubectl delete clusterloggings.management.cattle.io -n <cluster-id>
  2. kubectl delete projectloggings.management.cattle.io -n <cluster-id>