Introduction to Log Collections

This tutorial gives a brief introduction about the general steps of adding log receivers in KubeSphere.

• Before adding a log receiver, you need to enable any of the logging, events or auditing components. For more information, see Enable Pluggable Components.

To add a log receiver:

1. Log in to the web console of KubeSphere as admin.

2. Click Platform in the top left corner and select Cluster Management.

3. If you have enabled the , you can select a specific cluster. If you have not enabled the feature, refer to the next step directly.

4. Go to Log Collections in Cluster Settings.

5. Click Add Log Collector in the Logging tab.

   • At most one receiver can be added for each receiver type.
   • Different types of receivers can be added simultaneously.

A default Elasticsearch receiver will be added with its service address set to an Elasticsearch cluster if logging, , or auditing is enabled in ClusterConfiguration.

An internal Elasticsearch cluster will be deployed to the Kubernetes cluster if neither externalElasticsearchUrl nor externalElasticsearchPort is specified in when logging, events or is enabled. The internal Elasticsearch cluster is for testing and development only. It is recommended that you configure an external Elasticsearch cluster for production.

Log searching relies on the internal or external Elasticsearch cluster configured.

If the default Elasticsearch log receiver is deleted, refer to Add Elasticsearch as a Receiver to add a new one.

Kafka is often used to receive logs and serves as a broker to other processing systems like Spark. demonstrates how to add Kafka to receive Kubernetes logs.

If you need to output logs to more places other than Elasticsearch or Kafka, you can add Fluentd as a log receiver. Fluentd has numerous output plugins which can forward logs to various destinations such as S3, MongoDB, Cassandra, MySQL, syslog, and Splunk. Add Fluentd as a Receiver demonstrates how to add Fluentd to receive Kubernetes logs.

Starting from KubeSphere v3.0.0, the logs of Kubernetes events and the auditing logs of Kubernetes and KubeSphere can be archived in the same way as container logs. The tab Events or Auditing on the Log Collections page will appear if events or auditing is enabled accordingly in . You can go to the corresponding tab to configure log receivers for Kubernetes events or Kubernetes and KubeSphere auditing logs.

Container logs, Kubernetes events and Kubernetes and KubeSphere auditing logs should be stored in different Elasticsearch indices to be searched in KubeSphere. The index prefixes are:

• ks-logstash-events for Kubernetes events
• ks-logstash-auditing for Kubernetes and KubeSphere auditing logs

You can turn a log receiver on or off without adding or deleting it. To turn a log receiver on or off:

2. Click More and select Change Status.

3. Select Activate or Close to turn the log receiver on or off.

   

4. A log receiver’s status will be changed to Close if you turn it off, otherwise the status will be Collecting.

You can modify a log receiver or delete it:

1. On the Log Collections page, click a log receiver and go to the receiver’s detail page.

2. Edit a log receiver by clicking Edit or Edit YAML from the drop-down list.