Filtering Logs with ELK

Filtering log output for specific tasks

Prerequisite

• An Elasticsearch installation that aggregates DC/OS logs

Install, configure, and start Logstash

1. Install Logstash.

2. Create the following pattern file in your custom patterns directory, located at $PATTERNS_DIR:

3. Update the configuration file for your Logstash instance to include the following grok filter, where $PATTERNS_DIR is replaced with your custom patterns directory:

       grok {
           patterns_dir => "$PATTERNS_DIR"
           match => { "file" => "%{TASKPATH}" }
       }
   }

4. Start Logstash.

   Logstash will extract the agent, , executor, and run fields. These fields are shown in the metadata of all Mesos task log events. Elasticsearch queries will also show results from those fields.

Usage example

In the screenshots below, we are using Kibana hosted by logz.io, but your Kibana interface will look similar.

1. Figure 1. Logstash events

2. Click the disclosure triangle next to one of these events to view the details. This will show all of the fields extracted from the task log file path:

   

   Figure 2. Event details

Template examples

Here are some example query templates. Replace the template parameters $executor1, $framework2, and any others with the actual values from your cluster.

• Logs related to a specific executor $executor1, including logs for tasks run from that executor:

• Non-task logs related to a specific executor $executor1:

  "$executor1" AND NOT executor:$executor1

• Logs (including task logs) for a framework , if $executor1 and $executor2 are that framework’s executors:

• Non-task logs for a framework $framework1, if $executor1 and $executor2 are that framework’s executors:

  ("$framework1" OR "$executor1" OR "$executor2") AND NOT (framework:$framework1 OR executor:$executor1 OR executor:$executor2)

• Logs for a framework $framework1 on a specific agent host $agent_host1:

• Non-task logs for a framework $framework1 on a specific agent $agent1 with host :