Viewing the status of the log store

    You can view the status of your log store.

    Prerequisites

    • Cluster logging and Elasticsearch must be installed.

    Procedure

    1. Change to the project.

    2. To view the status:

      1. Get the name of the log store instance:

        1. $ oc get Elasticsearch

        Example output

        1. NAME            AGE
        2. elasticsearch   5h9m

      2. Get the log store status:

        1. $ oc get Elasticsearch <Elasticsearch-instance> -o yaml

        For example:

        1. $ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

        The output includes information similar to the following:

        Example output

        1. status: (1)
        2.   cluster: (2)
        3.     activePrimaryShards: 30
        4.     activeShards: 60
        5.     initializingShards: 0
        6.     numDataNodes: 3
        7.     numNodes: 3
        8.     pendingTasks: 0
        9.     relocatingShards: 0
        10.     status: green
        11.     unassignedShards: 0
        12.   clusterHealth: ""
        13.   conditions: [] (3)
        14.   nodes: (4)
        15.   - deploymentName: elasticsearch-cdm-zjf34ved-1
        16.     upgradeStatus: {}
        17.   - deploymentName: elasticsearch-cdm-zjf34ved-2
        18.     upgradeStatus: {}
        19.   - deploymentName: elasticsearch-cdm-zjf34ved-3
        20.     upgradeStatus: {}
        21.   pods: (5)
        22.     client:
        23.       failed: []
        24.       notReady: []
        25.       ready:
        26.       - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
        27.       - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
        28.       - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
        29.     data:
        30.       failed: []
        31.       notReady: []
        32.       ready:
        33.       - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
        34.       - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
        35.       - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
        36.     master:
        37.       failed: []
        38.       notReady: []
        39.       ready:
        40.       - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
        41.       - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
        42.       - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
        43.   shardAllocationEnabled: all

    The following are examples of some condition messages from the Status section of the Elasticsearch instance.

    This status message indicates a node has exceeded the configured low watermark and no shard will be allocated to this node.

    1. status:
    2.   nodes:
    3.   - conditions:
    4.     - lastTransitionTime: 2019-03-15T15:57:22Z
    5.         be allocated on this node.
    6.       reason: Disk Watermark Low
    7.       status: "True"
    8.       type: NodeStorage
    10.     upgradeStatus: {}

    This status message indicates a node has exceeded the configured high watermark and shards will be relocated to other nodes.

    1. status:
    2.   nodes:
    3.   - conditions:
    4.     - lastTransitionTime: 2019-03-15T16:04:45Z
    5.       message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
    6.         from this node.
    7.       reason: Disk Watermark High
    8.       status: "True"
    9.       type: NodeStorage
    10.     deploymentName: example-elasticsearch-cdm-0-1
    11.     upgradeStatus: {}

    This status message indicates the log store node selector in the CR does not match any nodes in the cluster:

    1. status:
    2.     nodes:
    3.     - conditions:
    4.       - lastTransitionTime: 2019-04-10T02:26:24Z
    5.         message: '0/8 nodes are available: 8 node(s) didn''t match node selector.'
    6.         reason: Unschedulable
    7.         status: "True"
    8.         type: Unschedulable

    This status message indicates that the log store CR uses a non-existent PVC.

    This status message indicates that your log store cluster does not have enough nodes to support your log store redundancy policy.

    1. status:
    2.   clusterHealth: ""
    3.   conditions:
    4.   - lastTransitionTime: 2019-04-17T20:01:31Z
    5.     message: Wrong RedundancyPolicy selected. Choose different RedundancyPolicy or
    6.       add more nodes with data roles
    7.     reason: Invalid Settings
    8.     status: "True"
    9.     type: InvalidRedundancy

    This status message indicates your cluster has too many control plane nodes (also known as the master nodes):

    1. status:
    2.   clusterHealth: green
    3.   conditions:
    4.     - lastTransitionTime: '2019-04-17T20:12:34Z'
    5.       message: >-
    6.         Invalid master nodes count. Please ensure there are no more than 3 total
    7.         nodes with master roles
    8.       reason: Invalid Settings
    9.       status: 'True'
    10.       type: InvalidMasters

    Viewing the status of the log store components

    You can view the status for a number of the log store components.

    Elasticsearch indices

    You can view the status of the Elasticsearch indices.

    1. Get the name of an Elasticsearch pod:

      1. $ oc get pods --selector component=elasticsearch -o name

      Example output

      1. pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
      2. pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
      3. pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

    2. Get the status of the indices:

      1. $ oc exec elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -- indices

      Example output

      1. Defaulting container name to elasticsearch.
      2. Use 'oc describe pod/elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -n openshift-logging' to see all of the containers in this pod.
      4. green  open   infra-000002                                                     S4QANnf1QP6NgCegfnrnbQ   3   1     119926            0        157             78
      5. green  open   audit-000001                                                     8_EQx77iQCSTzFOXtxRqFw   3   1          0            0          0              0
      6. green  open   .security                                                        iDjscH7aSUGhIdq0LheLBQ   1   1          5            0          0              0
      7. green  open   .kibana_-377444158_kubeadmin                                     yBywZ9GfSrKebz5gWBZbjw   3   1          1            0          0              0
      8. green  open   infra-000001                                                     z6Dpe__ORgiopEpW6Yl44A   3   1     871000            0        874            436
      9. green  open   app-000001                                                       hIrazQCeSISewG3c2VIvsQ   3   1       2453            0          3              1
      10. green  open   .kibana_1                                                        JCitcBMSQxKOvIq6iQW6wg   1   1          0            0          0              0
      11. green  open   .kibana_-1595131456_user1                                        gIYFIEGRRe-ka0W3okS-mQ   3   1          1            0          0              0

    Log store pods

    1. Get the name of a pod:

      1. $ oc get pods --selector component=elasticsearch -o name

      Example output

      1. pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
      2. pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
      3. pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

    2. Get the status of a pod:

      The output includes the following status information:

      Example output

      1. ....
      2. Status:             Running
      5. Containers:
      7.     Container ID:   cri-o://b7d44e0a9ea486e27f47763f5bb4c39dfd2
      8.     State:          Running
      9.       Started:      Mon, 08 Jun 2020 10:17:56 -0400
      10.     Ready:          True
      11.     Restart Count:  0
      12.     Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
      14. ....
      16.   proxy:
      17.     Container ID:  cri-o://3f77032abaddbb1652c116278652908dc01860320b8a4e741d06894b2f8f9aa1
      18.     State:          Running
      19.       Started:      Mon, 08 Jun 2020 10:18:38 -0400
      20.     Ready:          True
      21.     Restart Count:  0
      23. ....
      25. Conditions:
      26.   Type              Status
      27.   Initialized       True
      28.   Ready             True
      29.   ContainersReady   True
      30.   PodScheduled      True
      32. ....
      34. Events:          <none>

    Log storage pod deployment configuration

    You can view the status of the log store deployment configuration.

    1. Get the name of a deployment configuration:

      1. $ oc get deployment --selector component=elasticsearch -o name

      Example output

      1. deployment.extensions/elasticsearch-cdm-1gon-1
      2. deployment.extensions/elasticsearch-cdm-1gon-2
      3. deployment.extensions/elasticsearch-cdm-1gon-3

    2. Get the deployment configuration status:

      1. $ oc describe deployment elasticsearch-cdm-1gon-1

      The output includes the following status information:

      Example output

      1. ....
      2.   Containers:
      3.    elasticsearch:
      4.     Image:      registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.3
      5.     Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
      7. ....
      9. Conditions:
      10.   Type           Status   Reason
      11.   ----           ------   ------
      12.   Progressing    Unknown  DeploymentPaused
      13.   Available      True     MinimumReplicasAvailable
      15. ....
      17. Events:          <none>

    Log store replica set

    You can view the status of the log store replica set.

    1. Get the name of a replica set:

      1. $ oc get replicaSet --selector component=elasticsearch -o name
      3. replicaset.extensions/elasticsearch-cdm-1gon-1-6f8495
      4. replicaset.extensions/elasticsearch-cdm-1gon-2-5769cf
      5. replicaset.extensions/elasticsearch-cdm-1gon-3-f66f7d

    2. Get the status of the replica set:

      1. $ oc describe replicaSet elasticsearch-cdm-1gon-1-6f8495

      The output includes the following status information:

      Example output

      1. ....
      2.   Containers:
      3.    elasticsearch:
      4.     Image:      registry.redhat.io/openshift4/ose-logging-elasticsearch6@sha256:4265742c7cdd85359140e2d7d703e4311b6497eec7676957f455d6908e7b1c25
      5.     Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
      7. ....