Log Ingestion

    OpenSearch Log Ingestion consists of three components—, OpenSearch and —that fit into the OpenSearch ecosystem. The Data Prepper repository has several sample applications to help you get started.

    1. Log Ingestion relies on you adding log collection to your application’s environment to gather and send log data.

      (In the example below, is used as a log collector that collects log data from a file and sends the log data to Data Prepper).

    Example

    This example mimics the writing of log entries to a log file that are then processed by Data Prepper and stored in OpenSearch.

    Download or clone the . Then navigate to and open docker-compose.yml in a text editor. This file contains a container for:

    • Fluent Bit (fluent-bit)
    • Data Prepper (data-prepper)
    • OpenSearch Dashboards (opensearch-dashboards).

    Close the file and run docker-compose up --build to start the containers.

    Fluent-Bit will collect the log data and send it to Data Prepper:

    Data Prepper will process the log and index it:

    This should result in a single document being written to the OpenSearch cluster in the apache-logs index as defined in the log_pipeline.yaml file.

    Run the following command to see one of the raw documents in the OpenSearch cluster:

    The same data can be viewed in OpenSearch Dashboards by visiting the Discover page and searching the apache_logs index. Remember, you must create the index in OpenSearch Dashboards if this is your first time searching for the index.