Overview

    The audit reconciliation is based on the log reporting time, and each service participating in the audit will conduct real-time reconciliation according to the same log time. Through audit reconciliation, we can clearly understand InLong The transmission status of each module, and whether the data stream is lost or repeated

    1. The audit SDK is nested in the service that needs to be audited, audits the service, and sends the audit result to the audit access layer
    2. The audit proxy writes audit data to MQ (Pulsar or TubeMQ)
    3. The distribution service consumes the audit data of MQ, and writes the audit data to MySQL and Elasticsearch
    4. Application scenarios mainly include report display, audit reconciliation, etc.

    Audit Dimension

    Audit ID

    The receiving and sending of each module are respectively an independent audit item ID

    The transmission protocol between sdk, access layer, and distribution layer is Protocol Buffers

    Audit SDK Implementation Details

    1. Support local disaster recovery 2. Data Uniqueness 3. Reduce data loss caused by abnormal restart

    Main Logic Diagram

    Overview - 图2

    1. The sdk provides the add interface externally. The parameters are: audit_id, inlong_group_id, inlong_stream_id, number, size.
    2. The sdk uses log time+audit_id+inlong_group_id+inlong_stream_id as the key to perform real-time statistics.
    3. When the sending cycle is satisfied or the business program is actively triggered, the SDK will package the statistical results with the PB protocol and send the audit access layer.
    4. If (4) fails to send, put it into the failure queue, and continue to send in the next cycle.
    5. When the failure queue is greater than the threshold, perform disaster recovery through local files.

    Service Discovery

    Disaster Recovery

    1. When the SDK fails to send the access layer, it will be placed in the failure queue.
    2. When the failure queue reaches the threshold, it will be written to the local disaster recovery file.
    3. When the local disaster recovery file reaches the threshold, the old data will be eliminated (eliminated by time).

    Access layer Implementation

    Target

    1.High reliability
    2.at least once

    Main Logic Diagram

    Overview - 图4

    1. After the access layer receives the packet sent by the sdk, it writes the message queue.
    2. After writing the message queue successfully, return success to the sdk.
    3. The data protocol of the message queue is the PB protocol.
    4. Set the ack of the write message queue to -1 or all.

    1. High real-time performance (minute level) 2. Can operate tens of billions of audit data per day 3. Can be deduplicated

    Main Logic Diagram

    1. Distribution service AuditDds consumes messages in real time.
    2. According to the audit ID in the audit data, route the data to the corresponding Elasticsearch cluster.
    3. Each audit ID corresponds to an Elasticsearch index.

    Elasticsearch Index Design

    Index Name

    The index name consists of date + audit item ID, such as 20211019_1, 20211019_2.

    Index Field Schema

    Elasticsearch Index Storage Period

    Elasticsearch Write Design

    The relationship between inlong_group_id, inlong_stream_id, audit ID and Elasticsearch index

    Overview - 图6 The relationship between inlong_group_id, inlong_stream_id, audit ID and Elasticsearch index is 1:N in system design and service implementation

    Write Routing Policy

    Use inlong_group_id and inlong_stream_id to route to Elasticsearch shards to ensure that the same inlong_group_id and inlong_stream_id are stored in the same shard When writing the same inlong_group_id and inlong_stream_id to the same shard, when querying and aggregating, only one shard needs to be processed, which can greatly improve performance

    Optional DeduplicationBy doc_id

    Elasticsearch is resource-intensive for real-time deduplication. This function is optional through configuration.

    Use bulk to write, each batch of 5000, improve the write performance of the Elasticsearch cluster

    MySQL Distribution Implementation

    Target

    1. High real-time performance (minute level)
    2. Simple to deploy
    3. Can be deduplicated

    Main Logic Diagram

    Overview - 图8 MySQL distribution supports distribution to different MySQL instances according to the audit ID, and supports horizontal expansion.

    Usage introduction

    1. When the audit scale of the business is relatively small, less than ten million per day, you can consider using MySQL as the audit storage. Because the deployment of MySQL is much simpler than that of Elasticsearch, the resource cost will be much less.

    Main Logic Diagram

    UI Interface Display

    The front-end page pulls the audit data of each module through the interface layer and displays it.