DataProxy Plugin

    Concepts and Models

    DataProxy is a message flow architecture based on Apache Flume with its + Channel + Sink components. Here we focus on the sink layer alone.

    • MessageQueueZoneSink: The standard MQ sink provided by DataProxy, supposedly to support all kinds of MQ types.
    • MessageQueueHandler: The abstract MQ handler interface that deals with connecting, sending data to, and disconnecting the MQ cluster.
    • EventHandler: The interface to convert MQ message header and body when required. For example to convert the data protocol.

    The concepts introduced above can be represented by the following figure:

    Development Process

    • Implement the subclass plugin of MessageQueueHandler, namely KafKaHandler and its init / start /stop / send methods.
    • Implenent the EventHandler interface as ProtoBufferEventHandler and its parseHeader / parseBody method
    3.     @Override
    4.             INLONG_COMPRESSED_TYPE compressType) {
    5.         // retrieve, process and convert event header
    8.     @Override
    9.     public byte[] parseBody(IdTopicConfig idConfig, BatchPackProfile profile, INLONG_COMPRESSED_TYPE compressType)
    10.             throws IOException {
    11.         // retrieve and repack event to ProtoBuffer message
    12. }

    (See the full implementation org.apache.inlong.dataproxy.sink.mq.kafka.KafkaHandler from inlong-dataproxy module)

    Plugin Configuration