Pulsar connector overview

    Pulsar IO connectors enable you to easily create, deploy, and manage connectors that interact with external systems, such as Apache Cassandra, , and many others.

    Pulsar IO connectors come in two types: source and sink.

    This diagram illustrates the relationship between source, Pulsar, and sink:

    Common sources include other messaging systems and firehose-style data pipeline APIs.

    For the complete list of Pulsar built-in source connectors, see source connector.

    For the complete list of Pulsar built-in sink connectors, see .

    Processing guarantees are used to handle errors when writing messages to Pulsar topics.

    • Source: Pulsar ensures that writing messages to Pulsar topics respects to the processing guarantees. It is within Pulsar’s control.

    • Sink: the processing guarantees rely on the sink implementation. If the sink implementation does not handle retries in an idempotent way, the sink does not respect to the processing guarantees.

    When creating a connector, you can set the processing guarantee with the following semantics:

    • ATLEAST_ONCE
    • ATMOST_ONCE
    • Source
    • Sink

    For more information about the options of pulsar-admin sources create, see here.

    For more information about the options of , see .

    After creating a connector, you can update the processing guarantee with the following semantics:

    • ATLEAST_ONCE
    • ATMOST_ONCE
    • EFFECTIVELY_ONCE

    Here takes Admin CLI as an example. For more information about REST API or JAVA Admin API, see here.

    • Sink

    For more information about the options of pulsar-admin sources update, see .

    For more information about the options of pulsar-admin sinks update, see here.

    You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the with sources and sinks subcommands. For the latest and complete information, see .

    Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via the Connector Admin CLI or Functions Admin CLI, an instance is started on a worker. For more information, see .