Managing Connectors

    • Deploy builtin connectors
    • Monitor and update running connectors with Pulsar Admin CLI
    • Deploy customized connectors
    • Upgrade a connector

    Pulsar bundles several builtin connectors that should be used for moving data in and outof commonly used systems such as databases, messaging systems. Getting set up to use these builtin connectors is simple. You can followthe on installing builtin connectors. After setup, allthe builtin connectors will be automatically discovered by Pulsar brokers (or function-workers), so no additional installation steps arerequired.

    Configuring Pulsar IO connectors is straightforward. What you need to do is to provide a yaml configuration file when your run connectors.The yaml configuration file basically tells Pulsar where to locate the sources and sinks and how to connect those sources and sinks with Pulsar topics.

    Below is an example yaml configuration file for Cassandra Sink:

    The example yaml basically tells Pulsar which Cassandra cluster to connect, what is the and columnFamily to be used in Cassandra for collecting data,and how to map a Pulsar message into Cassandra table key and columns.

    For details, consult the documentation for .

    You can submit a source to be run in an existing Pulsar cluster using a command of this form:

      Here’s an example command:

      1. bin/pulsar-admin sources create --classname org.apache.pulsar.io.twitter.TwitterFireHose --archive ~/application.jar --tenant test --namespace ns1 --name twitter-source --destination-topic-name twitter_data

      Instead of submitting a source to run on an existing Pulsar cluster, you alternatively can run a source as a process on your local machine:

      1. bin/pulsar-admin sources localrun --classname  org.apache.pulsar.io.twitter.TwitterFireHose --archive ~/application.jar --tenant test --namespace ns1 --name twitter-source --destination-topic-name twitter_data

      If you are submitting a built-in source, you don't need to specify —classname and —archive.You can simply specify the source type —source-type. The command to submit a built-in source isin following form:

      Here's an example to submit a Kafka source:

      1. ./bin/pulsar-admin sources create \
      2.     --tenant test-tenant \
      4.     --name test-kafka-source \
      5.     --destination-topic-name pulsar_sink_topic \
      6.     --source-type kafka
      1. ./bin/pulsar-admin sinks create --classname  <classname> --archive <jar-location> --tenant test --namespace <namespace> --name <sink-name> --inputs <input-topics>

      Here’s an example command:

      1. ./bin/pulsar-admin sinks create --classname  org.apache.pulsar.io.cassandra --archive ~/application.jar --tenant test --namespace ns1 --name cassandra-sink --inputs test_topic

      Instead of submitting a sink to run on an existing Pulsar cluster, you alternatively can run a sink as a process on your local machine:

      If you are submitting a built-in sink, you don't need to specify —classname and —archive.You can simply specify the sink type —sink-type. The command to submit a built-in sink isin following form:

      1. ./bin/pulsar-admin sinks create \
      2.     --tenant <tenant> \
      3.     --name <sink-name> \
      5.     --sink-type <sink-type>

      Here's an example to submit a Cassandra sink:

      1. ./bin/pulsar-admin sinks create \
      2.     --tenant test-tenant \
      3.     --namespace test-namespace \
      4.     --name test-cassandra-sink \
      5.     --inputs pulsar_input_topic \
      6.     --sink-type cassandra

      Since Pulsar IO connectors are running as Pulsar Functions, so you can use commandsavailable in the pulsar-admin CLI tool.

      1. bin/pulsar-admin functions get \
      2.     --tenant <tenant> \
      3.     --namespace <namespace> \