HDFS

    How to create a HDFS Load Node

    The example below shows how to create a HDFS Load Node with :

    File Formats

    • CSV(Uncompressed)
    • JSON(JSON format for file system connector is not a typical JSON file but uncompressed newline delimited JSON.)
    • Avro(Support compression by configuring avro.codec.)
    • Orc(Compatible with Hive.)
    • Debezium-JSON
    • Canal-JSON
    • Raw

    Rolling Policy

    File Compaction

    The file sink supports file compactions, which allows applications to have smaller checkpoint intervals without generating a large number of files.

    Partition Commit

    Partition Commit Policy

    The partition strategy defines the specific operation of partition submission.

    • metastore：This strategy is only supported when hive.