System architecture

    When you config to store TSFile on HDFS, your data files will be in distributed storage. The system architecture is as below:

    Config and usage

    If you want to store TSFile and related data files in HDFS, here are the steps:

    Build server and Hadoop module by:

    Then, copy the target jar of Hadoop module hadoop-tsfile-0.9.3-jar-with-dependencies.jar into server target lib folder .../server/target/iotdb-server-0.9.3/lib.

    Edit user config in . Related configurations are:

    • tsfile_storage_fs
    • core_site_path
    Namecore_site_path
    DescriptionAbsolute file path of core-site.xml if Tsfile and related data files are stored in HDFS.
    TypeString
    Default/etc/hadoop/conf/core-site.xml
    EffectiveAfter restart system
    Namehdfs_site_path
    DescriptionAbsolute file path of hdfs-site.xml if Tsfile and related data files are stored in HDFS.
    TypeString
    Default/etc/hadoop/conf/hdfs-site.xml
    EffectiveAfter restart system
    • hdfs_ip
    Namehdfs_ip
    DescriptionIP of HDFS if Tsfile and related data files are stored in HDFS. If there are more than one hdfs_ip in configuration, Hadoop HA is used.
    TypeString
    Defaultlocalhost
    EffectiveAfter restart system
    • hdfs_port
    • dfs_nameservices
    Namehdfs_nameservices
    DescriptionNameservices of HDFS HA if using Hadoop HA
    TypeString
    Defaulthdfsnamespace
    EffectiveAfter restart system
    • dfs_ha_namenodes
    Namehdfs_ha_namenodes
    DescriptionNamenodes under DFS nameservices of HDFS HA if using Hadoop HA
    TypeString
    Defaultnn1,nn2
    EffectiveAfter restart system
    • dfs_ha_automatic_failover_enabled
    Namedfs_ha_automatic_failover_enabled
    DescriptionWhether using automatic failover if using Hadoop HA
    TypeBoolean
    Defaulttrue
    EffectiveAfter restart system
    • dfs_client_failover_proxy_provider
    Namehdfs_use_kerberos
    DescriptionWhether use kerberos to authenticate hdfs
    TypeString
    Defaultfalse
    EffectiveAfter restart system
    • kerberos_keytab_file_path
    Namekerberos_keytab_file_path
    DescriptionFull path of kerberos keytab file
    TypeString
    Default/path
    EffectiveAfter restart system
    • kerberos_principal
    Namekerberos_principal
    DescriptionKerberos pricipal
    TypeString
    Defaultyour principal
    EffectiveAfter restart system

    If you’d like to reset storage file system to local, just edit configuration tsfile_storage_fs to LOCAL. In this situation, if you have already had some data files on HDFS, you should either download them to local and move them to your config data file folder ( by default), or restart your process and import data to IoTDB.

    Frequent questions

    1. What Hadoop version does it support?

    A: Both Hadoop 2.x and Hadoop 3.x can be supported.

    1. When starting the server or trying to create timeseries, I encounter the error below:

    A: It indicates that you forget to put Hadoop module dependency in IoTDB server. You can solve it by:

    • Build Hadoop module: mvn clean package -pl hadoop -am -Dmaven.test.skip=true
    • Copy the target jar of Hadoop module hadoop-tsfile-0.9.3-jar-with-dependencies.jar into server target lib folder .