CREATE DATASOURCE TABLE

    Syntax

    Note that, the clauses between the USING clause and the AS SELECT clause can come in as any order. For example, you can write COMMENT table_comment after TBLPROPERTIES.

    • table_identifier

      Specifies a table name, which may be optionally qualified with a database name.

      Syntax: [ database_name. ] table_name

    • USING data_source

      Data Source is the input format used to create the table. Data source can be CSV, TXT, ORC, JDBC, PARQUET, etc.

    • OPTIONS

      Options of data source which will be injected to storage properties.

    • Partitions are created on the table, based on the columns specified.

    • CLUSTERED BY

      Partitions created on the table will be bucketed into fixed buckets based on the column specified for bucketing.

      NOTE: Bucketing is an optimization technique that uses buckets (and bucketing columns) to determine data partitioning and avoid data shuffle.

    • SORTED BY

      Specifies an ordering of bucket columns. Optionally, one can use ASC for an ascending order or DESC for a descending order after any column names in the SORTED BY clause. If not specified, ASC is assumed by default.

    • INTO num_buckets BUCKETS

      Specifies buckets numbers, which is used in CLUSTERED BY clause.

    • Path to the directory where table data is stored, which could be a path on distributed storage like HDFS, etc.

    • COMMENT

      A string literal to describe the table.

    • TBLPROPERTIES

      A list of key-value pairs that is used to tag the table definition.

    • AS select_statement

      The table is populated using the data from the select statement.

    Data Source Interaction

    A Data Source table acts like a pointer to the underlying data source. For example, you can create a table “foo” in Spark which points to a table “bar” in MySQL using JDBC Data Source. When you read/write table “foo”, you actually read/write table “bar”.

    For CREATE TABLE AS SELECT, Spark will overwrite the underlying data source with the data of the input query, to make sure the table gets created contains exactly the same data as the input query.

    2. --Use data source
    3. CREATE TABLE student (id INT, name STRING, age INT) USING CSV;
    5. --Use data from another table
    6. CREATE TABLE student_copy USING CSV
    8. --Omit the USING clause, which uses the default data source (parquet by default)
    9. CREATE TABLE student (id INT, name STRING, age INT);
    11. --Use parquet data source with parquet storage options
    12. --The columns 'id' and 'name' enable the bloom filter during writing parquet file,
    13. --column 'age' does not enable
    14. CREATE TABLE student_parquet(id INT, name STRING, age INT) USING PARQUET
    16.       'parquet.bloom.filter.enabled'='true',
    17.       'parquet.bloom.filter.enabled#age'='false'
    18.     );
    20. --Specify table comment and properties
    21. CREATE TABLE student (id INT, name STRING, age INT) USING CSV
    22.     COMMENT 'this is a comment'
    23.     TBLPROPERTIES ('foo'='bar');
    25. --Specify table comment and properties with different clauses order
    26. CREATE TABLE student (id INT, name STRING, age INT) USING CSV
    27.     TBLPROPERTIES ('foo'='bar')
    28.     COMMENT 'this is a comment';
    30. CREATE TABLE student (id INT, name STRING, age INT)
    31.     USING CSV
    33.     CLUSTERED BY (Id) INTO 4 buckets;
    35. --Create partitioned and bucketed table through CTAS
    36. CREATE TABLE student_partition_bucket
    37.     USING parquet
    38.     PARTITIONED BY (age)
    39.     CLUSTERED BY (id) INTO 4 buckets
    40.     AS SELECT * FROM student;
    42. --Create bucketed table through CTAS and CTE
    43. CREATE TABLE student_bucket
    44.     USING parquet
    45.     CLUSTERED BY (id) INTO 4 buckets (
    46.     WITH tmpTable AS (
    47.         SELECT * FROM student WHERE id > 100
    48.     )
    49.     SELECT * FROM tmpTable