Scalar User Defined Functions (UDFs)

    To define the properties of a user-defined function, the user can use some of the methods defined in this class.

    • asNonNullable(): UserDefinedFunction

    • asNondeterministic(): UserDefinedFunction

      Updates UserDefinedFunction to nondeterministic.

    • Updates UserDefinedFunction with a given name.

    Find full example code at “examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedScalar.scala” in the Spark repo.

    2. import org.apache.spark.sql.api.java.UDF1;
    3. import org.apache.spark.sql.expressions.UserDefinedFunction;
    4. import static org.apache.spark.sql.functions.udf;
    5. import org.apache.spark.sql.types.DataTypes;
    7. SparkSession spark = SparkSession
    8.   .builder()
    9.   .appName("Java Spark SQL UDF scalar example")
    10.   .getOrCreate();
    12. // Define and register a zero-argument non-deterministic UDF
    13. // UDF is deterministic by default, i.e. produces the same result for the same input.
    14. UserDefinedFunction random = udf(
    15.   () -> Math.random(), DataTypes.DoubleType
    16. random.asNondeterministic();
    17. spark.udf().register("random", random);
    18. spark.sql("SELECT random()").show();
    19. // +-------+
    21. // +-------+
    22. // |xxxxxxx|
    23. // +-------+
    25. // Define and register a one-argument UDF
    26. spark.udf().register("plusOne",
    27.   (UDF1<Integer, Integer>) x -> x + 1, DataTypes.IntegerType);
    28. spark.sql("SELECT plusOne(5)").show();
    29. // +----------+
    30. // |plusOne(5)|
    31. // +----------+
    32. // |         6|
    33. // +----------+
    35. // Define and register a two-argument UDF
    36. UserDefinedFunction strLen = udf(
    37.   (String s, Integer x) -> s.length() + x, DataTypes.IntegerType
    38. spark.udf().register("strLen", strLen);
    40. // +------------+
    41. // |UDF(test, 1)|
    42. // +------------+
    43. // |           5|
    44. // +------------+
    46. // UDF in a WHERE clause
    47. spark.udf().register("oneArgFilter",
    48.   (UDF1<Long, Boolean>) x -> x > 5, DataTypes.BooleanType);
    49. spark.range(1, 10).createOrReplaceTempView("test");
    50. spark.sql("SELECT * FROM test WHERE oneArgFilter(id)").show();
    51. // +---+
    52. // | id|
    53. // +---+
    54. // |  6|
    55. // |  7|
    56. // |  8|
    57. // |  9|
    58. // +---+