Array Expressions#

ArrayExcept#

By default, ArrayExcept is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.ArrayExcept.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Null handling and ordering may differ from Spark

ArrayIntersect#

By default, ArrayIntersect is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.ArrayIntersect.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Result array element order may differ from Spark when the right array is longer than the left (DataFusion probes the longer side).

  • array_intersect does not propagate non-UTF8_BINARY collations to the output array elements (https://github.com/apache/datafusion-comet/issues/2190)

ArrayJoin#

By default, ArrayJoin is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.ArrayJoin.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • Null handling may differ from Spark

  • array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190)

ArraysZip#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Not all input data types are supported; falls back to Spark for unsupported types

Size#

The following cases are not supported by Comet and always fall back to Spark, regardless of any allowIncompatible setting:

  • Only supports ArrayType input; MapType input is not supported

SortArray#

By default, SortArray is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set spark.comet.expression.SortArray.allowIncompatible=true to opt into Comet’s native implementation instead, which has the following differences from Spark:

  • When spark.comet.exec.strictFloatingPoint=true, sorting on floating-point types is not 100% compatible with Spark

The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):

  • Nested arrays with Struct or Null child values are not supported natively