Skip to main content
Version: v3.0.2

Spark dependencies

If you want to interact with Feature Store from a Spark session, several dependencies need to be added on the Spark Classpath. Supported Spark versions are 4.1.x.

Using S3 as the Feature Store storage:​

  • io.delta:delta-spark_4.1_2.13:4.1.0
  • org.apache.hadoop:hadoop-aws:${HADOOP_VERSION}
note

HADOOP_VERSION is the hadoop version your Spark is built for.

Version of delta-spark library needs to match your Spark version. Version 4.1.x can be used by Spark 4.1.x.

Using Azure Gen2 as the Feature Store storage:​

  • io.delta:delta-spark_4.1_2.13:4.1.0
  • featurestore-spark-dependencies.jar
  • org.apache.hadoop:hadoop-azure:${HADOOP_VERSION}
note

HADOOP_VERSION is the hadoop version your Spark is built for.

Version of delta-spark library needs to match your Spark version. Version 4.1.x can be used by Spark 4.1.x.

The Spark dependencies jar can be downloaded from the Downloads page.

General configuration​

Spark needs to be started with the following configuration to ensure that the time travel queries are correct:

  • spark.sql.session.timeZone=UTC
  • spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
  • spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

In case of running Databricks 11.3 and higher, following options need to be set as well:

  • databricks.loki.fileSystemCache.enabled=false

If you do not have Apache Spark started, please start it first.

Feedback