.. _run_on_windows: Use Sparkling Water in Windows Environments ------------------------------------------- Windows environments require several additional steps to run Spark and Sparkling Water. A great summary of the configuration steps is available `here `__. To use Sparkling Water in Windows environments: 1. Download the appropriate Spark distribution from the `Spark Downloads page `__. 2. Point the ``SPARK_HOME`` variable to the location of your Spark distribution: .. code:: bat SET SPARK_HOME= 3. From https://github.com/steveloughran/winutils, download ``winutils.exe`` for the Hadoop version that is referenced by your Spark distribution (For example, for ``spark-SUBST_SPARK_VERSION-bin-hadoop2.7.tgz``, you need ``wintutils.exe`` for Hadoop 2.7.) 4. Move ``winutils.exe`` into a new directory ``%SPARK_HOME%\hadoop\bin`` and set: .. code:: bat SET HADOOP_HOME=%SPARK_HOME%\hadoop 5. Create a new file ``%SPARK_HOME%\hadoop\conf\hive-site.xml``, which sets up a default Hive scratch directory. The best location is a writable temporary directory, for example ``%TEMP%\hive``: .. code:: xml hive.exec.scratchdir PUT HERE LOCATION OF TEMP FOLDER Scratch space for Hive jobs **Note**: You can also use the Hive default scratch directory, which is ``c:\tmp\hive``. In this case, you need to create the directory manually and call ``winutils.exe chmod -R 777 c:\tmp\hive`` to set up the correct permissions. 6. Set the ``HADOOP_CONF_DIR`` property: .. code:: bat SET HADOOP_CONF_DIR=%SPARK_HOME%\hadoop\conf 7. Go to the Sparkling Water directory and run the Sparkling Water shell: .. code:: bat bin/sparkling-shell.cmd