Install Spark on windows

I found a beautiful YouTube movie that showed how Spark can be installed on windows. I found this on https://www.youtube.com/watch?v=WlE7RNdtfwE . The movie provided a clear guide how to this up. It provides a step by step approach.

The first step is install JDK. I installed this from https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html. This allowed me install version 8, that was also used in the YouTube movie. The file that contains JDK is called jdk-8u181-windows-x64.exe. It will be installed in C:\Program Files\Java\jdk1.8.0_181 . This installation path will be used as environment variable for JAVA_HOME, that is then set to C:\Program Files\Java\jdk1.8.0_181 . The path must be changed as %JAVA_HOME%\bin must be added to the path.

The second step is to add winutils.exe to the system. It can be downloaded from GitHub or here. This can be stored in c:\winutils\bin. Create an environment variable HADOOP_HOME, that refers to c:\winutils. The path must be changed as %HADOOP_HOME%\bin must be added to the path.

The third step is to download Spark from spark.apache.org . I downloaded version 1.6.3. I downloaded the pre built version for Hadoop 2.6, which is the closest to the versions used in the YouTube movie. This must unzipped. I unzipped this into c;\spark-1.6.3-bin-hadoop2.6 . An environment variable SPARK_HOME was set to c:\spark-1.6.3-bin-hadoop2.6 . The path must be updated to %SPARK_HOME%\bin .

We then have as environment:

and the bin directories to HADOOP_HOME, JAVA_HOME and SPARK_HOME being added to the path.

The last step is install the scala IDE from scala-ide.org. This provides a directory eclipse that must stored on a disk.

We then can boot up SPARK with spark-shell that produces:

He, got Spark on windows now. Nice.

Door tom