{"id":1964,"date":"2018-09-23T15:26:09","date_gmt":"2018-09-23T13:26:09","guid":{"rendered":"http:\/\/van-maanen.com\/?p=1964"},"modified":"2018-09-23T15:26:09","modified_gmt":"2018-09-23T13:26:09","slug":"install-spark-on-windows","status":"publish","type":"post","link":"http:\/\/van-maanen.com\/?p=1964","title":{"rendered":"Install Spark on windows"},"content":{"rendered":"<p>I found a beautiful YouTube movie that showed how Spark can be installed on windows. I found this on https:\/\/www.youtube.com\/watch?v=WlE7RNdtfwE . The movie provided a clear guide how to this up. It provides a step by step approach.<\/p>\n<p>The first step is install JDK. I installed this from https:\/\/www.oracle.com\/technetwork\/java\/javase\/downloads\/jdk8-downloads-2133151.html. This allowed me install version 8, that was also used in the YouTube movie. The file that contains JDK is called jdk-8u181-windows-x64.exe. It will be installed in C:\\Program Files\\Java\\jdk1.8.0_181 . This installation path will be used as environment variable for JAVA_HOME, that is then set to C:\\Program Files\\Java\\jdk1.8.0_181 . The path must be changed as %JAVA_HOME%\\bin must be added to the path.<\/p>\n<p>The second step is to add winutils.exe to the system. It can be downloaded from GitHub or <a href=\"http:\/\/van-maanen.com\/wp-content\/uploads\/2018\/09\/winutils.exe\"> here. <\/a> This can be stored in c:\\winutils\\bin. Create an environment variable HADOOP_HOME, that refers to c:\\winutils. The path must be changed as %HADOOP_HOME%\\bin must be added to the path.<\/p>\n<p>The third step is to download Spark from spark.apache.org . I downloaded version 1.6.3. I downloaded the pre built version for Hadoop 2.6, which is the closest to the versions used in the YouTube movie. This must unzipped. I unzipped this into c;\\spark-1.6.3-bin-hadoop2.6 . An environment variable SPARK_HOME was set to c:\\spark-1.6.3-bin-hadoop2.6 . The path must be updated to %SPARK_HOME%\\bin .<\/p>\n<p>We then have as environment:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/van-maanen.com\/wp-content\/uploads\/2018\/09\/Screenshot-2018-09-23-16.51.07.png\" alt=\"\" width=\"487\" height=\"293\" class=\"alignnone size-full wp-image-1969\" \/> and the bin directories to HADOOP_HOME, JAVA_HOME and SPARK_HOME being added to the path.<\/p>\n<p>The last step is install the scala IDE from scala-ide.org. This provides a directory eclipse that must stored on a disk.<\/p>\n<p>We then can boot up SPARK with spark-shell that produces:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/van-maanen.com\/wp-content\/uploads\/2018\/09\/Screenshot-2018-09-23-17.40.14.png\" alt=\"\" width=\"769\" height=\"575\" class=\"alignnone size-full wp-image-1971\" \/><\/p>\n<p>He, got Spark on windows now. Nice.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I found a beautiful YouTube movie that showed how Spark can be installed on windows. I found this on https:\/\/www.youtube.com\/watch?v=WlE7RNdtfwE . The movie provided a clear guide how to this up. It provides a step by step approach. The first step is install JDK. I installed this from https:\/\/www.oracle.com\/technetwork\/java\/javase\/downloads\/jdk8-downloads-2133151.html. This allowed me install version 8, [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1964","post","type-post","status-publish","format-standard","hentry","category-allgemein"],"_links":{"self":[{"href":"http:\/\/van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts\/1964","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/van-maanen.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/van-maanen.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1964"}],"version-history":[{"count":0,"href":"http:\/\/van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts\/1964\/revisions"}],"wp:attachment":[{"href":"http:\/\/van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1964"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1964"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1964"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}