Setup Hadoop_Home in Windows

When ran Spark App in IntelliJ in Windows System, we have to setup HADOOP_HOME for Spark.
Otherwise, you will see below errors.


16/08/02 10:51:11 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
at org.apache.spark.sql.execution.datasources.PartitioningUtils$.<init>(PartitioningUtils.scala:415)


Solution:

1. Download winutils.exe (http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe) and save it in c:\Users\%USERNAME%\winutil\bin.
2. Set HADOOP_HOME environment variable:

   setx HADOOP_HOME "c:\Users\%USERNAME%\winutil"