Tuesday 2 August 2016

Setup Hadoop_Home in Windows

When ran Spark App in IntelliJ in Windows System, we have to setup HADOOP_HOME for Spark.
Otherwise, you will see below errors.


16/08/02 10:51:11 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
at org.apache.spark.sql.execution.datasources.PartitioningUtils$.<init>(PartitioningUtils.scala:415)


Solution:


  1. Download winutils.exe (http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe) and save it in c:\Users\%USERNAME%\winutil\bin.
  2. Set HADOOP_HOME environment variable:
    setx HADOOP_HOME "c:\Users\%USERNAME%\winutil"


1 comment:

  1. Hey, I have tried a lot of things and I'm still having this problem. Are there any other commands you know that could help me solve this? Thanks a lot!

    ReplyDelete