Tuesday 24 June 2014

Create Sqoop Action in Oozie using Hue WEB UI


1. Copy the DB driver into ooze's share lib
sudo -u oozie mkdir /tmp
sudo -u oozie mv sqljdbc4-3.0.jar /tmp
sudo -u oozie  hadoop fs -put /tmp/sqljdbc4-3.0.jar /user/oozie/share/lib/sqoop

or
put sqljdbc.jar to $AppPath/lib with the jars of pig script.

2. Error: Error parsing arguments for import

This is because Hue will submit this script through Oozie Sqoop Action. It has a particular way to specify the arguments. 
The Sqoop command can be specified either using the command element or multiple arg elements.
When using the command element, Oozie will split the command on every space into multiple arguments.
When using the arg elements, Oozie will pass each argument value as an argument to Sqoop.
The arg variant should be used when there are spaces within a single argument.
For example, --query "select * from A"

1. Argument way
Params Value:
import
--connect
jdbc:sqlserver://xxxxxx
--username
hadoopuser
--password
mypwd(no '')
--query
"select * from A where $CONDITIONS"
--fields-terminated-by
\t

2. Command way
<command>import --connect jdbc:sqlserver://xxxx --username hadoopuser -password mypwd</command>
<arg>--query</arg>
<arg>"select * from A where $CONDITIONS"<arg>

3. --options-file way
<arg>import</arg>
<arg>--options-file</arg>
<arg>${optionFile}</arg>
<file>${optionFile}</file>

in job.properties file
optionFile=option.par

option.par is in the same folder with workflow.xml


4. Import workflow.xml in Hue, but missing arguments tags.
  • The "argument" tags are not imported
  • The "exec" and "file" tags are imported but, add strings incorrectly

This is a bug of Hue. And will be fixed in Hue 2.5


Reference:

https://oozie.apache.org/docs/4.0.1/DG_SqoopActionExtension.html
https://issues.cloudera.org/browse/HUE-1321

https://gist.github.com/tmusabbir/7013556
http://grokbase.com/t/cloudera/cdh-user/1334q57427/issue-with-sqoop-options-file-option-in-oozie

1 comment: