Download files from sftp with input is a list of folder paths.
Don't miss "#!/usr/bin/python" in the beginning of .py
Otherwise, Error:
import: command not found
syntax error near unexpected token `('
#!/usr/bin/python from __future__ import print_function import os import time import zipimport import sys ftp_username='xxx' ftp_password='xxx' ftp_host='xxx' importer_pysftp = zipimport.zipimporter('pysftp.mod') pysftp = importer_pysftp.load_module('pysftp') importer_paramiko = zipimport.zipimporter('paramiko.mod') paramiko = importer_paramiko.load_module('paramiko') #download the whole folder to local sftp = pysftp.Connection(ftp_host, username=ftp_username, password=ftp_password) for line in sys.stdin: d_path = line.strip() sftp.walktree(d_path, fcallback=wtcb.file_cb, dcallback=wtcb.dir_cb, ucallback=wtcb.unk_cb) wtcb = pysftp.WTCallbacks() for fpath in wtcb.flist: print(fpath) sftp.close()-output is necessary, even though no output for mapper.
For CHD5.1.2
find /opt/cloudera/parcels/ -name *hadoop-streaming*.jar
to locate the path of hadoop-streaming.jar
$hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.7.0.jar \
-D mapred.reduce.tasks=10 \
-mapper ftp_file_list_mapper.py \
-input data/folder_list \
-output data/file_list \
-file ftp_file_list_mapper.py \
-file lib/pysftp.mod \
-file lib/paramiko.mod
We can set the reduce task number, even for a map alone job. It will use the default IdentityReducer from mapred.lib.
Thanks for sharing this informative information. For more you may refer http://www.s4techno.com/hadoop-training-in-pune/
ReplyDeletegood..
ReplyDeletePython Internship
Dotnet Internship
Java Internship
Web Design Internship
Php Internship
Android Internship
Big Data Internship
Cloud Internship
Hacking Internship
Robotics Internship
good for work
ReplyDeleteFree Internship for cse students in Chennai
R Programming Internship
Hadoop Training in Chennai
Free Internship Training in Chennai
Robotics Training chennai
Summer Internship For BSC students
Internships in Chennai for CSE
CCNA Institute in Chennai
Data Science Internship in Chennai
Aeronautical Engineering Internship
GOOD
ReplyDeletenodejs while loop
icici bank po interview questions and answers pdf
craterzone aptitude test
zensoft recruitment process
java developer resume 1 years experience
python developer resume pdf
infrrd private limited interview questions
js int max value
delete * from table oracle
t systems pune aptitude questions
Very Nice...
ReplyDeleteinternship in chennai for ece students with stipend
internship for mechanical engineering students in chennai
inplant training in chennai
free internship in pune for computer engineering students
internship in chennai for mca
iot internships
internships for cse students in
implant training in chennai
internship for aeronautical engineering students in bangalore
inplant training certificate
3d-modeler-resume-samples
ReplyDelete3d modeler resume samples
accounting-assistant-resume-sample
accounting-clerk-resume-sample
accounting-manager-resume-samples
account-manager-resume-examples
accounts-payable-resume-sample
admin-manager-resume-samples
advocate-resume-sample
advocate-resume-sample
good... nice... very useful..
ReplyDeleteassistant-director-resume-format
director-resume-sample
assistant-professor-resume-sample
back-office-executive-resume-samples
bank-branch-manager-resume-samples
basketball-coach-resume-sample-coach-resumes
bca-fresher-resume-sample
best-general-manager-resume-example
bpo-resume-freshers-sample
bpo-resume-samples-for-freshers
useful information..nice..
ReplyDeletedevops-engineer-resume-samples
digital-marketing-resume-samples
digital-marketing-resume-samples
electronics-engineer-resume-sample
engineering-lab-technician-resume-samples
english-teacher-cv-sample
english-teacher-resume-example
english-teacher-resume-sample
excel-expert-resume-sample
executive-secretary-resume-samples
useful information..nice..
ReplyDeletedevops-engineer-resume-samples
digital-marketing-resume-samples
digital-marketing-resume-samples
electronics-engineer-resume-sample
engineering-lab-technician-resume-samples
english-teacher-cv-sample
english-teacher-resume-example
english-teacher-resume-sample
excel-expert-resume-sample
executive-secretary-resume-samples
good .........very useful
ReplyDeletefresher-marketing-resume-sample
front-end-developer-resume-sample
full-stack-developer-resume-samples
fund-accountant-resume-samples
general-ledger-accountant-resume-sample
government-jobs-resume
hadoop-developer-sample-resume
hadoop-developer-sample-resume
hardware-and-networking-resume-samples
hardware-engineer-resume-sample
Good..
ReplyDeletecoronavirus update
inplant training in chennai
inplant training
inplant training in chennai for cse
inplant training in chennai for ece
inplant training in chennai for eee
inplant training in chennai for mechanical
internship in chennai
online internships
Thanks for the information. The one thing I have noticed in this website is that you were continuously updating the changes that you have been made. It is a good sign to attract more people and I appreciate it. Hope more update and news from you. thank u so much
ReplyDeleteAi & Artificial Intelligence Course in Chennai
PHP Training in Chennai
Ethical Hacking Course in Chennai Blue Prism Training in Chennai
UiPath Training in Chennai