python - Hadoop Non-splittable TextInputFormat -

-

is there way have whole file sent mapper without being split?

i have read this wondering if there way of doing same thing without having generate intermediate file. ideally, existing option on command line hadoop.

i using streaming facility python scripts on amazon emr.

just set configuration property mapred.min.split.size huge (10g):

-d mapred.min.split.size=10737418240

or compress input file using codec isn't splittable (gzip). .gz extension, textinputformat return false issplittable(filesystem, path) method


Comments

Post a Comment

Popular posts from this blog

How do you change vbulletin's home page to the forums instead of the default (CMS or Activity Stream)? -

-
i have created website using vbulletin have installed cms , blog modules in in admin default setting forums when enter homepage url redirecting me http://www.demo.com/content/ content cms page please help for vbulletin versions < 4.2. need modify forum index page located in top level of vbulletin directory. based on url in question it's file: http://www.demo.com/index.php here contents of file, can choose whether cms or forum default home page. /** * can choose default script here. uncomment appropriate line * set default script. note: uncomment 1 of these, must * add // comment out script(s) not want use * default script. * * can choose default script if not plan move * file root of website. */ /** * use cms default script: */ require('content.php'); /** * use forum default script: */ // require('forum.php'); for vbulletin versions >= 4.2. there's new "navigation manager". you'll find control settin...
Read more

c++ - Troubles when compiling phash program -

-
here (probably) simple problem: i'm trying use perceptual hashing library phash ubuntu 11.10. had ffmpeg installed, way i've done: sudo apt-get install libphash0 sudo apt-get install libphash0-dev then tried compile program: #include <iostream> #include <phash.h> using namespace std; int main() { ulong64 myhash=0; ph_dct_imagehash("test.jpg", myhash); cout<<myhash<<endl; } when compiling, prints out: undefined reference `ph_dct_imagehash' any suggestion? should do? in advance! matteo monti you forgot link phash library, linker cannot find function. try adding -l phash gcc command line (or update makefile). if doesn't work, maybe you'll need specify library path (the location of *.a file) using -l "/usr/lib/"
Read more