java - Error while bulk load in hbase -
i trying hbase - bulkload through java mapreduce program. running program in eclipse.
but getting following error:
12/06/14 20:04:28 info jvm.jvmmetrics: initializing jvm metrics processname=jobtracker, sessionid= 12/06/14 20:04:28 warn util.nativecodeloader: unable load native-hadoop library platform... using builtin-java classes applicable 12/06/14 20:04:29 warn mapred.jobclient: use genericoptionsparser parsing arguments. applications should implement tool same. 12/06/14 20:04:29 warn mapred.jobclient: no job jar file set. user classes may not found. see jobconf(class) or jobconf#setjar(string). 12/06/14 20:04:29 info input.fileinputformat: total input paths process : 1 12/06/14 20:04:29 warn snappy.loadsnappy: snappy native library not loaded 12/06/14 20:04:29 info mapred.jobclient: running job: job_local_0001 12/06/14 20:04:29 info mapred.maptask: io.sort.mb = 100 12/06/14 20:04:29 info mapred.maptask: data buffer = 79691776/99614720 12/06/14 20:04:29 info mapred.maptask: record buffer = 262144/327680 12/06/14 20:04:29 warn mapred.localjobrunner: job_local_0001 java.lang.illegalargumentexception: can't read partitions file @ org.apache.hadoop.hbase.mapreduce.hadoopbackport.totalorderpartitioner.setconf(totalorderpartitioner.java:111) @ org.apache.hadoop.util.reflectionutils.setconf(reflectionutils.java:62) @ org.apache.hadoop.util.reflectionutils.newinstance(reflectionutils.java:117) @ org.apache.hadoop.mapred.maptask$newoutputcollector.<init>(maptask.java:560) @ org.apache.hadoop.mapred.maptask.runnewmapper(maptask.java:639) @ org.apache.hadoop.mapred.maptask.run(maptask.java:323) @ org.apache.hadoop.mapred.localjobrunner$job.run(localjobrunner.java:210) caused by: java.io.filenotfoundexception: file _partition.lst not exist. @ org.apache.hadoop.fs.rawlocalfilesystem.getfilestatus(rawlocalfilesystem.java:383) @ org.apache.hadoop.fs.filterfilesystem.getfilestatus(filterfilesystem.java:251) @ org.apache.hadoop.fs.filesystem.getlength(filesystem.java:776) @ org.apache.hadoop.io.sequencefile$reader.<init>(sequencefile.java:1424) @ org.apache.hadoop.io.sequencefile$reader.<init>(sequencefile.java:1419) @ org.apache.hadoop.hbase.mapreduce.hadoopbackport.totalorderpartitioner.readpartitions(totalorderpartitioner.java:296) @ org.apache.hadoop.hbase.mapreduce.hadoopbackport.totalorderpartitioner.setconf(totalorderpartitioner.java:82) ... 6 more 12/06/14 20:04:30 info mapred.jobclient: map 0% reduce 0% 12/06/14 20:04:30 info mapred.jobclient: job complete: job_local_0001 12/06/14 20:04:30 info mapred.jobclient: counters: 0
i googled lot didn't find solution.
i tried run same program console , following error came:
hadoop jar /home/user/hbase-0.90.4-cdh3u2/lib/zookeeper-3.3.3-cdh3u2.jar /home/user/hadoop-0.20.2-cdh3u2/test.jar bulkloadhbase_1 /bulkload.txt /out exception in thread "main" java.lang.nosuchmethodexception: org.apache.zookeeper.server.quorum.quorumpeer.main([ljava.lang.string;) @ java.lang.class.getmethod(class.java:1605) @ org.apache.hadoop.util.runjar.main(runjar.java:180)
my code:
import java.io.ioexception; import org.apache.hadoop.conf.configuration; import org.apache.hadoop.fs.path; import org.apache.hadoop.hbase.hbaseconfiguration; import org.apache.hadoop.hbase.client.put; import org.apache.hadoop.hbase.io.immutablebyteswritable; import org.apache.hadoop.hbase.mapreduce.hfileoutputformat; import org.apache.hadoop.hbase.mapreduce.putsortreducer; import org.apache.hadoop.hbase.mapreduce.hadoopbackport.totalorderpartitioner; import org.apache.hadoop.hbase.util.bytes; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.job; import org.apache.hadoop.mapreduce.mapper; import org.apache.hadoop.mapreduce.lib.input.fileinputformat; import org.apache.hadoop.mapreduce.lib.input.keyvaluetextinputformat; public class bulkloadhbase_1 { public static class bulkloadhbase_1mapper extends mapper<text, text, immutablebyteswritable, put>{ public void map(text key, text value, context context ) throws ioexception, interruptedexception { system.out.println("key "+key.tostring()); system.out.println("values : "+value); system.out.println("context : "+context); immutablebyteswritable ibw = new immutablebyteswritable(bytes.tobytes(key.tostring())); string val = value.tostring(); byte[] b = bytes.tobytes(val); put p = new put(bytes.tobytes(key.tostring())); p.add(bytes.tobytes("cf"),bytes.tobytes("c"),bytes.tobytes(val)); context.write(ibw, p); } } public static void main(string[] args) throws exception { configuration conf = new configuration(); job job = new job(conf, "bulk-load"); job.setjarbyclass(bulkloadhbase_1.class); job.setmapperclass(bulkloadhbase_1mapper.class); job.setreducerclass(putsortreducer.class); job.setoutputkeyclass(immutablebyteswritable.class); job.setoutputvalueclass(put.class); job.setpartitionerclass(totalorderpartitioner.class); job.setinputformatclass(keyvaluetextinputformat.class); fileinputformat.addinputpath(job, new path("/home/user/desktop/bulkload.txt")); hfileoutputformat.setoutputpath(job, new path("/home/user/desktop/hbase_bulkoutput/")); system.exit(job.waitforcompletion(true) ? 0 : 1); } }
did started hbase in distributed mode?! if line:
org.apache.hadoop.mapred.localjobrunner$job.run(localjobrunner.java:210)
in stack trace shows map reduce job running in local mode instead of distributed mode.
also note if want run command within console input files must reside on hadoop file system not on regular (e.g. ntfs or ext3) file system.
regards
Comments
Post a Comment