HADOOPORACLE: 装载Hadoop 文件到数据库

问题:

1.在读取配置文件时，无法找到mapreduce.outputformat.class 的对应值

12/12/12 17:30:11 ERROR loader.OraLoader: mapreduce.outputformat.class is invalid

oracle.hadoop.loader.OraLoaderException: missing or invalid property "mapreduce.outputformat.class" value

解决：在执行hadoop   ${OLH_JAR}    oracle.hadoop.loader.OraLoader -conf /home/oracle/OLH/MyConf.xml 时，将MyConf.xml文件的路径写成了 HDFS文件系统的路径，将其改为LINUX 下的路径。

 

2.在进行连接时，无法连接监听器

解决:  没有配置好，本机的lisener,tnsname ,sqlnet 文件

 

3.在进行Loader时需要的文件， IN_DIRECTORY,OUT_DIRECTORY, DATAFILE,

重要的对象:

HDFS:

start:

mapred.input.dir,

   mapreduce.outputformat.class

   mapreduce.inputformat.class

  mapreduce.output.dir

end:

 oracle.hadoop.loader.loaderMapFile

oracle.hadoop.loader.targetTable

oracle.hadoop.loader.input.fieldNames

oracle.hadoop.loader.connection.url

 

-   <name>mapreduce.inputformat.class</name>

  <value>oracle.hadoop.loader.lib.input.DelimitedTextInputFormat</value>

 

 <name>mapred.input.dir</name>

  <value>/user/root/olh_lab_in/data.dat</value>

 

-   <name>mapreduce.outputformat.class</name>

  <value>oracle.hadoop.loader.lib.output.JDBCOutputFormat</value>

 

<name>mapred.output.dir</name>

  <value>/user/root/olh_lab_out</value>

 

- 

  <name>oracle.hadoop.loader.loaderMapFile</name>

  <value>file:///home/oracle/OLH/loaderMap_exercise1.xml</value>

    

<name>oracle.hadoop.loader.targetTable</name>
<value>SCOTT.OLH_TABLE</value>

 

<name>oracle.hadoop.loader.input.fieldNames</name>
<value>col1,col2,col3,col4</value>
  

 

<name>oracle.hadoop.loader.connection.url</name>

  <value>jdbc:oracle:thin:@${HOST}:${TCPPORT}/${SERVICE_NAME}</value>

 

<name>TCPPORT</name>

  <value>1521</value>

 

<name>HOST</name>

  <value>hadoop.oracle</value>

 

<name>ORACLE_SID</name>

  <value>hadoop</value>

 

<name>SERVICE_NAME</name>

  <value>hadoop</value>

 

<name>oracle.hadoop.loader.connection.user</name>

  <value>SCOTT</value>

  <description>AvroSchemaAwareDBInputFormat wants this case sensitive! (cause mapred.jdbc.username points here)</description>

 

<name>oracle.hadoop.loader.connection.password</name>

  <value>tiger</value>

 

 

http://docs.oracle.com/cd/E27101_01/doc.10/e27365/olh.htm

 

 

[oracle@hadoop admin]$ hadoop dfs -rmr /user/root/olh_lab_out
Deleted hdfs://hdnode1m:9000/user/root/olh_lab_out
[oracle@hadoop admin]$ hadoop jar $OLH_HOME/jlib/oraloader.jar oracle.hadoop.loader.OraLoader -conf /home/oracle/OLH/MyConf.xml
Oracle Loader for Hadoop Release 2.0.0 - Production

 

Copyright (c) 2011, 2012, Oracle and/or its affiliates. All rights reserved.

 

13/03/07 19:38:38 INFO loader.OraLoader: Oracle Loader for Hadoop Release 2.0.0 - Production

 

Copyright (c) 2011, 2012, Oracle and/or its affiliates. All rights reserved.

 

13/03/07 19:38:38 INFO loader.OraLoader: Built-Against: hadoop-0.20.2 hive-0.7.1-cdh3u3 avro-1.6.3 jackson-1.8.8
13/03/07 19:38:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
13/03/07 19:38:40 INFO loader.OraLoader: oracle.hadoop.loader.loadByPartition is disabled because table: OLH_TABLE is not partitioned
13/03/07 19:38:40 INFO loader.OraLoader: oracle.hadoop.loader.enableSorting disabled, no sorting key provided
13/03/07 19:38:40 INFO output.DBOutputFormat: Setting reduce tasks speculative execution to false for : oracle.hadoop.loader.lib.output.JDBCOutputFormat
13/03/07 19:38:40 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
13/03/07 19:38:40 WARN loader.OraLoader: Sampler error: the number of reduce tasks must be greater than one; the configured value is 1 . Job will continue without sampled information.
13/03/07 19:38:40 INFO loader.OraLoader: Sampling time=0D:0h:0m:0s:13ms (13 ms)
13/03/07 19:38:40 INFO loader.OraLoader: Submitting OraLoader job OraLoader
13/03/07 19:38:42 INFO input.FileInputFormat: Total input paths to process : 1
13/03/07 19:38:42 INFO input.FileInputFormat: Total input paths to process : 1
13/03/07 19:38:42 INFO mapred.MapTask: io.sort.mb = 100
13/03/07 19:38:43 INFO mapred.MapTask: data buffer = 79691776/99614720
13/03/07 19:38:43 INFO mapred.MapTask: record buffer = 262144/327680
13/03/07 19:38:43 INFO mapred.MapTask: Starting flush of map output
13/03/07 19:38:43 INFO mapred.MapTask: Finished spill 0
13/03/07 19:38:43 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
13/03/07 19:38:43 INFO mapred.LocalJobRunner:
13/03/07 19:38:43 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
13/03/07 19:38:43 INFO mapred.LocalJobRunner:
13/03/07 19:38:43 INFO mapred.Merger: Merging 1 sorted segments
13/03/07 19:38:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1196 bytes
13/03/07 19:38:43 INFO mapred.LocalJobRunner:
13/03/07 19:38:43 INFO output.DBOutputFormat: conf prop: defaultExecuteBatch: 100
13/03/07 19:38:43 INFO output.DBOutputFormat: conf prop: loadByPartition: false
13/03/07 19:38:43 INFO output.DBOutputFormat: Insert statement: INSERT INTO "SCOTT"."OLH_TABLE" ("COL1", "COL2", "COL3", "COL4") VALUES (?, ?, ?, ?)
13/03/07 19:38:43 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
13/03/07 19:38:43 INFO mapred.LocalJobRunner:
13/03/07 19:38:43 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
13/03/07 19:38:43 INFO output.JDBCOutputFormat: Committed work for task attempt attempt_local_0001_r_000000_0
13/03/07 19:38:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to /user/root/olh_lab_out
13/03/07 19:38:43 INFO mapred.LocalJobRunner: reduce > reduce
13/03/07 19:38:43 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
13/03/07 19:38:43 INFO loader.OraLoader: map 100% reduce 100%
13/03/07 19:38:43 INFO loader.OraLoader: Job complete: OraLoader (null)
13/03/07 19:38:43 INFO loader.OraLoader: Counters: 14
        FileSystemCounters
                FILE_BYTES_READ=18885058
                FILE_BYTES_WRITTEN=55486
                HDFS_BYTES_READ=82429
                HDFS_BYTES_WRITTEN=18957078
        Map-Reduce Framework
                Combine input records=0
                Combine output records=0
                Map input records=20
                Map output bytes=1154
                Map output records=20
                Reduce input groups=1
                Reduce input records=20
                Reduce output records=20
                Reduce shuffle bytes=0
                Spilled Records=40