zoukankan      html  css  js  c++  java
  • 【甘道夫】Sqoop1.99.3基础操作--导入Oracle的数据到HDFS

    第一步:进入clientShell
    fulong@FBI008:~$ sqoop.sh client
    Sqoop home directory: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200
    Sqoop Shell: Type 'help' or 'h' for help.

    sqoop:000> set server --host FBI003 --port 12000 --webapp sqoop
    Server is set successfully
    sqoop:000> show version --all
    client version:
      Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
      Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
    server version:
      Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
      Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
    Protocol version:
      [1]


    第二步:创建连接
    sqoop:000> create connection --cid 001
    Creating connection for connector with id 1
    Please fill following values to create new connection object
    Name: 147patents

    Connection configuration

    JDBC Driver Class: oracle.jdbc.driver.OracleDriver
    JDBC Connection String: jdbc:oracle:thin:@192.168.0.147:1521:ORCLGBK
    Username: zhaobiao
    Password: ******
    JDBC Connection Properties:
    There are currently 0 values in the map:
    entry#

    Security related configuration options

    Max connections: 100
    New connection was successfully created with validation status FINE and persistent id 1

    第三步:创建Job
    我这里顺便尝试了一下update命令。所以第一遍创建job时输入了错误的tablename:
    sqoop:000> create job
    Required argument --xid is missing.
    sqoop:000> create job --xid 1 --type import
    Creating job for connection with id 1
    Please fill following values to create new job object
    Name: ImportPatents

    Database configuration

    Schema name: zhaobiao
    Table name: zhaobiao
    Table SQL statement:
    Table column names:
    Partition column name:
    Nulls in partition column:
    Boundary query:

    Output configuration

    Storage type:
      0 : HDFS
    Choose: 0
    Output format:
      0 : TEXT_FILE
      1 : SEQUENCE_FILE
    Choose: 0
    Compression format:
      0 : NONE
      1 : DEFAULT
      2 : DEFLATE
      3 : GZIP
      4 : BZIP2
      5 : LZO
      6 : LZ4
      7 : SNAPPY
    Choose: 0
    Output directory: /Data/zhaobiao

    Throttling resources

    Extractors:
    Loaders:
    New job was successfully created with validation status FINE  and persistent id 1

    sqoop:000> update job
    Required argument --jid is missing.
    sqoop:000> update job --jid 1
    Updating job with id 1
    Please update job metadata:
    Name: ImportPatents

    Database configuration

    Schema name: zhaobiao
    Table name: CMS_NEWS_0625
    Table SQL statement:
    Table column names:
    Partition column name:
    Nulls in partition column:
    Boundary query:

    Output configuration

    Storage type:
      0 : HDFS
    Choose: 0
    Output format:
      0 : TEXT_FILE
      1 : SEQUENCE_FILE
    Choose: 0
    Compression format:
      0 : NONE
      1 : DEFAULT
      2 : DEFLATE
      3 : GZIP
      4 : BZIP2
      5 : LZO
      6 : LZ4
      7 : SNAPPY
    Choose: 0
    Output directory: /Data/zhaobiao

    Throttling resources

    Extractors:
    Loaders:
    Job was successfully updated with status FINE

    总结:
    1.create job时必须指定一个连接id
    2.create job完毕后。自己主动分配了一个job id
    3.update job时。须要指定job id
    4.创建Job过程中指定的Output directory指定的是HDFS中的路径

    第四步:运行job
    sqoop:000> start job --jid 1
    2014-06-27 12:44:17 CST: FAILURE_ON_SUBMIT
    Exception: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://fulonghadoop/Data/zhaobiao already exists
    sqoop:000> start job --jid 1
    Submission details
    Job ID: 1
    Server URL: http://FBI003:12000/sqoop/
    Created by: fulong
    Creation date: 2014-06-27 12:44:53 CST
    Lastly updated by: fulong
    External ID: job_1403248336137_0005
            http://FBI003:8088/proxy/application_1403248336137_0005/
    2014-06-27 12:44:53 CST: BOOTING  - Progress is not available
    sqoop:000> status job --jid 1
    Submission details
    Job ID: 1
    Server URL: http://FBI003:12000/sqoop/
    Created by: fulong
    Creation date: 2014-06-27 12:44:53 CST
    Lastly updated by: fulong
    External ID: job_1403248336137_0005
            http://FBI003:8088/proxy/application_1403248336137_0005/
    2014-06-27 12:50:11 CST: RUNNING  - 0.00 %

    sqoop:000> status job --jid 1
    Submission details
    Job ID: 1
    Server URL: http://FBI003:12000/sqoop/
    Created by: fulong
    Creation date: 2014-06-27 12:44:53 CST
    Lastly updated by: fulong
    External ID: job_1403248336137_0005
            http://FBI003:8088/proxy/application_1403248336137_0005/
    2014-06-27 13:36:49 CST: RUNNING  - 25.00 %
    sqoop:000> status job --jid 1
    Submission details
    Job ID: 1
    Server URL: http://FBI003:12000/sqoop/
    Created by: fulong
    Creation date: 2014-06-27 12:44:53 CST
    Lastly updated by: fulong
    External ID: job_1403248336137_0005
            http://FBI003:8088/proxy/application_1403248336137_0005/
    2014-06-27 13:39:24 CST: RUNNING  - 35.00 %
    sqoop:000> status job --jid 1
    Submission details
    Job ID: 1
    Server URL: http://FBI003:12000/sqoop/
    Created by: fulong
    Creation date: 2014-06-27 12:44:53 CST
    Lastly updated by: fulong
    External ID: job_1403248336137_0005
            http://FBI003:8088/proxy/application_1403248336137_0005/
    2014-06-27 13:47:19 CST: RUNNING  - 40.00 %
    sqoop:000> status job --jid 1
    Submission details
    Job ID: 1
    Server URL: http://FBI003:12000/sqoop/
    Created by: fulong
    Creation date: 2014-06-27 12:44:53 CST
    Lastly updated by: fulong
    External ID: job_1403248336137_0005
            http://FBI003:8088/proxy/application_1403248336137_0005/
    2014-06-27 14:08:36 CST: SUCCEEDED
    Counters:
            org.apache.hadoop.mapreduce.JobCounter
                    SLOTS_MILLIS_MAPS: 16499226
                    TOTAL_LAUNCHED_MAPS: 10
                    SLOTS_MILLIS_REDUCES: 0
                    OTHER_LOCAL_MAPS: 15
            org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter
                    BYTES_WRITTEN: 421226094
            org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
                    BYTES_READ: 0
            org.apache.hadoop.mapreduce.TaskCounter
                    MAP_INPUT_RECORDS: 0
                    MERGED_MAP_OUTPUTS: 0
                    PHYSICAL_MEMORY_BYTES: 1185689600
                    SPILLED_RECORDS: 0
                    COMMITTED_HEAP_BYTES: 627179520
                    CPU_MILLISECONDS: 307570
                    FAILED_SHUFFLE: 0
                    VIRTUAL_MEMORY_BYTES: 6542311424
                    SPLIT_RAW_BYTES: 1437
                    MAP_OUTPUT_RECORDS: 644083
                    GC_TIME_MILLIS: 5279
            org.apache.hadoop.mapreduce.FileSystemCounter
                    FILE_WRITE_OPS: 0
                    FILE_READ_OPS: 0
                    FILE_LARGE_READ_OPS: 0
                    FILE_BYTES_READ: 0
                    HDFS_BYTES_READ: 1437
                    FILE_BYTES_WRITTEN: 931780
                    HDFS_LARGE_READ_OPS: 0
                    HDFS_WRITE_OPS: 20
                    HDFS_READ_OPS: 40
                    HDFS_BYTES_WRITTEN: 431907904
            org.apache.sqoop.submission.counter.SqoopCounters
                    ROWS_READ: 644083
    Job executed successfully

    接下来的问题:怎样导入数据到Hbase中?


  • 相关阅读:
    多个文件目录下Makefile的写法
    通用多目录makefile的写法
    一个简单的通用Makefile实现
    Eclipse + CDT + YAGARTO + J-Link,STM32开源开发环境搭建与调试
    Eclipse-cdt 配合 gdbserver 进行 arm 程序远程调试 上
    Eclipse-cdt 配合 gdbserver 进行 arm 程序远程调试 下
    WiEngine+Eclipse+CDT+Sequoyah实现c++编程智能感知提示
    Docker 常用命令总结
    很好用的取代 PuTTY 的SSH远程登陆软件 Termius
    [转]如何优雅地进行参数校验?有什么提高代码稳壮性的方式?
  • 原文地址:https://www.cnblogs.com/brucemengbm/p/6934809.html
Copyright © 2011-2022 走看看