[大数据]sqoop安装与运用

文由

Oracle数据导入到HIVE

安装Sqoop

• 前提条件/环境

hadoop 2.7.3

• step1 下载安装包(sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz)

http://sqoop.apache.org/
      http://www.apache.org/dyn/closer.lua/sqoop/1.4.7
            https://mirror.bit.edu.cn/apache/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

• step2 解压，并改目录名为sqoop-1.4.7

tar -xvf /opt/software/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/software/sqoop-1.4.7

mv /opt/software/sqoop-1.4.7.bin__hadoop-2.6.0 /opt/software/sqoop-1.4.7

• step3 配置环境变量:HADOOP_HOME/SQOOP_HOME

vim /etc/profile
      # export SQOOP_HOME=/opt/software/sqoop-1.4.7
      SQOOP_HOME=/opt/software/sqoop-1.4.7
      HADOOP_HOME=/usr/sdp/2.6.0.0-1245/hadoop-2.7.3
      PATH=$PATH:$SQOOP_HOME/bin:$HADOOP_HOME/bin
      export PATH SQOOP HADOOP_HOME

source /etc/profile

[补充：查看hadoop目录]

# whereis hadoop
hadoop: /usr/bin/hadoop /usr/lib/hadoop /etc/hadoop

# ll /usr/lib/hadoop
lrwxrwxrwx 1 root root 35 Dec 10 17:33 /usr/lib/hadoop -> /usr/sdp/2.6.0.0-1245/hadoop-2.7.3/

[补充:如果不配置HADOOP_HOME，则:]

# sqoop help
Error: /opt/software/sqoop-1.4.7/bin/../../hadoop-mapreduce does not exist!
Please set $HADOOP_MAPRED_HOME to the root of your Hadoop MapReduce installation.

• step4 测试是否安装/启动成功

# sqoop help
Warning: /opt/software/sqoop-1.4.7/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/software/sqoop-1.4.7/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/software/sqoop-1.4.7/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/sdp/2.6.0.0-1245/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/sdp/2.6.0.0-1245/tez-0.9.0/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/sdp/2.6.0.0-1245/hadoop-2.7.3/lib/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/01/18 21:32:31 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.

• step5 配置MYSQL数据库驱动:mysql-connector-java-5.1.41-bin.jar
  将mysql-connector-java-5.1.41-bin.jar放入/opt/software/sqoop-1.4.7/lib目录下

MySQL Product Archives -

• step6 进入/opt/software/sqoop-1.4.7/bin运行
  查看是否能连接上数据库，并列出数据库

./sqoop list-databases --connect jdbc:mysql://153.0.163.195:3306/ --username root –P

迁移数据

Oracle导入Hive

[格式]

sqoop import 
--connect jdbc:oracle:thin:eas/eas@192.168.100.199:1521:eas   
--table TMP_NEEDCLEAN_PROVIDER2 
--hive-import --hive-database xxx 
--hive-table RPT_PROVIDER_NEEDCLEAN2   
--hive-overwrite 
--delete-target-dir 
--hive-drop-import-delims 
--null-string '' --null-non-string ''
 -m1 

[示例]

sqoop import 
--connect jdbc:oracle:thin:@10.0.3.3:1521:sid 
--username abc 
--password r123  
--table CMX.SALES  
-m 1 
--hive-import 
--hive-overwrite 
--hive-database ra 
--hive-table "cmx_sales" 
--null-non-string '' 
--null-string '' 
--delete-target-dir 
--hive-drop-import-delims

MySQL导入Hive

sqoop import 
--connect jdbc:mysql://localhost:3306/sqoop 
--username root 
--password root 
--table student 
--null-string '@@@' 
--null-non-string '###'

–null-string的含义是 string类型的字段，当Value是NULL，替换成指定的字符，该例子中为” <-数据库那种空值
–null-non-string的含义是非string类型的字段，当Value是NULL，替换成指定字符，该例子中为”<-数据库那种空值

参考文献

• sqoop安装及使用 - 博客园
• sqoop安装教程 - 简书
• mysql-connector-java-5.-bin.jar 下载方法 - 博客园
• 安装Sqoop的时候出现Error: /root/training/sqoop-1.4.5.bin__hadoop-0.23/../hadoop-mapreduce does not exist! - CSDN
• oracle到hive数据类型转换 - CSDN