有问题。。。。。。。。
数据迁移工具sqoop
sqoop安装
[root@sqoop data]# wget wget http://apache.fayea.com/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
[root@sqoop data]# tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
拷贝hadoop 目录到 sqoop服务器/data目录上
[root@sqoop data]# scp -r namenode:/data/hadoop /data/
修改sqoop 配置文件，指定hadoop 目录
[root@sqoop conf]# cp /data/sqoop/conf/sqoop-env-template.sh /data/sqoop/conf/sqoop-env.sh
[root@sqoop conf]# vim /data/sqoop/config/sqoop-env.sh
export HADOOP_COMMON_HOME=/data/hadoop
export HADOOP_MAPRED_HOME=/data/hadoop/share/hadoop/mapreduce
关闭hbase hive zk 等报警，通过sqoop 可以把 关系型数据库存储到 hbase中
[root@sqoop bin]# vim /data/sqoop/bin/configure-sqoop
## Moved to be a runtime check in sqoop. #if [ ! -d "${HBASE_HOME}" ]; then # echo "Warning: $HBASE_HOME does not exist! HBase imports will fail." # echo 'Please set $HBASE_HOME to the root of your HBase installation.' #fi ## Moved to be a runtime check in sqoop. #if [ ! -d "${HCAT_HOME}" ]; then # echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail." # echo 'Please set $HCAT_HOME to the root of your HCatalog installation.' #fi #if [ ! -d "${ACCUMULO_HOME}" ]; then # echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail." # echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.' fi #if [ ! -d "${ZOOKEEPER_HOME}" ]; then # echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail." # echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.' #fi
###也可以设置环境变量
vim /etc/profile HADOOP_HOME=/data/hadoop/ PATH=$HADOOP_HOME/bin:$PATH:$HOME/bin HADOOP_MAPRED_HOME=/data/hadoop/share/hadoop/mapreduce PATH=$HADOOP_MAPRED_HOME/bin:$PATH:$HOME/bin
sqool 使用命名
[root@sqoop bin]# ./sqoop help 16/12/12 14:51:49 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 usage: sqoop COMMAND [ARGS] Available commands: codegen Generate code to interact with database records create-hive-table Import a table definition into Hive eval Evaluate a SQL statement and display the results export Export an HDFS directory to a database table help List available commands import Import a table from a database to HDFS import-all-tables Import tables from a database to HDFS import-mainframe Import datasets from a mainframe server to HDFS job Work with saved jobs list-databases List available databases on a server list-tables List available tables in a database merge Merge results of incremental imports metastore Run a standalone Sqoop metastore version Display version information See 'sqoop help COMMAND' for information on a specific command.
官方文档：http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html
把192.168.20.60 的mysql 的senyintmain 导入到 hdfs中
wget http://central.maven.org/maven2/mysql/mysql-connector-java/6.0.5/mysql-connector-java-6.0.5.jar
cp mysql-connector-java-6.0.5.jar /data/sqoop/lib
[hadoop@sqoop ~]$ /data/sqoop/bin/sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --table sys_log
hdfs 权限是 hadoop. supergroup,所以需要把 /data hadoop sqoop 修改权限
[root@sqoop data]# chown -R hadoop.hadoop hadoop/ sqoop
[root@sqoop data]# su - hadoop
[hadoop@sqoop data]# [hadoop@sqoop ~]$ /data/sqoop/bin/sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --table sys_log
database 没有数据库权限,在mysql 192.168.20.60服务器上添加权限
[root@localhost:mysql>grant all on *.* to 'fengjian'@'192.168.20.%' IDENTIFIED BY '123456';
在此执行
[hadoop@sqoop data]# [hadoop@sqoop ~]$ /data/sqoop/bin/sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username feng --password 123456 --table sys_log --target-dir /sqoop/td2 -m 2 --fields-terminated-by ' ' --columns 'id,account,income' --where 'id>2 and id<=9'
显示192.168.20.60 中的数据库
[hadoop@sqoop ~]$ [hadoop@sqoop ~]$ /data/sqoop/bin/sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --table sys_log
连接mysql 需要使用mysql连接的jar包
--table trade_detail 导入到hdfs的mysql 表
--target-dir 导入到hdfs的目录名称
-m 2 使用2个map
--fields-terminated-by ' ' 设置分隔符，默认使用逗号分隔，
--columns "id,account,income" 设置导入的列
--where 'id>2 and id<=9' 加上查询条件
如果有条件 where id > 5 and $CONDITIONS'
./sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --query 'select * from trade_detail where id > 5 and $CONDITIONS' -m 1 --target-dir /sqoop/td3
如果没有条件 where $CONDITIONS'
./sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --query 'select * from trade_detail where $CONDITIONS' -m 1 --target-dir /sqoop/td3
--query 指定sql语句 后面必须加 $CONDITIONS
-m 1 必须使用1个map
如果指定map为多个，必须指定使用--split-by
./sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --query 'select * from trade_detail where id > 5 and $CONDITIONS' -m 2 --target-dir /sqoop/td4 --split-by trade_detail.id
用trade_detail.id 分隔，1个map将所有的数据都读取进来，不用切分数据。
建议使用单引号，如果使用双引号 "$CONDITIONS"
从hdfs 导入到mysql中
在msql表中创建一个 与 hdfs 相同的表结构
create table feng like templatesql;
./sqoop import --connect jdbc:mysql://192.168.20.60:3306/fengjian --username fengjian --password 123456 --export-dir '/sqoop/td3/' --table feng -m 1 --fields-terminated-by ' '