查看现有link
sqoop:000> show link
+-----------+------------------------+---------+
| Name | Connector Name | Enabled |
+-----------+------------------------+---------+
| mysqllink | generic-jdbc-connector | true |
| mysql2 | generic-jdbc-connector | true |
+-----------+------------------------+---------+
删除link
sqoop:000> delete link -name mysqllink
sqoop:000> show link
+--------+------------------------+---------+
| Name | Connector Name | Enabled |
+--------+------------------------+---------+
| mysql2 | generic-jdbc-connector | true |
+--------+------------------------+---------+
sqoop:000> delete link -name mysql2
sqoop:000> show link
+------+----------------+---------+
| Name | Connector Name | Enabled |
+------+----------------+---------+
+------+----------------+---------+
创建mysql link
sqoop:000> show connector
+------------------------+---------+------------------------------------------------------------+----------------------+
| Name | Version | Class | Supported Directions |
+------------------------+---------+------------------------------------------------------------+----------------------+
| generic-jdbc-connector | 1.99.7 | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO |
| kite-connector | 1.99.7 | org.apache.sqoop.connector.kite.KiteConnector | FROM/TO |
| oracle-jdbc-connector | 1.99.7 | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO |
| ftp-connector | 1.99.7 | org.apache.sqoop.connector.ftp.FtpConnector | TO |
| hdfs-connector | 1.99.7 | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO |
| kafka-connector | 1.99.7 | org.apache.sqoop.connector.kafka.KafkaConnector | TO |
| sftp-connector | 1.99.7 | org.apache.sqoop.connector.sftp.SftpConnector | TO |
+------------------------+---------+------------------------------------------------------------+----------------------+
sqoop:000> create link -c generic-jdbc-connector
Creating link for connector with name generic-jdbc-connector Please fill following values to create new link object
Name: mysqllink
Database connection
Driver class: com.mysql.jdbc.Driver
Connection String: jdbc:mysql://127.0.0.1:3306/test
Username: root
Password: ****
Fetch Size: 30
Connection Properties: There are currently 0 values in the map: entry#
SQL Dialect
Identifier enclose: 此处为空格
New link was successfully created with validation status OK and name mysqllink
查看新建的链接：
sqoop:000> show link
+-----------+------------------------+---------+
| Name | Connector Name | Enabled |
+-----------+------------------------+---------+
| mysqllink | generic-jdbc-connector | true |
+-----------+------------------------+---------+
创建hdfs链接
sqoop:000> create link -c hdfs-connector
Creating link for connector with name hdfs-connector Please fill following values to create new link object
Name: hdfslink
HDFS cluster
URI: localhost:9000
Conf directory: /root/hadoop/hadoop-2.7.4/etc/hadoop
Additional configs:: There are currently 0 values in the map: entry#
New link was successfully created with validation status OK and name hdfslink
sqoop:000> show link
+-----------+------------------------+---------+
| Name | Connector Name | Enabled |
+-----------+------------------------+---------+
| mysqllink | generic-jdbc-connector | true |
| hdfslink | hdfs-connector | true |
--------------------------------------------------------------------------------------
在创建mysql链接时，Identifier enclose：指定SQL中标识符的定界符，也就是说，有的SQL标示符是一个引号：select * from "table_name"，这种定界符在MySQL中是会报错的。这个属性默认值就是双引号，所以不能使用回车，必须将之覆盖，可以使用空格覆盖
hadoop fs -mkdir /sqoop
在sqoop客户端设置查看job详情：
set option --name verbose --value true
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
org.apache.hadoop.ipc.RemoteException: User: root is not allowed to impersonate root
如下处理：
在core-site.xml添加如下属性，其中
hadoop.proxyuser.root.groups中的root是用户，value里面的root是group
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>root</value>
<description>Allow the superuser oozie to impersonate any members of the group group1 and group2</description>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>localhost</value>
<description>The superuser can connect only from host1 and host2 to impersonate a user</description>
</property>
--------------------------------------------------------------------------
create job -f "mysql-link" -t "hdfs-link"
-f指定from，即是数据源位置，-t指定to，即是目的地位置。本例是从MySQL传递数据到HDFS，所以就是from mysql to HDFS。参数值就是在创建链接（link）时指定的Name。
以下是各个属性
Name：一个标示符，自己指定即可。
Schema Name：指定Database或Schema的名字，在MySQL中，Schema同Database类似，具体什么区别没有深究过，但官网描述在创建时差不多。这里指定数据库名字为db_ez即可，本例的数据库。
Table Name：本例使用的数据库表为tb_forhadoop，自己指定导出的表。多表的情况请自行查看官方文档。
SQL Statement：就是sql查询语句，文档上说需要指定一个$condition，但我一直没有创建成功，貌似是一个条件子句。
配置完以上几项，又回出现element#提示符，提示输入一些hash值，这里我没有再指定什么。直接回车过。而以下几个配置我也是直接回车，使用默认值，大概是一些与数据库相关的参数。
Partition column:
Partition column nullable:
Boundary query
Last value
后面需要配置数据目的地各项值：
Null alue：大概说的是如果有空值用什么覆盖
File format：指定在HDFS中的数据文件是什么文件格式，这里使用TEXT_FILE，即最简单的文本文件。
Compression codec：用于指定使用什么压缩算法进行导出数据文件压缩，我指定NONE，这个也可以使用自定义的压缩算法CUSTOM，用Java实现相应的接口。
Custom codec：这个就是指定的custom压缩算法，本例选择NONE，所以直接回车过去。
Output directory：指定存储在HDFS文件系统中的路径，这里最好指定一个存在的路径，或者存在但路劲下是空的，貌似这样才能成功。
Append mode：用于指定是否是在已存在导出文件的情况下将新数据追加到数据文件中。
Extractors：不清楚是什么，我取了一个1
Loaders：同上
最后再次出现element#提示符，用于输入extra mapper jars的属性，可以什么都不写。直接回车。
至此若出现successful则证明已经成功创建。
创建job mysqltohdfs，将mysql导入到hdfs:
sqoop:000> show job -n mysqltohdfs
2018-05-28 17:12:55,421 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-28 17:12:55,425 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:12:55,431 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
1 job(s) to show:
Job with name mysqltohdfs (Enabled: true, Created by root at 5/28/18 3:52 PM, Updated by root at 5/28/18 3:52 PM)
Throttling resources
Extractors:
Loaders:
Classpath configuration
Extra mapper jars:
From link: mysqllink
Database source
Schema name: test 数据库必填
Table name: role 表名必填
SQL statement:
Column names:
Partition column:
Partition column nullable:
Boundary query:
Incremental read
Check column:
Last value:
To link: hdfslink
Target configuration
Override null value:
Null value:
File format: TEXT_FILE 必选
Compression codec: NONE 必选
Custom codec:
Output directory: hdfs://localhost:9000/sqoop 数据输出目标路径
Append mode:
sqoop:000> start job -n mysqltohdfs
2018-05-28 17:07:02,252 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-28 17:07:02,256 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,259 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,262 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,263 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,264 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,268 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Found primary key columns [roleid]
2018-05-28 17:07:02,268 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Using partition column: roleid
2018-05-28 17:07:02,269 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Using min/max query: SELECT MIN( roleid ), MAX( roleid ) FROM test . role
2018-05-28 17:07:02,269 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Boundaries for the job: min=1, max=650, columnType=4
2018-05-28 17:07:02,269 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Using dataSql: SELECT * FROM test . role WHERE ${CONDITIONS}
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Field names: roleid , name , dateid , addr , sex , level
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file mapred-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file kms-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file core-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file hdfs-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file yarn-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file httpfs-site.xml
2018-05-28 17:07:02,318 [pool-1-thread-5] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - Running on unsecured cluster, skipping delegation token generation.
2018-05-28 17:07:02,318 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsToInitializer - Using working directory: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2
2018-05-28 17:07:02,354 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,904 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:02,905 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:02,905 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:02,905 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:02,906 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:02,909 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:02,960 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:02,960 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:02,960 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,008 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,009 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,014 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,014 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,014 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,014 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,015 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,018 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,020 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,021 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,021 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,103 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,104 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,107 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,107 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,107 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,107 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,108 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,116 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,122 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,122 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,123 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,178 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,179 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,182 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,183 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,184 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,184 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,184 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,186 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,187 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,187 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,188 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,273 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,274 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,276 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,277 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,277 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,277 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,277 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,279 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,280 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,280 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,281 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,340 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,340 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,347 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,347 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,347 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,348 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,348 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,351 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,352 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,352 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,352 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,423 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,424 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,428 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,429 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,429 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,429 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,439 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,442 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,444 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,444 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,446 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,535 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,535 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,538 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,538 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,539 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,540 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,541 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,565 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,566 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,566 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,567 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,618 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,618 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,621 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,621 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,621 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,621 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,622 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,634 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,635 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,635 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,636 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,685 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,686 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,688 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,689 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,689 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,689 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,689 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,691 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid <= ?
2018-05-28 17:07:03,731 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,731 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,731 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,771 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,772 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,775 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer
2018-05-28 17:07:03,776 [Thread-141] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer - Running generic JDBC connector destroyer
2018-05-28 17:07:03,777 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsToDestroyer
2018-05-28 17:07:03,795 [Thread-141] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,798 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt of size 53
2018-05-28 17:07:03,816 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt of size 120
2018-05-28 17:07:03,838 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/60f3cc4e-3e1e-4276-9e75-4ce652a82be6.txt of size 0
2018-05-28 17:07:03,849 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt of size 50
2018-05-28 17:07:03,862 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt of size 58
2018-05-28 17:07:03,874 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/881898e2-8545-437e-b4cb-4838ac489c87.txt of size 102
2018-05-28 17:07:03,895 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/a61f0265-0379-4a8b-8e70-f367fb2479a3.txt of size 0
2018-05-28 17:07:03,908 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/ad14cb0d-29e4-486e-b342-d6bec60d2463.txt of size 0
2018-05-28 17:07:03,919 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/be952dc7-ab88-4ab5-9ec1-0c2e8d064ccf.txt of size 0
2018-05-28 17:07:03,933 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/e3f5125d-872e-4312-b1dd-2154be3b74f0.txt of size 0
2018-05-28 17:07:03,960 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer
2018-05-28 17:07:03,960 [Thread-141] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer - Running generic JDBC connector destroyer
2018-05-28 17:07:03,961 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsToDestroyer
2018-05-28 17:07:03,979 [Thread-141] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
Exception in thread "Thread-141" java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse;
at org.apache.hadoop.mapred.JobEndNotifier.httpNotification(JobEndNotifier.java:76)
at org.apache.hadoop.mapred.JobEndNotifier.localRunnerNotification(JobEndNotifier.java:86)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:562)
2018-05-28 17:07:05,326 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:04 CST: FAILED
虽然有异常抛出，但数据成功写入hdfs,查看结果如下：
[root@host ~]# hdfs dfs -ls -R /sqoop
-rw-r--r-- 1 root supergroup 53 2018-05-28 17:07 /sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
-rw-r--r-- 1 root supergroup 120 2018-05-28 17:07 /sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
-rw-r--r-- 1 root supergroup 0 2018-05-28 17:07 /sqoop/60f3cc4e-3e1e-4276-9e75-4ce652a82be6.txt
-rw-r--r-- 1 root supergroup 50 2018-05-28 17:07 /sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
-rw-r--r-- 1 root supergroup 58 2018-05-28 17:07 /sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
-rw-r--r-- 1 root supergroup 102 2018-05-28 17:07 /sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
-rw-r--r-- 1 root supergroup 0 2018-05-28 17:07 /sqoop/a61f0265-0379-4a8b-8e70-f367fb2479a3.txt
-rw-r--r-- 1 root supergroup 0 2018-05-28 17:07 /sqoop/ad14cb0d-29e4-486e-b342-d6bec60d2463.txt
-rw-r--r-- 1 root supergroup 0 2018-05-28 17:07 /sqoop/be952dc7-ab88-4ab5-9ec1-0c2e8d064ccf.txt
-rw-r--r-- 1 root supergroup 0 2018-05-28 17:07 /sqoop/e3f5125d-872e-4312-b1dd-2154be3b74f0.txt
[root@host ~]# hdfs dfs -cat /sqoop/*
400,NULL,'2017-11-15 14:49:56.000','anhui hefei',0,4
1,NULL,'2017-11-16 14:49:11.000','henan luohe linying',1,10
40,NULL,'2017-11-13 14:50:25.000','guangdong shenzhen',1,20
110,NULL,'2017-11-14 14:50:47.000','beijing',1,20
200,NULL,'2017-11-14 14:49:47.000','shandong qingdao',0,8
600,NULL,'2017-11-15 14:50:05.000','hunan changsha',0,91
650,NULL,'2017-11-01 17:24:34.000',NULL,1,29
查看作业的状态
sqoop:000> status job -n mysqltohdfs
2018-05-29 10:55:44,766 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-29 10:55:44,768 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 10:55:44,771 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 10:55:44,789 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
Submission details
Job Name: mysqltohdfs
Server URL: http://localhost:12000/sqoop/
Created by: root
Creation date: 2018-05-28 17:07:02 CST
Lastly updated by: root
External ID: job_local1297704653_0002
http://localhost:8080/
2018-05-28 17:07:04 CST: FAILED
hdfs导入到mysql
创建空表role1：
mysql> create table role1 select * from role limit 0; Query OK, 0 rows affected (0.31 sec) Records: 0 Duplicates: 0 Warnings: 0
mysql> select * from role1 ;
Empty set (0.00 sec)
创建job
sqoop:000> create job -f hdfslink -t mysqllink
Creating job for links with from name hdfslink and to name mysqllink 2018-05-29 11:17:23,362 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:17:23,363 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:17:23,367 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:17:23,368 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit Please fill following values to create new job object Name: hdfs2018-05-29 11:17:27,053 [UpdateThread] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
Name: hdfstomysql
Input configuration
Input directory: hdfs://localhost:9000/sqoop
Override null value: Null value:
Incremental import
Incremental type: 0 : NONE 1 : NEW_FILES
Choose: 0
Last imported date:
Database target
Schema name: test
Table name: role1
Column names:
There are currently 0 values in the list: element#
Staging table:
Clear stage table:
Throttling resources
Extractors: Loaders:
Classpath configuration
Extra mapper jars: There are currently 0 values in the list: element# 2018-05-29 11:18:02,792 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request 2018-05-29 11:18:02,793 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:18:02,795 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:18:02,798 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit New job was successfully created with validation status OK and name hdfstomysql
查看job:
sqoop:000> show job
2018-05-29 11:18:06,188 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-29 11:18:06,193 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:06,196 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:06,198 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
+----+-------------+------------------------------------+------------------------------------+---------+
| Id | Name | From Connector | To Connector | Enabled |
+----+-------------+------------------------------------+------------------------------------+---------+
| 15 | mysqltohdfs | mysqllink (generic-jdbc-connector) | hdfslink (hdfs-connector) | true |
| 17 | hdfstomysql | hdfslink (hdfs-connector) | mysqllink (generic-jdbc-connector) | true |
+----+-------------+------------------------------------+------------------------------------+---------+
执行job
sqoop:000> start job -n hdfstomysql
2018-05-29 11:18:19,429 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-29 11:18:19,432 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,434 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,437 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,438 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,440 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,442 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file mapred-site.xml
2018-05-29 11:18:19,442 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file kms-site.xml
2018-05-29 11:18:19,442 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file core-site.xml
2018-05-29 11:18:19,443 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file hdfs-site.xml
2018-05-29 11:18:19,443 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file yarn-site.xml
2018-05-29 11:18:19,443 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file httpfs-site.xml
2018-05-29 11:18:19,484 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsFromInitializer - Input directory: hdfs://localhost:9000/sqoop
2018-05-29 11:18:19,489 [pool-1-thread-5] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - Running on unsecured cluster, skipping delegation token generation.
2018-05-29 11:18:19,509 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer - Using query to insert data: INSERT INTO test . role1 VALUES (?,?,?,?,?,?)
2018-05-29 11:18:19,514 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,666 [pool-1-thread-5] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:19,671 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
2018-05-29 11:18:19,671 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/60f3cc4e-3e1e-4276-9e75-4ce652a82be6.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/a61f0265-0379-4a8b-8e70-f367fb2479a3.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/ad14cb0d-29e4-486e-b342-d6bec60d2463.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/be952dc7-ab88-4ab5-9ec1-0c2e8d064ccf.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/e3f5125d-872e-4312-b1dd-2154be3b74f0.txt
2018-05-29 11:18:20,386 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,388 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,389 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,389 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,389 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt (offset=0, end=26, length=26), hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt (offset=26, end=53, length=27)}
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 0
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 26
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 26
2018-05-29 11:18:20,470 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,473 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,489 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 53
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 26
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 53
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 27
2018-05-29 11:18:20,503 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,511 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 53
2018-05-29 11:18:20,511 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 53
2018-05-29 11:18:20,512 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,512 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,512 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,565 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,565 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,589 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,590 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,590 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,590 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,590 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=0, end=39, length=39)}
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 0
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 39
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 39
2018-05-29 11:18:20,622 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,624 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,627 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,628 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,628 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,628 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,629 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,630 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,633 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,633 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,633 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,634 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,634 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=39, end=78, length=39)}
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 39
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 78
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 39
2018-05-29 11:18:20,667 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 60
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,672 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,672 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,673 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,675 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,675 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,675 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,675 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,675 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=78, end=99, length=21), hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=99, end=120, length=21)}
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 78
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 99
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 21
2018-05-29 11:18:20,700 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 120
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 99
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 120
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 21
2018-05-29 11:18:20,712 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,715 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 120
2018-05-29 11:18:20,715 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,716 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,716 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,716 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,717 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,717 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,721 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,721 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,721 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,721 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,721 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt (offset=0, end=25, length=25), hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt (offset=25, end=50, length=25)}
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 0
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 25
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 25
2018-05-29 11:18:20,750 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,751 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,753 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 50
2018-05-29 11:18:20,753 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
2018-05-29 11:18:20,753 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 25
2018-05-29 11:18:20,754 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 50
2018-05-29 11:18:20,754 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 25
2018-05-29 11:18:20,759 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,762 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 50
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 50
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,764 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,764 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,767 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,767 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,767 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,767 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,767 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt (offset=0, end=29, length=29), hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt (offset=29, end=58, length=29)}
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 0
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 29
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 29
2018-05-29 11:18:20,796 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,798 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,801 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 58
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 29
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 58
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 29
2018-05-29 11:18:20,808 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,811 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 58
2018-05-29 11:18:20,811 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 58
2018-05-29 11:18:20,811 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,812 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,812 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,813 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,813 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,815 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,816 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,816 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,816 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,817 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt (offset=0, end=39, length=39)}
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 0
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 39
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 39
2018-05-29 11:18:20,843 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,844 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 102
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,849 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,849 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,851 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,852 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,852 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,852 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,852 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt (offset=39, end=70, length=31), hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt (offset=70, end=102, length=32)}
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 39
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 70
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 31
2018-05-29 11:18:20,882 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,889 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 57
2018-05-29 11:18:20,889 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 102
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - from offset 70
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - to offset 102
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - of length 32
2018-05-29 11:18:20,898 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 102
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 102
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,904 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,905 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,914 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsFromDestroyer
2018-05-29 11:18:20,915 [Thread-255] INFO org.apache.sqoop.connector.hdfs.HdfsFromDestroyer - Running HDFS connector destroyer
2018-05-29 11:18:20,915 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer
2018-05-29 11:18:20,915 [Thread-255] INFO org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer - Running generic JDBC connector destroyer
2018-05-29 11:18:20,926 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsFromDestroyer
2018-05-29 11:18:20,926 [Thread-255] INFO org.apache.sqoop.connector.hdfs.HdfsFromDestroyer - Running HDFS connector destroyer
2018-05-29 11:18:20,927 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer
2018-05-29 11:18:20,927 [Thread-255] INFO org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer - Running generic JDBC connector destroyer
Exception in thread "Thread-255" java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse;
at org.apache.hadoop.mapred.JobEndNotifier.httpNotification(JobEndNotifier.java:76)
at org.apache.hadoop.mapred.JobEndNotifier.localRunnerNotification(JobEndNotifier.java:86)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:562)
2018-05-29 11:18:22,031 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:21 CST: FAILED
查看mysql表中数据
mysql> select * from test.role1;
+--------+------+---------------------+---------------------+------+-------+
| roleid | name | dateid | addr | sex | level |
+--------+------+---------------------+---------------------+------+-------+
| 400 | NULL | 2017-11-15 14:49:56 | anhui hefei | 0 | 4 |
| 1 | NULL | 2017-11-16 14:49:11 | henan luohe linying | 1 | 10 |
| 40 | NULL | 2017-11-13 14:50:25 | guangdong shenzhen | 1 | 20 |
| 110 | NULL | 2017-11-14 14:50:47 | beijing | 1 | 20 |
| 200 | NULL | 2017-11-14 14:49:47 | shandong qingdao | 0 | 8 |
| 600 | NULL | 2017-11-15 14:50:05 | hunan changsha | 0 | 91 |
| 650 | NULL | 2017-11-01 17:24:34 | NULL | 1 | 29 |
+--------+------+---------------------+---------------------+------+-------+
7 rows in set (0.00 sec)