| Argument | Description |
|---|---|
--check-column (col) |
Specifies the column to be examined when determining which rows to import. (the column should not be of type CHAR/NCHAR/VARCHAR/VARNCHAR/ LONGVARCHAR/LONGNVARCHAR) |
--incremental (mode) |
Specifies how Sqoop determines which rows are new. Legal values for mode include append and lastmodified. |
--last-value (value) |
Specifies the maximum value of the check column from the previous import. |
Append模式增量导入
sqoop import --connect 'jdbc:mysql://localhost:3306/test'
--username root
--password root
--table role
-hive-database sqoopdb -hive-table role
--hive-import -m -1
--incremental append
--check-column dateid
--last-value "2017-12-01"
--check-column 可以是一列或者多列，用来检测数据是否按增量数据导入
--last-value 上次导入的检测字段的最大值
--incremental 增量的导入模式，有两种分别为Append和Lastmodified
测试没问题
Lastmodified增量导入
[root@host ~]# sqoop import --connect 'jdbc:mysql://localhost:3306/test' --username root --password root --table role -hive-database sqoopdb -hive-table role --hive-import -m -1 --incremental lastmodified --check-column dateid --last-value "2015-12-01"
Warning: /root/sqoop1/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /root/sqoop1/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /root/sqoop1/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/06/06 16:59:25 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
18/06/06 16:59:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/06/06 16:59:26 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
18/06/06 16:59:26 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
--incremental lastmodified option for hive imports is not supported. Please remove the parameter --incremental lastmodified.
发现导入hive不支持