zoukankan      html  css  js  c++  java
  • 用Sqoop实现数据HDFS到mysql到Hive

    【Hive】用Sqoop实现数据HDFS到mysql到Hive
    大数据协作框架
    “大数据协作框架”其实是一个统称,主要是以下四个框架
    • 数据转换工具    Sqoop
    • 文件收集库框架    Flume
    • 任务调度框架    Oozie
    • 大数据WEB工具Hue
     
    Sqoop作用
    将关系数据库中的某张表数据抽取到Hadoop的HDFS文件系统中,底层运行的还是MapReduce。
    利用MapReduce加快数据传输速度。
    批处理方式进行数据传输。
    也可以将HDFS上的文件数据或者Hive表中的数据导出到关系型数据库当中的某张表中。
     
    HDFS →RDBMS
    1. sqoop export
    2. --connect jdbc:mysql://xxx:3306/xxx
    3. --username xxx
    4. --password xxx
    5. --table xxx
    6. --export-dir xxx
    RDBMS→Hive
    1. sqoop import
    2. --connect jdbc:mysql://xxx:3306/xxx
    3. --username xxx
    4. --password xxx
    5. --fields-terminated-by " "
    6. --table xxx
    7. --hive-import
    8. --hive-table xxx
    Hive→RDBMS
    1. sqoop export
    2. --connect jdbc:mysql://xxx:3306/xxx
    3. --username xxx
    4. --password xxx
    5. --table xxx
    6. --export-dir xxx
    7. --input-fields-terminated-by ' '
    RDBMS→HDFS
    1. sqoop import
    2. --connect jdbc:mysql://xxx:3306/xxx
    3. --username xxx
    4. --password xxx
    5. --table xxx
    6. --target-dir xxx
    规律:
    1. 从RDBMS导入到HDFS或者Hive中的都使用import;从HDFS或者Hive导出到RDBMS中的都使用export;以HDFS和Hive为参考,根据数据流向选择关键字。
    2. connect、username、password、table四个参数为每一种传输都必须的;其中connect参数格式均为--connect jdbc:mysql://主机名:3306/数据库名(使用mysql数据库);table是指明mysql中的表名。
    3. export-dir参数只有在导出数据到RDBMS中时才会用到,含义为表在hdfs中存放的路径。
    区别:
    • HDFS →RDBMS:指明table:mysql中的表,需要自行先创建;指明export-dir:HDFS中数据的存储路径
    • RDBMS→Hive:指明fields-terminated-by:指定分隔符,分隔符指的是存放在Hive中的数据的分隔符,如果目标将存储在Hive中可理解为编码格式,若目标将存储在RDBMS上,则可理解为解码格式;指明table:mysql中的表名;指明hive-import:导入到hive操作;指明hive-table:hive中的表名。注意:table参数不可以与用户家目录下已存在的目录重名,因为sqoop导数据到hive会先将数据导入到HDFS上,然后再将数据load到hive中,最后把这个目录再删除掉。
    • Hive→RDBMS:指明table:mysql中的表名;指明export-dir:hive在hdfs中存储的路径;指明hive-table:hive中的表名。
    • RDBMS→HDFS:指明table:mysql里的表名;指明target-dir:hdfs存储数据的目录;
     
    Sqoop安装
    配置Sqoop1.x
    conf目录【sqoop-env-template.sh】
    • export HADOOP_COMMON_HOME=Hadoop目录
    • export HADOOP_MAPRED_HOME=Hadoop目录
    • export HIVE_HOME=Hive目录
    • export ZOOCFGDIR=Zookeeper目录
    将mysqlJDBC驱动包拷到sqoop的lib目录下
    测试sqoop
    1. bin/sqoop list-databases
    2. --connect jdbc:mysql://主机名:3306
    3. --username root
    4. --password 123456
    查看本地mysql
    1. mysql> show databases;
    2. +--------------------+
    3. | Database |
    4. +--------------------+
    5. | information_schema |
    6. | metastore |
    7. | mysql |
    8. | test |
    9. +--------------------+
    10. 4 rows in set (0.00 sec)
    11. mysql> use test;
    12. Reading table information for completion of table and column names
    13. You can turn off this feature to get a quicker startup with -A
    14. Database changed
    15. mysql> show tables;
    16. +----------------+
    17. | Tables_in_test |
    18. +----------------+
    19. | my_user |
    20. +----------------+
    21. 1 row in set (0.00 sec)
    22. mysql> desc my_user;
    23. +---------+--------------+------+-----+---------+----------------+
    24. | Field | Type | Null | Key | Default | Extra |
    25. +---------+--------------+------+-----+---------+----------------+
    26. | id | tinyint(4) | NO | PRI | NULL | auto_increment |
    27. | account | varchar(255) | YES | | NULL | |
    28. | passwd | varchar(255) | YES | | NULL | |
    29. +---------+--------------+------+-----+---------+----------------+
    30. 3 rows in set (0.00 sec)
    31. mysql> select * from my_user;
    32. +----+----------+----------+
    33. | id | account | passwd |
    34. +----+----------+----------+
    35. | 1 | admin | admin |
    36. | 2 | johnny | 123456 |
    37. | 3 | zhangsan | zhangsan |
    38. | 4 | lisi | lisi |
    39. | 5 | test | test |
    40. | 6 | qiqi | qiqi |
    41. | 7 | hangzhou | hangzhou |
    42. +----+----------+----------+
    43. 7 rows in set (0.00 sec)
    hive创建相同结构的空表
    1. hive (test)> create table h_user(
    2. > id int,
    3. > account string,
    4. > passwd string
    5. > )row format delimited fields terminated by ' ';
    6. OK
    7. Time taken: 0.113 seconds
    8. hive (test)> desc h_user;
    9. OK
    10. col_name data_type comment
    11. id int
    12. account string
    13. passwd string
    14. Time taken: 0.228 seconds, Fetched: 3 row(s)
    从本地mysql导出数据到Hive里
    1. bin/sqoop import
    2. --connect jdbc:mysql://cdaisuke:3306/test
    3. --username root
    4. --password 123456
    5. --table my_user
    6. --num-mappers 1
    7. --delete-target-dir
    8. --fields-terminated-by " "
    9. --hive-database test
    10. --hive-import
    11. --hive-table h_user
    12. hive (test)> select * from h_user;
    13. OK
    14. h_user.id h_user.account h_user.passwd
    15. 1 admin admin
    16. 2 johnny 123456
    17. 3 zhangsan zhangsan
    18. 4 lisi lisi
    19. 5 test test
    20. 6 qiqi qiqi
    21. 7 hangzhou hangzhou
    22. Time taken: 0.061 seconds, Fetched: 7 row(s)
    从mysql导入到HDFS里
    1. bin/sqoop import
    2. --connect jdbc:mysql://cdaisuke:3306/test
    3. --username root
    4. --password 123456
    5. --table my_user
    6. --num-mappers 3
    7. --target-dir /user/hadoop/
    8. --delete-target-dir
    9. --fields-terminated-by " "
    10. ------------------------------------------------------------
    11. [hadoop@cdaisuke sqoop-1.4.5-cdh5.3.6]$ bin/sqoop import
    12. > --connect jdbc:mysql://cdaisuke:3306/test
    13. > --username root
    14. > --password 123456
    15. > --table my_user
    16. > --num-mappers 3
    17. > --target-dir /user/hadoop/
    18. > --delete-target-dir
    19. > --fields-terminated-by " "
    20. 18/08/14 00:02:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.6
    21. 18/08/14 00:02:11 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
    22. 18/08/14 00:02:12 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
    23. 18/08/14 00:02:12 INFO tool.CodeGenTool: Beginning code generation
    24. 18/08/14 00:02:13 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `my_user` AS t LIMIT 1
    25. 18/08/14 00:02:13 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `my_user` AS t LIMIT 1
    26. 18/08/14 00:02:13 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/modules/hadoop-2.5.0-cdh5.3.6_Hive
    27. Note: /tmp/sqoop-hadoop/compile/7c8bdb7cd3df7b2f4b48700704f46f65/my_user.java uses or overrides a deprecated API.
    28. Note: Recompile with -Xlint:deprecation for details.
    29. 18/08/14 00:02:18 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/7c8bdb7cd3df7b2f4b48700704f46f65/my_user.jar
    30. 18/08/14 00:02:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    31. 18/08/14 00:02:22 INFO tool.ImportTool: Destination directory /user/hadoop is not present, hence not deleting.
    32. 18/08/14 00:02:22 WARN manager.MySQLManager: It looks like you are importing from mysql.
    33. 18/08/14 00:02:22 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
    34. 18/08/14 00:02:22 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
    35. 18/08/14 00:02:22 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
    36. 18/08/14 00:02:22 INFO mapreduce.ImportJobBase: Beginning import of my_user
    37. 18/08/14 00:02:22 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
    38. 18/08/14 00:02:22 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    39. 18/08/14 00:02:23 INFO client.RMProxy: Connecting to ResourceManager at slave01/192.168.79.140:8032
    40. 18/08/14 00:02:28 INFO db.DBInputFormat: Using read commited transaction isolation
    41. 18/08/14 00:02:28 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `my_user`
    42. 18/08/14 00:02:28 INFO mapreduce.JobSubmitter: number of splits:3
    43. 18/08/14 00:02:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533652222364_0078
    44. 18/08/14 00:02:29 INFO impl.YarnClientImpl: Submitted application application_1533652222364_0078
    45. 18/08/14 00:02:29 INFO mapreduce.Job: The url to track the job: http://slave01:8088/proxy/application_1533652222364_0078/
    46. 18/08/14 00:02:29 INFO mapreduce.Job: Running job: job_1533652222364_0078
    47. 18/08/14 00:02:50 INFO mapreduce.Job: Job job_1533652222364_0078 running in uber mode : false
    48. 18/08/14 00:02:50 INFO mapreduce.Job: map 0% reduce 0%
    49. 18/08/14 00:03:00 INFO mapreduce.Job: map 33% reduce 0%
    50. 18/08/14 00:03:01 INFO mapreduce.Job: map 67% reduce 0%
    51. 18/08/14 00:03:02 INFO mapreduce.Job: map 100% reduce 0%
    52. 18/08/14 00:03:02 INFO mapreduce.Job: Job job_1533652222364_0078 completed successfully
    53. 18/08/14 00:03:02 INFO mapreduce.Job: Counters: 30
    54. File System Counters
    55. FILE: Number of bytes read=0
    56. FILE: Number of bytes written=394707
    57. FILE: Number of read operations=0
    58. FILE: Number of large read operations=0
    59. FILE: Number of write operations=0
    60. HDFS: Number of bytes read=295
    61. HDFS: Number of bytes written=106
    62. HDFS: Number of read operations=12
    63. HDFS: Number of large read operations=0
    64. HDFS: Number of write operations=6
    65. Job Counters
    66. Launched map tasks=3
    67. Other local map tasks=3
    68. Total time spent by all maps in occupied slots (ms)=25213
    69. Total time spent by all reduces in occupied slots (ms)=0
    70. Total time spent by all map tasks (ms)=25213
    71. Total vcore-seconds taken by all map tasks=25213
    72. Total megabyte-seconds taken by all map tasks=25818112
    73. Map-Reduce Framework
    74. Map input records=7
    75. Map output records=7
    76. Input split bytes=295
    77. Spilled Records=0
    78. Failed Shuffles=0
    79. Merged Map outputs=0
    80. GC time elapsed (ms)=352
    81. CPU time spent (ms)=3600
    82. Physical memory (bytes) snapshot=316162048
    83. Virtual memory (bytes) snapshot=2523156480
    84. Total committed heap usage (bytes)=77766656
    85. File Input Format Counters
    86. Bytes Read=0
    87. File Output Format Counters
    88. Bytes Written=106
    89. 18/08/14 00:03:02 INFO mapreduce.ImportJobBase: Transferred 106 bytes in 40.004 seconds (2.6497 bytes/sec)
    90. 18/08/14 00:03:02 INFO mapreduce.ImportJobBase: Retrieved 7 records.
    设置3个map任务
    --num-mappers 3
    设置HDFS目标存储目录
    --target-dir /user/hadoop/
    如果设置目录存在则删除此目录
    --delete-target-dir
    从Hive导出到mysql
    在mysql创建新表
    1. create table user_export(
    2. id tinyint(4) not null auto_increment,
    3. account varchar(255) default null,
    4. passwd varchar(255) default null,
    5. primary key(id)
    6. );
    用sqoop导出数据
    1. bin/sqoop export
    2. --connect jdbc:mysql://cdaisuke:3306/test
    3. --username root
    4. --password 123456
    5. --table user_export
    6. --num-mappers 1
    7. --fields-terminated-by " "
    8. --export-dir /user/hive/warehouse/test.db/h_user
    9. ----------------------------------------------------
    10. [hadoop@cdaisuke sqoop-1.4.5-cdh5.3.6]$ bin/sqoop export
    11. > --connect jdbc:mysql://cdaisuke:3306/test
    12. > --username root
    13. > --password 123456
    14. > --table user_export
    15. > --num-mappers 1
    16. > --fields-terminated-by " "
    17. > --export-dir /user/hive/warehouse/test.db/h_user
    18. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
    19. 18/08/14 00:16:32 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.6
    20. 18/08/14 00:16:32 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
    21. 18/08/14 00:16:33 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
    22. 18/08/14 00:16:33 INFO tool.CodeGenTool: Beginning code generation
    23. 18/08/14 00:16:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_export` AS t LIMIT 1
    24. 18/08/14 00:16:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_export` AS t LIMIT 1
    25. 18/08/14 00:16:34 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/modules/hadoop-2.5.0-cdh5.3.6_Hive
    26. Note: /tmp/sqoop-hadoop/compile/6823ffae505b34f7ae8b9881bae4b898/user_export.java uses or overrides a deprecated API.
    27. Note: Recompile with -Xlint:deprecation for details.
    28. 18/08/14 00:16:39 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/6823ffae505b34f7ae8b9881bae4b898/user_export.jar
    29. 18/08/14 00:16:39 INFO mapreduce.ExportJobBase: Beginning export of user_export
    30. 18/08/14 00:16:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    31. 18/08/14 00:16:40 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
    32. 18/08/14 00:16:43 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    33. 18/08/14 00:16:43 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
    34. 18/08/14 00:16:43 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    35. 18/08/14 00:16:43 INFO client.RMProxy: Connecting to ResourceManager at slave01/192.168.79.140:8032
    36. 18/08/14 00:16:48 INFO input.FileInputFormat: Total input paths to process : 1
    37. 18/08/14 00:16:48 INFO input.FileInputFormat: Total input paths to process : 1
    38. 18/08/14 00:16:48 INFO mapreduce.JobSubmitter: number of splits:1
    39. 18/08/14 00:16:48 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
    40. 18/08/14 00:16:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533652222364_0079
    41. 18/08/14 00:16:50 INFO impl.YarnClientImpl: Submitted application application_1533652222364_0079
    42. 18/08/14 00:16:50 INFO mapreduce.Job: The url to track the job: http://slave01:8088/proxy/application_1533652222364_0079/
    43. 18/08/14 00:16:50 INFO mapreduce.Job: Running job: job_1533652222364_0079
    44. 18/08/14 00:17:11 INFO mapreduce.Job: Job job_1533652222364_0079 running in uber mode : false
    45. 18/08/14 00:17:11 INFO mapreduce.Job: map 0% reduce 0%
    46. 18/08/14 00:17:27 INFO mapreduce.Job: map 100% reduce 0%
    47. 18/08/14 00:17:27 INFO mapreduce.Job: Job job_1533652222364_0079 completed successfully
    48. 18/08/14 00:17:27 INFO mapreduce.Job: Counters: 30
    49. File System Counters
    50. FILE: Number of bytes read=0
    51. FILE: Number of bytes written=131287
    52. FILE: Number of read operations=0
    53. FILE: Number of large read operations=0
    54. FILE: Number of write operations=0
    55. HDFS: Number of bytes read=258
    56. HDFS: Number of bytes written=0
    57. HDFS: Number of read operations=4
    58. HDFS: Number of large read operations=0
    59. HDFS: Number of write operations=0
    60. Job Counters
    61. Launched map tasks=1
    62. Data-local map tasks=1
    63. Total time spent by all maps in occupied slots (ms)=13426
    64. Total time spent by all reduces in occupied slots (ms)=0
    65. Total time spent by all map tasks (ms)=13426
    66. Total vcore-seconds taken by all map tasks=13426
    67. Total megabyte-seconds taken by all map tasks=13748224
    68. Map-Reduce Framework
    69. Map input records=7
    70. Map output records=7
    71. Input split bytes=149
    72. Spilled Records=0
    73. Failed Shuffles=0
    74. Merged Map outputs=0
    75. GC time elapsed (ms)=73
    76. CPU time spent (ms)=1230
    77. Physical memory (bytes) snapshot=113061888
    78. Virtual memory (bytes) snapshot=838946816
    79. Total committed heap usage (bytes)=45613056
    80. File Input Format Counters
    81. Bytes Read=0
    82. File Output Format Counters
    83. Bytes Written=0
    84. 18/08/14 00:17:27 INFO mapreduce.ExportJobBase: Transferred 258 bytes in 44.2695 seconds (5.8279 bytes/sec)
    85. 18/08/14 00:17:27 INFO mapreduce.ExportJobBase: Exported 7 records.
    86. -----------------------------------------------------------------
    87. mysql> select * from user_export;
    88. +----+----------+----------+
    89. | id | account | passwd |
    90. +----+----------+----------+
    91. | 1 | admin | admin |
    92. | 2 | johnny | 123456 |
    93. | 3 | zhangsan | zhangsan |
    94. | 4 | lisi | lisi |
    95. | 5 | test | test |
    96. | 6 | qiqi | qiqi |
    97. | 7 | hangzhou | hangzhou |
    98. +----+----------+----------+
    99. 7 rows in set (0.00 sec)
    从HDFS导出到mysql
    在mysql创建新表
     
    1. create table my_user2(
    2. id tinyint(4) not null auto_increment,
    3. account varchar(255) default null,
    4. passwd varchar(255) default null,
    5. primary key (id)
    6. );
    7. ---------------------------------------------------------
    8. [hadoop@cdaisuke sqoop-1.4.5-cdh5.3.6]$ bin/sqoop export
    9. > --connect jdbc:mysql://cdaisuke:3306/test
    10. > --username root
    11. > --password 123456
    12. > --table my_user2
    13. > --num-mappers 1
    14. > --fields-terminated-by " "
    15. > --export-dir /user/hadoop
    16. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
    17. 18/08/14 00:39:51 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.6
    18. 18/08/14 00:39:51 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
    19. 18/08/14 00:39:52 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
    20. 18/08/14 00:39:52 INFO tool.CodeGenTool: Beginning code generation
    21. 18/08/14 00:39:53 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `my_user2` AS t LIMIT 1
    22. 18/08/14 00:39:53 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `my_user2` AS t LIMIT 1
    23. 18/08/14 00:39:53 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/modules/hadoop-2.5.0-cdh5.3.6_Hive
    24. Note: /tmp/sqoop-hadoop/compile/7222f42cd6507a21fdcef7600bd14a20/my_user2.java uses or overrides a deprecated API.
    25. Note: Recompile with -Xlint:deprecation for details.
    26. 18/08/14 00:39:59 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/7222f42cd6507a21fdcef7600bd14a20/my_user2.jar
    27. 18/08/14 00:39:59 INFO mapreduce.ExportJobBase: Beginning export of my_user2
    28. 18/08/14 00:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    29. 18/08/14 00:40:00 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
    30. 18/08/14 00:40:04 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
    31. 18/08/14 00:40:04 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
    32. 18/08/14 00:40:04 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    33. 18/08/14 00:40:04 INFO client.RMProxy: Connecting to ResourceManager at slave01/192.168.79.140:8032
    34. 18/08/14 00:40:09 INFO input.FileInputFormat: Total input paths to process : 3
    35. 18/08/14 00:40:09 INFO input.FileInputFormat: Total input paths to process : 3
    36. 18/08/14 00:40:09 INFO mapreduce.JobSubmitter: number of splits:1
    37. 18/08/14 00:40:09 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
    38. 18/08/14 00:40:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533652222364_0084
    39. 18/08/14 00:40:11 INFO impl.YarnClientImpl: Submitted application application_1533652222364_0084
    40. 18/08/14 00:40:11 INFO mapreduce.Job: The url to track the job: http://slave01:8088/proxy/application_1533652222364_0084/
    41. 18/08/14 00:40:11 INFO mapreduce.Job: Running job: job_1533652222364_0084
    42. 18/08/14 00:40:30 INFO mapreduce.Job: Job job_1533652222364_0084 running in uber mode : false
    43. 18/08/14 00:40:30 INFO mapreduce.Job: map 0% reduce 0%
    44. 18/08/14 00:40:46 INFO mapreduce.Job: map 100% reduce 0%
    45. 18/08/14 00:40:46 INFO mapreduce.Job: Job job_1533652222364_0084 completed successfully
    46. 18/08/14 00:40:46 INFO mapreduce.Job: Counters: 30
    47. File System Counters
    48. FILE: Number of bytes read=0
    49. FILE: Number of bytes written=131229
    50. FILE: Number of read operations=0
    51. FILE: Number of large read operations=0
    52. FILE: Number of write operations=0
    53. HDFS: Number of bytes read=365
    54. HDFS: Number of bytes written=0
    55. HDFS: Number of read operations=10
    56. HDFS: Number of large read operations=0
    57. HDFS: Number of write operations=0
    58. Job Counters
    59. Launched map tasks=1
    60. Data-local map tasks=1
    61. Total time spent by all maps in occupied slots (ms)=13670
    62. Total time spent by all reduces in occupied slots (ms)=0
    63. Total time spent by all map tasks (ms)=13670
    64. Total vcore-seconds taken by all map tasks=13670
    65. Total megabyte-seconds taken by all map tasks=13998080
    66. Map-Reduce Framework
    67. Map input records=7
    68. Map output records=7
    69. Input split bytes=250
    70. Spilled Records=0
    71. Failed Shuffles=0
    72. Merged Map outputs=0
    73. GC time elapsed (ms)=89
    74. CPU time spent (ms)=1670
    75. Physical memory (bytes) snapshot=115961856
    76. Virtual memory (bytes) snapshot=838946816
    77. Total committed heap usage (bytes)=45613056
    78. File Input Format Counters
    79. Bytes Read=0
    80. File Output Format Counters
    81. Bytes Written=0
    82. 18/08/14 00:40:46 INFO mapreduce.ExportJobBase: Transferred 365 bytes in 42.3534 seconds (8.618 bytes/sec)
    83. 18/08/14 00:40:46 INFO mapreduce.ExportJobBase: Exported 7 records.
    84. ------------------------------------------------------------------------
    85. mysql> select * from my_user2;
    86. +----+----------+----------+
    87. | id | account | passwd |
    88. +----+----------+----------+
    89. | 1 | admin | admin |
    90. | 2 | johnny | 123456 |
    91. | 3 | zhangsan | zhangsan |
    92. | 4 | lisi | lisi |
    93. | 5 | test | test |
    94. | 6 | qiqi | qiqi |
    95. | 7 | hangzhou | hangzhou |
    96. +----+----------+----------+
    97. 7 rows in set (0.00 sec)
     
  • 相关阅读:
    ES6核心内容精讲--快速实践ES6(三)
    ES6核心内容精讲--快速实践ES6(二)
    ES6核心内容精讲--快速实践ES6(一)
    webpack入门
    centos 7 nodejs 安装
    android结合Jenkins使用V2签名
    dockerfile 创建Jenkins镜像
    ubuntu docker 安装
    shell sed语法详解
    mac关于stf开源工具安装
  • 原文地址:https://www.cnblogs.com/buffercache/p/14238817.html
Copyright © 2011-2022 走看看