大数据相关常用命令行或操作

bin/zkServer.sh start

bin/zkServer.sh stop

启动Hadoop

1 hadoop102

sbin/start-dfs.sh

2 hadoop103

sbin/start-yarn.sh

1 hadoop103

sbin/stop-yarn.sh

2 hadoop102

sbin/stop-dfs.sh

启动Hbase

bin/hbase-daemon.sh start master

bin/hbase-daemon.sh start regionserver

bin/hbase-daemon.sh stop master

bin/hbase-daemon.sh stop regionserver

bin/start-hbase.sh

bin/stop-hbase.sh

软连接

ln -s /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml /opt/module/hbase131/conf/core-site.xml

ln -s /opt/module/hadoop-3.1.3/etc/hadoop/hdfs-site.xml /opt/module/hbase131/conf/hdfs-site.xml

 

Web端查看HDFS的NameNode

浏览器中输入：

http://hadoop102:9870

查看HDFS上存储的数据信息

Web端查看YARN的ResourceManager

浏览器中输入：

http://hadoop103:8088

查看YARN上运行的Job信息

查看JobHistory

历史服务器地址

http://hadoop102:19888/jobhistory

myhadoop.sh start

myhadoop.sh stop

jpsall

Group name: ddl

  Commands: alter, alter_async, alter_status, create, describe, disable, disable_all, drop, drop_all, enable, enable_all, exists, get_table, is_disabled, is_enabled, list, locate_region, show_filters

  Group name: dml

  Commands: append, count, delete, deleteall, get, get_counter, get_splits, incr, put, scan, truncate, truncate_preserve

  Group name: namespace

  Commands: alter_namespace, create_namespace, describe_namespace, drop_namespace, list_namespace, list_namespace_tables

2021-09-28T00:20:50.734899Z 1 [Note] A temporary password is generated for root@localhost: BMIzeYgW*6ef

初始化元数据

schematool -initSchema -dbType mysql –verbose

 

1、 数据导入，要求将CSV格式或者EXCEL格式的文件导入到HIVE数据仓库中；

2、数据汇总：在HIVE中执行SQL语言按照要求进行数据汇总；

3、数据可视化展示：将汇总结果导出到MySQL；

sqoop路径：/opt/module/sqoop

把指定文件放到hadoop指定路径：hadoop fs -put stu1.txt /user/hive/warehouse/stu

hive启动(/opt/module/hive)：bin/hive

测试流程：

①hive路径下建表：test1

create table test1

(InvoiceNo String, StockCode String, Description String, Quantity String, InvoiceDate String, UnitPrice String, CustomerID String, Country String)

ROW format delimited fields terminated by ',' STORED AS TEXTFILE;

②导入数据：

load data local inpath '/opt/module/data/test.csv' into table test1;

select * from test1;

③进入mysql：mysql -uroot -p000000

（创建数据库命令：create database company;）

（进入对应数据库命令：use company;）

④将汇总结果导出到MySQL：

1.建表（可视化建表）：

2.sqoop路径下：

bin/sqoop export

> --connect jdbc:mysql://master:3306/mysql

> --username root

> --password 000000

> --table test1

> --num-mappers 1

> --export-dir /user/hive/warehouse/test1

> --input-fields-terminated-by ","