如何hive视图
1.mysql数据库
[centos@s201 ~]$ mysql -uroot -proot mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | azkaban | | big12 | | hive | | mysql | | performance_schema | +--------------------+ 6 rows in set (0.05 sec)
2.找hive库
show databases; +--------------------+ | Database | +--------------------+ | information_schema | | azkaban | | big12 | | hive | | mysql | | performance_schema | +--------------------+ mysql> use hive;
3.观表
show tables; +---------------------------+ | Tables_in_hive | +---------------------------+ | AUX_TABLE | | BUCKETING_COLS | | CDS | | COLUMNS_V2 | | COMPACTION_QUEUE | | COMPLETED_COMPACTIONS | | COMPLETED_TXN_COMPONENTS | | DATABASE_PARAMS | | DBS | | DB_PRIVS | | DELEGATION_TOKENS | | FUNCS | | FUNC_RU | | GLOBAL_PRIVS | | HIVE_LOCKS | | IDXS | | INDEX_PARAMS | | KEY_CONSTRAINTS | | MASTER_KEYS | | NEXT_COMPACTION_QUEUE_ID | | NEXT_LOCK_ID | | NEXT_TXN_ID | | NOTIFICATION_LOG | | NOTIFICATION_SEQUENCE | | NUCLEUS_TABLES | | PARTITIONS | | PARTITION_EVENTS | | PARTITION_KEYS | | PARTITION_KEY_VALS | | PARTITION_PARAMS | | PART_COL_PRIVS | | PART_COL_STATS | | PART_PRIVS | | ROLES | | ROLE_MAP | | SDS | | SD_PARAMS | | SEQUENCE_TABLE | | SERDES | | SERDE_PARAMS | | SKEWED_COL_NAMES | | SKEWED_COL_VALUE_LOC_MAP | | SKEWED_STRING_LIST | | SKEWED_STRING_LIST_VALUES | | SKEWED_VALUES | | SORT_COLS | | TABLE_PARAMS | | TAB_COL_STATS | | TBLS | | TBL_COL_PRIVS | | TBL_PRIVS | | TXNS | | TXN_COMPONENTS | | TYPES | | TYPE_FIELDS | | VERSION | | WRITE_SET | +---------------------------+
4.TBLS表结构
mysql> desc TBLS; +--------------------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------------+--------------+------+-----+---------+-------+ | TBL_ID | bigint(20) | NO | PRI | NULL | | | CREATE_TIME | int(11) | NO | | NULL | | | DB_ID | bigint(20) | YES | MUL | NULL | | | LAST_ACCESS_TIME | int(11) | NO | | NULL | | | OWNER | varchar(767) | YES | | NULL | | | RETENTION | int(11) | NO | | NULL | | | SD_ID | bigint(20) | YES | MUL | NULL | | | TBL_NAME | varchar(128) | YES | MUL | NULL | | | TBL_TYPE | varchar(128) | YES | | NULL | | | VIEW_EXPANDED_TEXT | mediumtext | YES | | NULL | | | VIEW_ORIGINAL_TEXT | mediumtext | YES | | NULL | | +--------------------+--------------+------+-----+---------+-------+
5.根据TBL_TYPE找到视图
mysql> select * from TBLS where tbl_type='VIRTUAL_VIEW'; +--------+-------------+-------+------------------+--------+-----------+-------+----------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | TBL_ID | CREATE_TIME | DB_ID | LAST_ACCESS_TIME | OWNER | RETENTION | SD_ID | TBL_NAME | TBL_TYPE | VIEW_EXPANDED_TEXT | VIEW_ORIGINAL_TEXT | +--------+-------------+-------+------------------+--------+-----------+-------+----------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | 187 | 1544269997 | 6 | 0 | centos | 0 | 222 | a | VIRTUAL_VIEW | select `a`.`id`, `a`.`tag`, count(*) as `count` from (select `temptags`.`id`, `xx`.`tag` from `big12`.`temptags` lateral view explode(`parsejson`(`temptags`.`json`)) `xx` as `tag`) `a` group by `a`.`id`, `a`.`tag` | select id, tag, count(*) as count from (select id, tag from temptags lateral view explode(parsejson(json)) xx as tag) a group by id, tag | | 193 | 1544336757 | 6 | 0 | centos | 0 | 229 | a1 | VIRTUAL_VIEW | select `logevent`.`deviceid`, `logevent`.`musicid`, sum(cast(`logevent`.`mark` as int)) as `sum` from `big12`.`logevent` where `logevent`.`musicid` is not null group by `logevent`.`deviceid`, `logevent`.`musicid` | select deviceid, musicid, sum(cast(mark as int)) as sum from logevent where musicId is not null group by deviceid, musicid | | 194 | 1544336831 | 6 | 0 | centos | 0 | 230 | a2 | VIRTUAL_VIEW | select `a1`.`deviceid` , `a1`.`musicid`, `a1`.`sum`, max(`a1`.`sum`)over(partition by `a1`.`deviceid`) as `sum2` from `big12`.`a1` | select deviceid , musicid, sum, max(sum)over(partition by deviceid) as sum2 from a1 | | 227 | 1550817816 | 6 | 0 | centos | 0 | 262 | zz1 | VIRTUAL_VIEW | select `duowan_parquet`.`id`, `duowan_parquet`.`name`, `duowan_parquet`.`pass`, `duowan_parquet`.`email`, `duowan_parquet`.`nickname` from `big12`.`duowan_parquet` where substring(`duowan_parquet`.`id`,1,1) in (1,2,3,4,5,6,8,9,0) | select * from duowan_parquet where substring(id,1,1) in (1,2,3,4,5,6,8,9,0) | +--------+-------------+-------+------------------+--------+-----------+-------+----------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
6.查看存储库信息的DBS表
mysql> desc DBS; +-----------------+---------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------------+---------------+------+-----+---------+-------+ | DB_ID | bigint(20) | NO | PRI | NULL | | | DESC | varchar(4000) | YES | | NULL | | | DB_LOCATION_URI | varchar(4000) | NO | | NULL | | | NAME | varchar(128) | YES | UNI | NULL | | | OWNER_NAME | varchar(128) | YES | | NULL | | | OWNER_TYPE | varchar(10) | YES | | NULL | | +-----------------+---------------+------+-----+---------+-------+
select * from DBS limit 10; +-------+-----------------------+------------------------------------------------+-------------+------------+------------+ | DB_ID | DESC | DB_LOCATION_URI | NAME | OWNER_NAME | OWNER_TYPE | +-------+-----------------------+------------------------------------------------+-------------+------------+------------+ | 1 | Default Hive database | hdfs://s201/user/hive/warehouse | default | public | ROLE | | 6 | NULL | hdfs://s201/user/hive/warehouse/big12.db | big12 | centos | USER | | 11 | NULL | hdfs://s201/user/hive/warehouse/music164.db | music164 | centos | USER | | 16 | NULL | hdfs://s201/user/hive/warehouse/big12_umeng.db | big12_umeng | centos | USER | | 21 | NULL | hdfs://s201/user/hive/warehouse/big12_2.db | big12_2 | centos | USER | | 26 | NULL | hdfs://s201/user/hive/warehouse/iml.db | iml | centos | USER | | 31 | NULL | hdfs://s201/user/hive/warehouse/wqbin.db | wqbin | centos | USER | +-------+-----------------------+------------------------------------------------+-------------+------------+------------+
7.如何删除视图跑路的脚本如下
7.1连接mysql
import pymysql conn = pymysql.connect(host='192.168.154.201', user='root', passwd='root', db='hive') cur = conn.cursor() # 查询 sql = "select NAME,TBL_NAME from TBLS a join DBS b on a.DB_ID=b.DB_ID " reCount = cur.execute(sql) # 返回受影响的行数 print(reCount) data = cur.fetchall() # 返回数据,返回的是tuple类型 print(data) cur.close() conn.close()
(('big12', 'a'), ('big12', 'a1'), ('big12', 'a2'), ('big12', 'zz1'))
7.2删除hive视图
使用pyhive连接hive删除所有视图
import pymysql conn = pymysql.connect(host='192.168.154.201', user='root', passwd='root', db='hive') cur = conn.cursor() # 查询 sql = "select NAME,TBL_NAME from TBLS a join DBS b on a.DB_ID=b.DB_ID where a.TBL_TYPE='VIRTUAL_VIEW'" reCount = cur.execute(sql) # 返回受影响的行数 data = cur.fetchall() # 返回数据,返回的是tuple类型 print(data) cur.close() conn.close() from pyhive import hive import thrift import sasl import thrift_sasl conn = hive.Connection(host='192.168.154.201', port=10000, database='big12',auth='NOSASL') cursor=conn.cursor() for a,b in data: cursor.execute("drop view "+a+"."+b) conn.close()