hive函数

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF

重点学习的是字符串的操作，所以建立一个空表就可以了，

create table dual(id string);

load 一个文件dual.dat 只有一个空格 

load data local inpath '/home/hadoop/study/study2/dual.dat' overwrite  into table  dual;

1: jdbc:hive2://localhost:10000> select 2*33 from dual;
+------+--+
| _c0  |
+------+--+
| 66   |
+------+--+

SHOW FUNCTIONS;
DESCRIBE FUNCTION <function_name>;
DESCRIBE FUNCTION EXTENDED <function_name> 显示函数的扩展信息

1: jdbc:hive2://localhost:10000> select 22=23 from dual;
+--------+--+
|  _c0   |
+--------+--+
| false  |
+--------+--+
1 row selected (0.717 seconds)
1: jdbc:hive2://localhost:10000> select 22!=23 from dual;
+-------+--+
|  _c0  |
+-------+--+
| true  |
+-------+--+

0: jdbc:hive2://localhost:10000> select 'aaac' rlike  'a'  from dual; 
+-------+--+
|  _c0  |
+-------+--+
| true  |
+-------+--+
1 row selected (1.138 seconds)
0: jdbc:hive2://localhost:10000> select 'aaac' rlike  'b'  from dual; 
+--------+--+
|  _c0   |
+--------+--+
| false  |
+--------+--+

0: jdbc:hive2://localhost:10000> select 'aaac' rlike null from dual;
+-------+--+
| _c0 |
+-------+--+
| NULL |
+-------+--+
1 row selected (0.276 seconds)
0: jdbc:hive2://localhost:10000> select null rlike 'ss' from dual;
+-------+--+
| _c0 |
+-------+--+
| NULL |
+-------+--+

如果A或B为NULL，则为NULL，如果A的任何（可能为空）子字符串与Java正则表达式B匹配，则为TRUE，否则为FALSE。例如，'foobar'RLIKE'foo'评估为TRUE，“foobar”RLIKE'^ f。* r $'也是如此。

0: jdbc:hive2://localhost:10000> select concat('hello',' kitty') from dual; 
+--------------+--+
|     _c0      |
+--------------+--+
| hello kitty  |
+--------------+--+

0: jdbc:hive2://localhost:10000> select round(2.3333) from dual;
+------+--+
| _c0 |
+------+--+
| 2.0 |
+------+--+

0: jdbc:hive2://localhost:10000> select negative(2.3333) from dual;
+----------+--+
| _c0 |
+----------+--+
| -2.3333 |
+----------+--+

0: jdbc:hive2://localhost:10000> select ascii('A')  from dual; 
+------+--+
| _c0  |
+------+--+
| 65   |
+------+--+

0: jdbc:hive2://localhost:10000> select find_in_set('ab','abc,s,ab,c,cf')from dual; 
+------+--+
| _c0  |
+------+--+
| 3    |
+------+--+
返回strList中str的首次出现，其中strList是以逗号分隔的字符串。如果任一参数为空，则返回null。如果第一个参数包含任何逗号，则返回0。例如，find_in_set（'ab'，'abc，b，ab，c，def'）返回3。

0: jdbc:hive2://localhost:10000> select cast(23.098 as int) from dual; 
+------+--+
| _c0  |
+------+--+
| 23   |
+------+--+

0: jdbc:hive2://localhost:10000> select if(2>1,'yes','no') from dual;
+------+--+
| _c0 |
+------+--+
| yes |
+------+--+

0: jdbc:hive2://localhost:10000> select get_json_object('{"id":"1","name":"oo"}','$.name');
+------+--+
| _c0 |
+------+--+
| oo |
+------+--+

注意第一个参数是一个json对象，后面的一个$.name ,取json对象中的值，第一个参数可以提前设置

例如

set hivevar:msg={ "message":"2015/12/08 09:14:4", "client": "10.108.24.253", "server": "passport.suning.com", "request": "POST /ids/needVerifyCode HTTP/1.1", "server": "passport.sing.co", "version":"1", "timestamp":"2015-12-08T01:14:43.273Z", "type":"B2C","center":"JSZC", "system":"WAF","clientip":"192.168.61.4", "host":"wafprdweb03", "path":"/usr/local/logs/waf.error.log", "redis":"192.168.24.46"}

select get_json_object(‘${hivevar:msg}’,’$.server’)

0: jdbc:hive2://localhost:10000> select parse_url('http://www.baidu.com/p/wew.jsp?id=2&name=wee','QUERY') from dual limit 1 ;
+----------------+--+
|      _c0       |
+----------------+--+
| id=2&name=wee  |
+----------------+--+
从URL返回指定的部分。 partToExtract的有效值包括HOST，PATH，QUERY，REF，PROTOCOL，AUTHORITY，FILE和USERINFO。
例如，parse_url（'http://facebook.com/path1/p.php?k1=v1
这个函数只能查询一个partToExtract的有效值，查询多个时返回null

0: jdbc:hive2://localhost:10000> select parse_url_tuple('http://www.baidu.com/p/wew.jsp?id=2&name=wee','HOST','QUERY') from dual limit 1 ;
+----------------+----------------+--+
| c0 | c1 |
+----------------+----------------+--+
| www.baidu.com | id=2&name=wee |
+----------------+----------------+--+
1 row selected (0.56 seconds)

可以查询多个partToExtract的有效值

0: jdbc:hive2://localhost:10000> select concat_ws('-','2017','05','06') from dual;
+-------------+--+
|     _c0     |
+-------------+--+
| 2017-05-06  |
+-------------+--+
与concat的区别在于这个方法可以指定连接符

0: jdbc:hive2://localhost:10000> select collect_set(name) from myexternaltable; 
返回的是经过去重的数组

.......(mapreduce)
+------------------------------------------------------------------------------------------------------------------+--+
|                                                       _c0                                                        |
+------------------------------------------------------------------------------------------------------------------+--+
| ["oo","pp","ll","i9i","kkj","ujn","aa","zx","sdfa","4sad","3d3","sadf","gdh","asdf4","asdfsadf","asdf","asddd"]  |

0: jdbc:hive2://localhost:10000> select collect_list(name) from myexternaltable; 列出字段的所有值，列出来不去重

窗口函数

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics

hive 处理json相关的函数split，可以结合自定义函数解析复杂的json，带有嵌套的

get_json_object也可以解析这种json

split(string str, string pat)
Splits str around pat (pat is a regular expression).

 create table t_rating as 
 select split(parseJson(line),'	')[0] as movieid,
 split(parseJson(line),'	')[1] as rate,
 split(parseJson(line),'	')[2] as timestring,
 split(parseJson(line),'	')[3] as uid from t_json  limit 10;

get_json_object
A limited version of JSONPath is supported:
$ : Root object
. : Child operator
[] : Subscript operator for array
* : Wildcard for []
Syntax not supported that's worth noticing:
: Zero length string as key
.. : Recursive descent
@ : Current object/element
() : Script expression
?() : Filter (script) expression.
[,] : Union operator
[start:end.step] : array slice operator
Example: src_json table is a single column (json), single row table:
+----+
                               json
+----+
{"store":
  {"fruit":[{"weight":8,"type":"apple"},{"weight":9,"type":"pear"}],
   "bicycle":{"price":19.95,"color":"red"}
  },
 "email":"amy@only_for_json_udf_test.net",
 "owner":"amy"
}
+----+
The fields of the json object can be extracted using these queries:
hive> SELECT get_json_object(src_json.json, '$.owner') FROM src_json;
amy
 
hive> SELECT get_json_object(src_json.json, '$.store.fruit[0]') FROM src_json;
{"weight":8,"type":"apple"}
 
hive> SELECT get_json_object(src_json.json, '$.non_exist_key') FROM src_json;
NULL
Built-in Aggregate Functions (UDAF)

transform:调用脚本来解析数据

https://cwiki.apache.org//confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-parse_url_tuple

LATERAL VIEW 行变列.

parse_url_tuple 解析URL

parse_url_tuple
The parse_url_tuple() UDTF is similar to parse_url(), but can extract multiple parts of a given URL, returning the data in a tuple. Values for a particular key in QUERY can be extracted by appending a colon and the key to the partToExtract argument, for example, parse_url_tuple('http://facebook.com/path1/p.php?k1=v1&k2=v2#Ref1', 'QUERY:k1', 'QUERY:k2') returns a tuple with values of 'v1','v2'. This is more efficient than calling parse_url() multiple times. All the input parameters and output column types are string.
SELECT b.*
FROM src LATERAL VIEW parse_url_tuple(fullurl, 'HOST', 'PATH', 'QUERY', 'QUERY:id') b as host, path, query, query_id LIMIT 1;