awk数组
1.1 awk命令用法
awk 参数 ‘条件{动作}’
1.2 awk执行过程
1.3 条件
1， 正则表达式
2， 比较表达式 NR>2
3， 范围
4， 特殊条件 BEGIN END
1.4 awk实践
1.4.1 awk简单实践
[root@jz files]# cat red.log 环境准备
Zhang Dandan 41117397 :250:100:175
Zhang Xiaoyu 390320151 :155:90:201
Meng Feixue 80042789 :250:60:50
Wu Waiwai 70271111 :250:80:75
Liu Bingbing 41117483 :250:100:175
Wang Xiaoai 3515064655 :50:95:135
Zi Gege 1986787350 :250:168:200
Li Youjiu 918391635 :175:75:300
Lao Nanhai 918391635 :250:100:175
1. 显示Xiaoyu的姓氏和id号码
[root@jz files]# awk '$2~/Xiaoyu/{print $1,$3}' red.log
2．显示所有以41开头的ID号码的人的全名和ID号
[root@jz files]# awk '$3~/^41/{print $1,$2,$3}' red.log
3.显示所有ID号码最后一位数字是1或5的人的全名
[root@jz files]# awk '$3~/[15]$/{print $1,$2}' red.log
4.显示Xiaoyu的捐款.每个值时都有以$开头.如$520$200$135
[root@jz files]# awk '$2~/Xiaoyu/{gsub(/:/,"$",$NF);print}' red.log
5.显示第一行到第五行
[root@jz files]# awk 'NR==1,NR==5' red.log
6显示3开头的行开始到5开头的行结束
[root@jz files]# awk '/3/,/5/' num.txt
7．显示从{开始到}结束
[root@jz files]# awk '/{/,/}/' num.txt
8，把从{开始到}结束内容里的oldboy替换为lidao
[root@jz files]# awk '/{/,/}/{gsub(/oldboy/,”lidao”,);print}' num.txt
9，BEGIN{} 在awk读取文件内容之前显示
[root@jz files]# awk 'BEGIN{print "this is start"} {print $0}' red.log
10，END{} 在awk读取文件内容之后执行
统计etc/services的空行数
[root@jz files]# awk '/^$/{i++}END{print i}' /etc/services 求次数
11，求总和
[root@jz files]# awk '{i=i+$1}END{print i}' a.txt
1.5 awk数组
[root@oldboy files]# sort -t "." -rnk4 ip.txt |uniq -c |sort -rn
4 10.0.0.13
2 10.0.0.15
2 10.0.0.12
1 10.0.0.88
1 10.0.0.33
1 10.0.0.24
1 10.0.0.228
1 10.0.0.17
1 10.0.0.121
1 10.0.0.111
1 10.0.0.11
#-t 指定分隔符
#-r 逆序
#-n 按照数字进行排序
#-k 第几列
统计access.log中每个ip出现的次数（第1列）
[root@oldboy files]# awk '{print $1}' access.log |sort |uniq -c |sort -rn|head
12049 58.220.223.62
10856 112.64.171.98
1982 114.83.184.139
1662 117.136.66.10
1318 115.29.245.13
961 223.104.5.197
957 116.216.0.60
939 180.111.48.14
871 223.104.5.202
869 223.104.4.139
[root@oldboy files]# awk '{h[$1]++}END{for( police in h ) print police,h[police]}' ip.log
10.0.0.17 1
10.0.0.228 1
10.0.0.111 1
10.0.0.121 1
10.0.0.11 1
10.0.0.12 2
10.0.0.88 1
10.0.0.13 4
10.0.0.24 1
10.0.0.33 1
10.0.0.15 2
[root@oldboy files]# #统计 分类出现次数
访问IP求和
[root@oldboy files]# awk '{h[$1]++}END{for( police in h ) print police,h[police]}' access.log |sort -rnk2 |head
58.220.223.62 12049
112.64.171.98 10856
114.83.184.139 1982
117.136.66.10 1662
115.29.245.13 1318
223.104.5.197 961
116.216.0.60 957
180.111.48.14 939
223.104.5.202 871
223.104.4.139 869
1.1 考试题：处理以下文件内容,将域名取出并根据域名进行计数排序处理(百度和sohu面试题)
[root@oldboy files]# cat oldboy.txt
http://www.etiantian.org/index.html
http://www.etiantian.org/1.html
http://post.etiantian.org/index.html
http://mp3.etiantian.org/index.html
http://www.etiantian.org/3.html
http://post.etiantian.org/2.html
[root@oldboy files]# awk -F"/+" '{print $2}' oldboy.txt |sort|uniq -c
1 mp3.etiantian.org
2 post.etiantian.org
3 www.etiantian.org
awk数组：
[root@oldboy files]# awk -F"/+" '{h[$2]++}'END'{for(p in h) print p,h[p]}' oldboy.txt
mp3.etiantian.org 1
post.etiantian.org 2
www.etiantian.org 3
1.2 secure-20161219 查看破解最多10个ip 显示出被破解次数最多10个用户
[root@oldboy files]# head secure-20161219
Dec 11 03:49:23 localhost sshd[27086]: Did not receive identification string from 123.31.34.190 port 55390
Dec 11 03:49:24 localhost sshd[27087]: Invalid user support from 123.31.34.190 port 55493
Dec 11 03:49:24 localhost sshd[27087]: input_userauth_request: invalid user support [preauth]
Dec 11 03:49:25 localhost sshd[27087]: error: Could not get shadow information for NOUSER
Dec 11 03:49:25 localhost sshd[27087]: Failed password for invalid user support from 123.31.34.190 port 55493 ssh2
查看破解最多10个ip
[root@oldboy files]# awk '/Failed password/{h[$(NF-3)]++}END{for(p in h) print p,h[p]}' secure-20161219|sort -nrk2|head
218.65.30.25 68652
218.65.30.53 34326
218.87.109.154 21201
112.85.42.103 18065
112.85.42.99 17164
218.87.109.151 17163
218.87.109.150 17163
218.65.30.61 17163
218.65.30.126 17163
218.65.30.124 17163
显示出被破解次数最多10个用户
[root@oldboy files]# awk '/Failed password/{h[$(NF-5)]++}END{for(p in h) print p,h[p]}' secure-20161219|sort -nrk2|head
root 364610
admin 725
user 245
oracle 119
support 104
guest 79
test 70
ubnt 47
pi 41
webadmin 36
access每个ip地址访问了网站次数
[root@oldboy files]# awk '{h[$1]++}END{for(p in h) print p,h[p]}' access.log|sort -nrk2|head
58.220.223.62 12049
112.64.171.98 10856
114.83.184.139 1982
117.136.66.10 1662
115.29.245.13 1318
223.104.5.197 961
116.216.0.60 957
180.111.48.14 939
223.104.5.202 871
223.104.4.139 869
1.3 计算两列求和
[root@oldboy files]# cat 1.txt
a 2
b 3
c 4
a 5
a 5
d 6
e 7
b 3
f 5
[root@oldboy files]# awk '{h[$1]=h[$1]+$2}END{for(p in h) print p,h[p]}' 1.txt
a 12
b 6
c 4
d 6
e 7
f 5
统计access.log中每个ip使用的流量总数 显示流量总数最多的前10
[root@oldboy files]# awk '{h[$1]=h[$1]+$10}END{for(p in h)print p,h[p]}' access.log |sort -nrk2|head
114.83.184.139 31362956
117.136.66.10 22431302
116.216.30.47 21466000
223.104.5.197 21464856
116.216.0.60 19145329
114.141.164.180 17219553
114.111.166.22 17121524
223.104.5.202 16911512
116.228.21.187 15969887
112.64.171.98 15255013