文本处理三剑客之sed

---

文本处理三剑客之sed知识概述

三剑客sed命令概述介绍
三剑客sed命令执行原理命令基本语法
三剑客sed命令实操练习增删改查

一sed命令概述

sed - stream editor for filtering and transforming text

1）可以查找过滤筛选出指定的信息
2）擅长编辑文件内容信息
3）擅长对文件行进行操作

命令语法结构：sed     [参数]      ' 条件   指令'     文件信息

定义条件：根据行号   根据内容    根据正则信息

定义指令：满足条件的信息处理动作（添加删除修改输出）

二sed命令执行原理

sed命令执行过程核心原则：找什么信息，做什么事情

找什么信息==定义查找条件

做什么事情==操作执行命令（替换删除增加输出）

三sed命令的查询操作

实际上是输入操作

（一）根据行号筛选单行信息

[root@centos71 ~]# sed  -n   '3p'   /etc/selinux/config
# SELINUX= can take one of these three values:

（二）根据行号筛选多行信息

连续的多行信息

[root@centos71 ~]# sed  -n   '2,4p'   /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.

不连续的多行信息

[root@centos71 ~]# cat  -n /etc/selinux/config
     1    
     2    # This file controls the state of SELinux on the system.
     3    # SELINUX= can take one of these three values:
     4    #     enforcing - SELinux security policy is enforced.
     5    #     permissive - SELinux prints warnings instead of enforcing.
     6    #     disabled - No SELinux policy is loaded.
     7    SELINUX=enforcing
     8    # SELINUXTYPE= can take one of three values:
     9    #     targeted - Targeted processes are protected,
    10    #     minimum - Modification of targeted policy. Only selected processes are protected. 
    11    #     mls - Multi Level Security protection.
    12    SELINUXTYPE=targeted
    13    
    14    
    15    
[root@centos71 ~]#  sed  -n   '1p;3p;5p'  /etc/selinux/config

# SELINUX= can take one of these three values:
#     permissive - SELinux prints warnings instead of enforcing.

（三）根据字符筛选单行信息   

/ /这种结构和搜索替换是类似的，搜索替换包含了搜索

搜索替换sed   's///g'

[root@centos71 test]# sed   -n  '/dis/p'   selinux.txt 
#     disabled - No SELinux policy is loaded.
SELINUX=disabled

（四）根据字符筛选多行信息

连续的多行信息

筛选从dis开头，target结尾的行

[root@centos71 test]# cat  selinux.txt

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

[root@centos71 test]# sed   -n  '/dis/,/target/p'   selinux.txt 
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
[root@centos71 test]# 

不连续的多行信息

[root@centos71 ~]# sed  -n  '/dis/p;/en/p'  /etc/selinux/config  
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled

区间范围查找，搜索的起始字符一定要的，结束字符不一定要

有头无尾可以（壁虎），有尾无头不可以，直接死了

是贪婪匹配，每行进行过滤

四sed命令删除操作

（一）根据行号进行单行删除

[root@centos71 test]# cat  -n  selinux.txt
     1    
     2    # This file controls the state of SELinux on the system.
     3    # SELINUX= can take one of these three values:
     4    #     enforcing - SELinux security policy is enforced.
     5    #     permissive - SELinux prints warnings instead of enforcing.
     6    #     disabled - No SELinux policy is loaded.
     7    SELINUX=disabled
     8    # SELINUXTYPE= can take one of three values:
     9    #     targeted - Targeted processes are protected,
    10    #     minimum - Modification of targeted policy. Only selected processes are protected. 
    11    #     mls - Multi Level Security protection.
    12    SELINUXTYPE=targeted
    13    
    14    
    15    
[root@centos71 test]# sed   '6d'  selinux.txt

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

（二）根据行号进行多行删除

连续多行

注意操作是在内存的模式空间操作，对磁盘的block操作才真正操作

[root@centos71 test]# cat  -n  selinux.txt
     1    
     2    # This file controls the state of SELinux on the system.
     3    # SELINUX= can take one of these three values:
     4    #     enforcing - SELinux security policy is enforced.
     5    #     permissive - SELinux prints warnings instead of enforcing.
     6    #     disabled - No SELinux policy is loaded.
     7    SELINUX=disabled
     8    # SELINUXTYPE= can take one of three values:
     9    #     targeted - Targeted processes are protected,
    10    #     minimum - Modification of targeted policy. Only selected processes are protected. 
    11    #     mls - Multi Level Security protection.
    12    SELINUXTYPE=targeted
    13    
    14    
    15    
[root@centos71 test]# sed   '1,11d'  selinux.txt
SELINUXTYPE=targeted

不连续多行

[root@centos71 test]# cat  -n  selinux.txt
     1    
     2    # This file controls the state of SELinux on the system.
     3    # SELINUX= can take one of these three values:
     4    #     enforcing - SELinux security policy is enforced.
     5    #     permissive - SELinux prints warnings instead of enforcing.
     6    #     disabled - No SELinux policy is loaded.
     7    SELINUX=disabled
     8    # SELINUXTYPE= can take one of three values:
     9    #     targeted - Targeted processes are protected,
    10    #     minimum - Modification of targeted policy. Only selected processes are protected. 
    11    #     mls - Multi Level Security protection.
    12    SELINUXTYPE=targeted
    13    
    14    
    15    
[root@centos71 test]# sed   '7d;12d'  selinux.txt

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.

五sed命令添加操作

（一）根据行号单行添加信息

巧记：a表示add+after

[root@centos71 test]# cat  -n  selinux.txt
     1    
     2    # This file controls the state of SELinux on the system.
     3    # SELINUX= can take one of these three values:
     4    #     enforcing - SELinux security policy is enforced.
     5    #     permissive - SELinux prints warnings instead of enforcing.
     6    #     disabled - No SELinux policy is loaded.
     7    SELINUX=disabled
     8    # SELINUXTYPE= can take one of three values:
     9    #     targeted - Targeted processes are protected,
    10    #     minimum - Modification of targeted policy. Only selected processes are protected. 
    11    #     mls - Multi Level Security protection.
    12    SELINUXTYPE=targeted
    13    
    14    
    15    
[root@centos71 test]# sed   '1a  hahahah'   selinux.txt 

hahahah
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

（二）根据行号多行添加信息

连续多行

[root@centos71 ~]# cat  -n  /etc/issue  
     1    S
     2    Kernel 
 on an m
     3    
[root@centos71 ~]# sed  '1,2aHAPPY'  /etc/issue
S
HAPPY
Kernel 
 on an m
HAPPY

不连续多行

要加-e

[root@centos71 ~]# cat -n  /etc/issue
     1    S
     2    Kernel 
 on an m
     3    
[root@centos71 ~]# sed  -e  '1aHAPPY'   -e  '2aHAPPY'    -e  '3aHAPPY'  /etc/issue
S
HAPPY
Kernel 
 on an m
HAPPY

HAPPY

在指定行后面添加多行信息

[root@centos71 test]# sed   '1a  hahaha
xixixi'   selinux.txt 

hahaha
xixixi
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

在指定行前面插入信息

i插入，联系vim的i就是插入

[root@centos71 test]# sed   '1i  hahaha
xixixi'   selinux.txt 
hahaha
xixixi

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

企业常见配置文件添加信息方法

在文件的最后一行添加内容

$表示最后一行；在最后一行后面附加新的信息

[root@centos71 ~]# cat -n  /etc/issue
     1    S
     2    Kernel 
 on an m
     3    
[root@centos71 ~]# sed   '$ahahaxixi'  /etc/issue
S
Kernel 
 on an m

hahaxixi

基础命令是最低级别的，不能识别正则符号 ，会认为是字符

grep只能识别基础正则

sed只能识别基础正则

awk统计分析命令，是高级命令，可以识别基础正则和扩展正则

这种情况类似于高级语言和机器语言

六sed命令修改操作

（一）根据单行信息搜索替换

[root@centos71 test]# sed  '7s#disabled#enforcing#g'  selinux.txt

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

（二）利用正则将信息匹配出来再做替换

[root@centos71 ~]# cat  -n  /etc/selinux/config
     1    
     2    # This file controls the state of SELinux on the system.
     3    # SELINUX= can take one of these three values:
     4    #     enforcing - SELinux security policy is enforced.
     5    #     permissive - SELinux prints warnings instead of enforcing.
     6    #     disabled - No SELinux policy is loaded.
     7    SELINUX=disabled
     8    # SELINUXTYPE= can take one of three values:
     9    #     targeted - Targeted processes are protected,
    10    #     minimum - Modification of targeted policy. Only selected processes are protected. 
    11    #     mls - Multi Level Security protection.
    12    SELINUXTYPE=targeted
    13    
    14    
    15    
[root@centos71 ~]# sed '2,6s/#/abc/g'   /etc/selinux/config

abc This file controls the state of SELinux on the system.
abc SELINUX= can take one of these three values:
abc     enforcing - SELinux security policy is enforced.
abc     permissive - SELinux prints warnings instead of enforcing.
abc     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

（三）利用后向引用前项做替换——修改网卡地址

[root@centos71 ~]# sed -r '/IPADDR/s#(.*=).*#1192.168.111.200#g' /etc/sysconfig/network-scripts/ifcfg-eth0 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eth0
UUID=9d9e2656-f3ac-4f75-8722-3136d239985d
DEVICE=eth0
ONBOOT=yes
IPADDR=192.168.111.200
PREFIX=24
GATEWAY=10.0.0.254
IPV6_PRIVACY=no
DNS1=223.5.5.5

练习——sed批量创建用户信息test01 .. test10并设置随机密码(6位)

 

 

（一）创建用户

[root@centos71 ~]# seq  -w  10
01
02
03
04
05
06
07
08
09
10

[root@centos71 test]# seq -w   10  |  sed  -r   's#(.*)# useradd  test1#g'
 useradd  test01
 useradd  test02
 useradd  test03
 useradd  test04
 useradd  test05
 useradd  test06
 useradd  test07
 useradd  test08
 useradd  test09
 useradd  test10

（二）设置随机密码

[root@centos71 ~]# # seq -w 10|sed -r 's#(.*)#useradd test1;echo  password  | passwd --stdin test1#g'

（三）保存随机密码

 注意因为设置的是随机密码，所以要使用变量

[root@centos71 ~]# seq  -w 10 |  sed -r  's#(.*)#useradd test1;passwd=$(tr -cd 'a-zA-Z0-9' </dev/urandom|head -c 6);
echo $passwd | passwd --stdin test1; echo test1 $passwd >>/tmp/user_passwd#g'

练习——实现批量修改文件名称

将文件名称扩展名改为jpg

问题我的操作可以吗？

[root@centos71 test]# touch   happy{01..10}.txt
[root@centos71 test]# ls  happy*
happy01.txt  happy03.txt  happy05.txt  happy07.txt  happy09.txt  happy.txt
happy02.txt  happy04.txt  happy06.txt  happy08.txt  happy10.txt
[root@centos71 test]# ls  happy*  |  sed   -r  's#(.*.)txt#1jpg#g' 
happy01.jpg
happy02.jpg
happy03.jpg
happy04.jpg
happy05.jpg
happy06.jpg
happy07.jpg
happy08.jpg
happy09.jpg
happy10.jpg
happy.jpg

[root@centos71 test]# sed   -r  's#(.*.)txt#1jpg#g'  happy.txt
happy01.jpg
happy02.jpg
happy03.jpg
happy04.jpg
happy05.jpg
happy06.jpg
happy07.jpg
happy08.jpg
happy09.jpg
happy10.jpg

法二

[root@centos71 test]# ls  happy*
happy01.txt  happy03.txt  happy05.txt  happy07.txt  happy09.txt  happy.txt
happy02.txt  happy04.txt  happy06.txt  happy08.txt  happy10.txt
[root@centos71 test]# ls  happy* | sed   -r  's#(.*)txt#mv 1txt 1jpg#g'mv happy01.txt happy01.jpg
mv happy02.txt happy02.jpg
mv happy03.txt happy03.jpg
mv happy04.txt happy04.jpg
mv happy05.txt happy05.jpg
mv happy06.txt happy06.jpg
mv happy07.txt happy07.jpg
mv happy08.txt happy08.jpg
mv happy09.txt happy09.jpg
mv happy10.txt happy10.jpg
[root@centos71 test]# ls  happy* | sed   -r  's#(.*)txt#mv 1txt 1jpg#g'    | bash
[root@centos71 test]# ls  happy*
happy01.jpg  happy03.jpg  happy05.jpg  happy07.jpg  happy09.jpg  happy.txt
happy02.jpg  happy04.jpg  happy06.jpg  happy08.jpg  happy10.jpg

练习——利用sed命令取出IP地址

（一）取出有IP地址所在行

[root@centos71 ~]#   ip a s eth0|sed -n '3p'
    inet 10.0.0.200/24 brd 10.0.0.255 scope global noprefixroute eth0

（二）取出IP地址

将前面/后面减掉，简言之掐头去尾留中间

[root@centos71 ~]# ip a  s  eth0  |    sed   -n  '3p' |   sed   -r  's#.* (.*)/.*#1#g'
10.0.0.200

巧记：

双引号比单引号更多，作用更大,比如可以调用变量

七sed命令使用忠告

（一）sed操作时候要做备份

真正删除

 -i[SUFFIX], --in-place[=SUFFIX]
                 edit files in place (makes backup if SUFFIX supplied)

-i---将模式空间操作影响到磁盘中也就是将模式空间信息--覆盖--磁盘文件中

在编辑文件时，直接多文件进行备份sed   -i.bak

注意sed命令在使用时用到多个参数，要将-i参数写在所有参数后面

下面对文件进行搜索替换的同时也进行了备份

[root@centos71 test]# cat  -n /etc/selinux/config 
     1    
     2    # This file controls the state of SELinux on the system.
     3    # SELINUX= can take one of these three values:
     4    #     enforcing - SELinux security policy is enforced.
     5    #     permissive - SELinux prints warnings instead of enforcing.
     6    #     disabled - No SELinux policy is loaded.
     7    SELINUX=disabled
     8    # SELINUXTYPE= can take one of three values:
     9    #     targeted - Targeted processes are protected,
    10    #     minimum - Modification of targeted policy. Only selected processes are protected. 
    11    #     mls - Multi Level Security protection.
    12    SELINUXTYPE=targeted
    13    
    14    
    15
[root@centos71 test]#  sed  -ri.bak    '7s#disabled#enforcing#'   /etc/selinux/config
[root@centos71 test]# ls  /etc/selinux/config*
/etc/selinux/config  /etc/selinux/config.bak
[root@centos71 test]# ls  /etc/selinux/config* -l
-rw-r--r-- 1 root root 543 Dec 19 19:43 /etc/selinux/config
-rw-r--r-- 1 root root 542 Dec 17 16:59 /etc/selinux/config.bak
[root@centos71 test]# diff   /etc/selinux/config*
7c7
< SELINUX=enforcing
---
> SELINUX=disabled

（二）sed在使用参数时不要一起使用参数-i、-n

如果做了第1步，那么第2步做了还可以恢复，但是对企业还是会有损失

所以小心驶得万年船

八sed特别说明

注意单引号、双引号、三引号的作用是一样的，查看执行过程

[root@centos71 test]# cat   sed.test.sh
#/bin/bash
echo  single quotes 
sed    -n   '/127.0.0.1/p'   /etc/hosts
echo  tow quotes 
sed    -n   ''/127.0.0.1/p''   /etc/hosts
echo  three quotes 
sed    -n   '''/127.0.0.1/p'''   /etc/hosts

[root@centos71 test]# bash  -x  sed.test.sh
+ echo single quotes
single quotes
+ sed -n /127.0.0.1/p /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
+ echo tow quotes
tow quotes
+ sed -n /127.0.0.1/p /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
+ echo three quotes
three quotes
+ sed -n /127.0.0.1/p /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
[root@centos71 test]# cat  -n   /etc/hosts
     1    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
     2    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6