2. How to use cgroups?
The user can access and manage cgroups directly and indirectly (with LXC, libvirt or Docker).
Install the necessary packages:
$ sudo apt-get install libcgroup1 cgroup-tools
Now, the enabled cgroups can be seen via proc filesystem or sysfs:
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 9 2 1
cpu 4 134 1
cpuacct 4 134 1
blkio 7 134 1
memory 5 163 1
devices 11 134 1
freezer 2 2 1
net_cls 3 2 1
perf_event 10 2 1
net_prio 3 2 1
hugetlb 8 2 1
pids 6 136 1
$ ls -l /sys/fs/cgroup/
total 0
dr-xr-xr-x 6 root root 0 Nov 13 00:55 blkio
drwxr-xr-x 2 root root 60 Nov 13 01:00 cgmanager
lrwxrwxrwx 1 root root 11 Nov 13 00:55 cpu -> cpu,cpuacct
lrwxrwxrwx 1 root root 11 Nov 13 00:55 cpuacct -> cpu,cpuacct
dr-xr-xr-x 6 root root 0 Nov 13 00:55 cpu,cpuacct
dr-xr-xr-x 3 root root 0 Nov 13 00:55 cpuset
dr-xr-xr-x 6 root root 0 Nov 13 00:55 devices
dr-xr-xr-x 3 root root 0 Nov 13 00:55 freezer
dr-xr-xr-x 3 root root 0 Nov 13 00:55 hugetlb
dr-xr-xr-x 6 root root 0 Nov 13 00:55 memory
lrwxrwxrwx 1 root root 16 Nov 13 00:55 net_cls -> net_cls,net_prio
dr-xr-xr-x 3 root root 0 Nov 13 00:55 net_cls,net_prio
lrwxrwxrwx 1 root root 16 Nov 13 00:55 net_prio -> net_cls,net_prio
dr-xr-xr-x 3 root root 0 Nov 13 00:55 perf_event
dr-xr-xr-x 6 root root 0 Nov 13 00:55 pids
dr-xr-xr-x 6 root root 0 Nov 13 00:55 systemd
cgroups can be configured directly via the sysfs. For example, let’s create a small bash script named test_cgroups.sh for demonstration:
#!/bin/bash
while :
do
echo "Print line" > /dev/tty
sleep 5
done
Run above script:
$ chmod +x test_cgroups.sh
$ ./test_cgroups.sh
Print line
Print line
Print line
...
...
Change directory to /sys/fs/cgroup/devices where devices represents kind of resources that allows or denies access to devices by tasks in a cgroup:
$ cd sys/fs/cgroup/devices
Then, create a directory cgroups_test_group:
# mkdir cgroups_test_group
After creation of the cgroups_test_group directory, the following files will be generated:
$ ls -l /sys/fs/cgroup/devices/cgroups_test_group
total 0
-rw-r--r-- 1 root root 0 Nov 16 02:05 cgroup.clone_children
-rw-r--r-- 1 root root 0 Nov 16 02:05 cgroup.procs
--w------- 1 root root 0 Nov 16 02:05 devices.allow
--w------- 1 root root 0 Nov 16 02:05 devices.deny
-r--r--r-- 1 root root 0 Nov 16 02:05 devices.list
-rw-r--r-- 1 root root 0 Nov 16 02:05 notify_on_release
-rw-r--r-- 1 root root 0 Nov 16 02:05 tasks
The tasks file contains PIDs (Process ID) of processes which will be attached to the cgroups_test_group, the devices.deny file contains list of denied devices. By default, a newly created group has no any limits for devices access. In order to forbid a device (in this case, it’s /dev/tty), the devices.deny file should be modified:
# echo "c 5:0 w" > devices.deny
In the above command, the c indicates that /dev/tty is a character device, 5:0 is major and minor numbers of the device. The last w is write permission, so the above command forbids tasks to write to the /dev/tty.
$ ls -l /dev/tty
crw-rw-rw- 1 root tty 5, 0 Nov 18 17:02 /dev/tty
After that, re-run the script test_cgroups.sh:
$ ./test_cgroups.sh
Print line
Print line
Print line
...
...
then add the PID of this process to the tasks file:
# echo $(pidof -x test_cgroups.sh) > /sys/fs/cgroup/devices/cgroups_test_group/tasks
The result will be as expected:
$ ./test_cgroups.sh
Print line
Print line
Print line
./test_cgroups.sh: line 5: /dev/tty: Operation not permitted
./test_cgroups.sh: line 5: /dev/tty: Operation not permitted
...
...
An other example when running docker container
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
98225055fa39 ubuntu "/bin/bash" 47 seconds ago Up 30 seconds
$ cat /sys/fs/cgroup/device/docker/98225055fa394b388e988b067b77dda61e53027ee944e4e0fd7887e19cdcf341/tasks
13556
During starting up of a docker container, docker creates a cgroup for processes in this container:
$ docker run -it ubuntu
$ top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 18508 1848 1444 S 0.0 0.0 0:00.01 bash
12 root 20 0 36628 1924 1420 R 0.0 0.0 0:00.01 top
Now, the cgroup of above process will be seen on host machine:
$ systemd-cgls
Control group /:
-.slice
├─1429 /sbin/cgmanager -m name=systemd
├─docker
│ └─98225055fa394b388e988b067b77dda61e53027ee944e4e0fd7887e19cdcf341
│ └─13556 /bin/bash
type
type can have one of the following three values:
a — applies to all devices, both character devices and block devices
b — specifies a block device
c — specifies a character device
在/devices/cgroup 目录下创建目录 first,并设置禁止设备读：
root@ubuntu:/sys/fs/cgroup/devices# mkdir first
root@ubuntu:/sys/fs/cgroup/devices/first# echo "a 1:5 r" > devices.deny
在另外一个终端中设置：
root@ubuntu:~# cgexec -g devices:first dd if=/dev/zero of=zero bs=1M count=128 &
可见提示如下：
[1] 8973
root@ubuntu:/sys/fs/cgroup/devices# mkdir first
root@ubuntu:/sys/fs/cgroup/devices# ls -al
total 0
dr-xr-xr-x 10 root root 0 Sep 24 18:06 .
drwxr-xr-x 15 root root 380 Sep 24 18:06 ..
-rw-r--r-- 1 root root 0 Sep 25 06:25 cgroup.clone_children
-rw-r--r-- 1 root root 0 Sep 25 06:25 cgroup.procs
-r--r--r-- 1 root root 0 Sep 25 06:25 cgroup.sane_behavior
drwxr-xr-x 2 root root 0 Oct 16 10:07 default
--w------- 1 root root 0 Sep 25 06:25 devices.allow
--w------- 1 root root 0 Sep 25 06:25 devices.deny
-r--r--r-- 1 root root 0 Sep 25 06:25 devices.list
drwxr-xr-x 3 root root 0 Oct 9 15:45 docker
drwxr-xr-x 2 root root 0 Nov 17 19:47 first
drwxr-xr-x 4 root root 0 Oct 13 18:45 kubepods
drwxr-xr-x 4 root root 0 Oct 13 22:56 kubepods.slice
-rw-r--r-- 1 root root 0 Sep 25 06:25 notify_on_release
-rw-r--r-- 1 root root 0 Sep 25 06:25 release_agent
drwxr-xr-x 66 root root 0 Sep 24 18:06 system.slice
-rw-r--r-- 1 root root 0 Sep 25 06:25 tasks
drwxr-xr-x 2 root root 0 Oct 31 11:10 test.slice
drwxr-xr-x 2 root root 0 Sep 24 18:06 user.slice
root@ubuntu:/sys/fs/cgroup/devices# ls first/
cgroup.clone_children cgroup.procs devices.allow devices.deny devices.list notify_on_release tasks
root@ubuntu:/sys/fs/cgroup/devices#