虚拟化集群中PBS-Torque的部署

1. 概述

本篇博客主要介绍在centos7操作系统集群中部署，配置和使用pbs调度系统的过程

centos7版本：CentOS Linux release 7.9.2009 (Core)

pbs调度系统版本：torque-6.1.3.tar.gz

集群信息：

使用三个虚拟节点部署pbs系统

节点名称	节点IP	节点角色	节点服务
node16	192.168.80.16	管理节点，登陆节点	pbs_server pbs_schd
node17	192.168.80.17	节点节点	pbs_mom
node18	192.168.80.18	计算节点	pbs_mom

本篇只是pbs调度软件torque的基本部署，配置，使用。更加复杂的功能，比如MPI环境，图形界面显示，GPU调度，Munge认证，数据库信息获取，高可用配置，未做详细的探究，后期有时间再进行完善。

2. 部署

一般集群都需要时间统一，全局身份认证，这两个步骤在本篇博客不作介绍。

本篇博客的node16-18，已经通过ldap和sssd实现了全局身份认证。

同时约定使用node16的一个软件安装目录作为共享目录，共享给node17和node18

2.1 创建和挂载共享目录

node16上执行mkdir -p /hpc/torque/6.1.3，该目录用来安装torque软件，共享给其他节点

执行mkdir -p /hpc/packages/作为编译torque的工作目录

编辑/etc/exportfs文件，内容如下：

/hpc 192.168.80.0/24(rw,no_root_squash,no_all_squash)
/home 192.168.80.0/24(rw,no_root_squash,no_all_squash)

执行：systemctl start nfs && systemctl enable nfs设置nfs启动和开机启动

执行：exportfs -r，使共享目录即时生效

在node17,node18上分别执行：

mkdir -p /hpc
mount.nfs 192.168.80.16:/hpc /hpc
mount.nfs 192.168.80.16:/home /home

2.2 部署torque软件

下载

wget http://wpfilebase.s3.amazonaws.com/torque/torque-6.1.3.tar.gz

解压

tar -zxvf torque-6.1.3.tar.gz -C /hpc/packages，解压到编译安装torque的工作目录

配置安装信息

# 1. yum安装软件依赖
yum -y install libxml2-devel boost-devel openssl-devel libtool readline-devel pam-devel hwloc-devel numactl-devel tcl-devel tk-devel
# yum groupinstall "GNOME Desktop" "Graphical Administrator Tools" 如果编译选项有--enable-gui时，需要安装图像界面依赖
# 2. configure传参，配置编译安装信息
./configure 
	--prefix=/hpc/torque/6.1.3 
	--mandir=/hpc/torque/6.1.3/man  
	--enable-cgroups 
	--enable-syslog 
	--enable-drmaa 
	--enable-gui 
	--with-xauth 
	--with-hwloc 
	--with-pam 
	--with-tcl 
	--with-tk 
	# --enable-numa-support 	#这个参数加上，要编辑mom.layout,不清楚规则，暂时取消
# 3. 更新。后期可能添加对MPI，GPU，Munge认证，高可用配置等的支持，本篇后期补充

执行结束后:

Building components: server=yes mom=yes clients=yes
                     gui=yes drmaa=yes pam=yes
PBS Machine type    : linux
Remote copy         : /usr/bin/scp -rpB
PBS home            : /var/spool/torque
Default server      : node13

Unix Domain sockets : 
Linux cpusets       : no
Tcl                 : -L/usr/lib64 -ltcl8.5 -ldl -lpthread -lieee -lm
Tk                  : -L/usr/lib64 -ltk8.5 -lX11 -L/usr/lib64 -ltcl8.5 -ldl -lpthread -lieee -lm
Authentication      : trqauthd

Ready for 'make'.

编译和安装

# 1. 编译，安装
make -j4 && make install
# 2. 生成安装脚本
# make packages

执行make packages输出：

这一步可以不做。在本篇博客中，工作目录在共享文件系统，因此只需要在每个节点执行make install即可。

[root@node16 torque-6.1.3]# make packages
Building packages from /hpc/packages/torque-6.1.3/tpackages
rm -rf /hpc/packages/torque-6.1.3/tpackages
mkdir /hpc/packages/torque-6.1.3/tpackages
Building ./torque-package-server-linux-x86_64.sh ...
libtool: install: warning: remember to run `libtool --finish /hpc/torque/6.1.3/lib'
Building ./torque-package-mom-linux-x86_64.sh ...
libtool: install: warning: remember to run `libtool --finish /hpc/torque/6.1.3/lib'
Building ./torque-package-clients-linux-x86_64.sh ...
libtool: install: warning: remember to run `libtool --finish /hpc/torque/6.1.3/lib'
Building ./torque-package-gui-linux-x86_64.sh ...
Building ./torque-package-pam-linux-x86_64.sh ...
libtool: install: warning: remember to run `libtool --finish /lib64/security'
Building ./torque-package-drmaa-linux-x86_64.sh ...
libtool: install: warning: relinking `libdrmaa.la'
libtool: install: warning: remember to run `libtool --finish /hpc/torque/6.1.3/lib'
Building ./torque-package-devel-linux-x86_64.sh ...
libtool: install: warning: remember to run `libtool --finish /hpc/torque/6.1.3/lib'
Building ./torque-package-doc-linux-x86_64.sh ...
Done.

The package files are self-extracting packages that can be copied
and executed on your production machines.  Use --help for options.

执行 libtool --finish /hpc/torque/6.1.3/lib

这一步可以不做，make install 操作默认操作

[root@node16 torque-6.1.3]# libtool --finish /hpc/torque/6.1.3/lib
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/sbin" ldconfig -n /hpc/torque/6.1.3/lib
----------------------------------------------------------------------
Libraries have been installed in:
   /hpc/torque/6.1.3/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------

接下来是配置环境变量和配置启动脚本

此时执行ls -lrt /usr/lib/systemd/system，发现目录下已经有了

-rw-r--r--  1 root root 1284 10月 12 11:17 pbs_server.service
-rw-r--r--  1 root root  704 10月 12 11:17 pbs_mom.service
-rw-r--r--  1 root root  335 10月 12 11:17 trqauthd.servic

少了一个pbs_sched.service启动脚本，从目录/hpc/packages/torque-6.1.3/contrib/systemd目录拷贝到系统中

cp /hpc/packages/torque-6.1.3/contrib/systemd/pbs_sched.service /usr/lib/systemd//system

此时执行ls -lrt /etc/profile.d能够看到目录下已经有了torque.sh，只需要执行source /etc/profile就可以了

3. 配置

3.1 配置管理节点

3.1.1添加pbs管理用户

这里设置为root用户

./torque.setup，这个脚本注释：create pbs_server database and default queue

[root@node16 torque-6.1.3]# ./torque.setup root
hostname: node13
Currently no servers active. Default server will be listed as active server. Error  15133
Active server name: node13  pbs_server port is: 15001
trqauthd daemonized - port /tmp/trqauthd-unix
trqauthd successfully started
initializing TORQUE (admin: root)

You have selected to start pbs_server in create mode.
If the server database exists it will be overwritten.
do you wish to continue y/(n)?
# 输入y

3.1.2 启动组件认证服务

3.1.1的步骤会启动认证服务trqauthd，执行ps axu|grep trqauthd可验证

后续通过systemctl start trqauthd时会报错，因此此时建议执行pkill -f trqauthd先处理掉该进程，再通过systemctl start trqauthd启动

pkill -f trqauthd
systemctl start trqauthd
systemctl enable trqauthd

3.1.3 启动主服务

配置计算节点，vim /var/spool/torque/server_priv/nodes

node17 np=4
node18 np=4

然后执行以下命令

systemctl status pbs_server 
systemctl start pbs_server #如果这一步执行失败，查看是否已经启动了pbs_server,如果启动执行pkill -f pbs_server,然后再执行此命令
systemctl enable pbs_server

执行qnodes查看信息

如果没有qnodes命令，执行source /etc/profile加载环境变量

node17
     state = down
     power_state = Running
     np = 4
     ntype = cluster
     mom_service_port = 15002
     mom_manager_port = 15003
     total_sockets = 0
     total_numa_nodes = 0
     total_cores = 0
     total_threads = 0
     dedicated_sockets = 0
     dedicated_numa_nodes = 0
     dedicated_cores = 0
     dedicated_threads = 0

node18
     state = down
     power_state = Running
     np = 4
     ntype = cluster
     mom_service_port = 15002
     mom_manager_port = 15003
     total_sockets = 0
     total_numa_nodes = 0
     total_cores = 0
     total_threads = 0
     dedicated_sockets = 0
     dedicated_numa_nodes = 0
     dedicated_cores = 0
     dedicated_threads = 0
# node17和node18因为没有启动pbs_mom,所以状态显示为down

3.1.4 启动调度服务

在node16上还要执行systemctl start pbs_sched，否则提交的作业全部为Q状态

设置开机启动systemctl enable pbs_sched

3.2 配置计算节点

3.1部分完成了管理节点node16的部署，包括：

• yum安装依赖环境
• 解压源码，配置编译信息，编译安装
• 配置管理用户
• 编辑配置文件
• 启动trqauthed服务，启动pbs_server服务，启动pbs_sched服务

计算节点需要完成的内容：

• yum安装依赖环境
• 配置管理节点信息
• 执行安装脚本，或者make install
• 启动trqauthd服务，启动pbs_mom服务

因为所有的操作均在共享目录下进行，因此只需要在node17和node18节点上执行make install即可

4. 使用

4.1 查看和激活队列

在3.1.1过程中执行torque.setup执行后，会默认添加一个batch队列，并设置了队列的一些基本属性

此时需要执行qmgr active queue batch，才能够往这个队列提交作业

提交作业在管理节点上执行，在计算节点执行会报错

[liwl01@node18 ~]$ echo "sleep 120"|qsub
qsub: submit error (Bad UID for job execution MSG=ruserok failed validating liwl01/liwl01 from node18)

在node16上提交作业

[liwl01@node16 ~]$ echo "sleep 300"|qsub
1.node16

计算中执行，S表示的作业状态为R，运行状态

[liwl01@node16 ~]$ qstat -a -n

node16: 
                                                                                  Req'd       Req'd       Elap
Job ID                  Username    Queue    Jobname          SessID  NDS   TSK   Memory      Time    S   Time
----------------------- ----------- -------- ---------------- ------ ----- ------ --------- --------- - ---------
1.node16                liwl01      batch    STDIN             20038     1      1       --   01:00:00 R  00:04:17
   node17/0

计算结束后，S表示的作业状态C，完成状态

[liwl01@node16 ~]$ qstat -a -n

node16: 
                                                                                  Req'd       Req'd       Elap
Job ID                  Username    Queue    Jobname          SessID  NDS   TSK   Memory      Time    S   Time
----------------------- ----------- -------- ---------------- ------ ----- ------ --------- --------- - ---------
1.node16                liwl01      batch    STDIN             20038     1      1       --   01:00:00 C       -- 
   node17/0

而qnodes执行结果

[liwl01@node16 ~]$ qnodes 
node17
     state = free
     power_state = Running
     np = 4
     ntype = cluster
     jobs = 0/1.node16
     status = opsys=linux,uname=Linux node17 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64,sessions=20038,nsessions=1,nusers=1,idletime=2159,totmem=3879980kb,availmem=3117784kb,physmem=3879980kb,ncpus=4,loadave=0.00,gres=,netload=1039583908,state=free,varattr= ,cpuclock=Fixed,macaddr=00:00:00:80:00:17,version=6.1.3,rectime=1634101823,jobs=1.node16
     mom_service_port = 15002
     mom_manager_port = 15003
     total_sockets = 1
     total_numa_nodes = 1
     total_cores = 4
     total_threads = 4
     dedicated_sockets = 0
     dedicated_numa_nodes = 0
     dedicated_cores = 0
     dedicated_threads = 1

5. 维护

待后期更新