mongodb的高可用实验，主从复制和副本集

mongodb不愧是功能上都比较完备的NoSQL数据库，其高可用方面做的明显要好一些。

主从复制的设置比较简单，关键是使用--master、--slave和--source参数，启动主从服务的命令如下

XXXXX@XXXXX-asus:~$ sudo mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --port 10000 --master --rest --nojournal
all output going to: /var/log/mongodb/mongodb.log
log file [/var/log/mongodb/mongodb.log] exists; copied to temporary file [/var/log/mongodb/mongodb.log.2014-03-15T02-46-11]
XXXXX@XXXXX-asus:~$ sudo mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --port 10000 --master --rest --nojournal --fork
forked process: 5969
all output going to: /var/log/mongodb/mongodb.log
log file [/var/log/mongodb/mongodb.log] exists; copied to temporary file [/var/log/mongodb/mongodb.log.2014-03-15T02-46-58]
child process started successfully, parent exiting
XXXXX@XXXXX-asus:~$ sudo mongod --dbpath /var/lib/mongodb_slave --logpath /var/log/mongodb/mongodb_slave.log --port 10001 --slave --source localhost:10000 --rest --nojournal --fork
forked process: 5987
all output going to: /var/log/mongodb/mongodb_slave.log
log file [/var/log/mongodb/mongodb_slave.log] exists; copied to temporary file [/var/log/mongodb/mongodb_slave.log.2014-03-15T02-47-12]
child process started successfully, parent exiting
XXXXX@XXXXX-asus:~$ ps -ef | grep mongod
root      5969     1  0 10:46 ?        00:00:00 mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --port 10000 --master --rest --nojournal --fork
root      5987     1  0 10:47 ?        00:00:00 mongod --dbpath /var/lib/mongodb_slave --logpath /var/log/mongodb/mongodb_slave.log --port 10001 --slave --source localhost:10000 --rest --nojournal --fork
XXXXX     6050  2732  0 10:48 pts/0    00:00:00 grep --color=auto mongod

可见，master进程在启动的时候使用了--master参数监听了10000端口，随后slave进程启动，使用了--slave --source localhost:10000，从master复制，并监听10001端口。做了主从的配置之后，在master上做的改动会立刻同步到slave上，如下所示

XXXXX@XXXXX-asus:~$ mongo localhost:10000
MongoDB shell version: 2.2.4
connecting to: localhost:10000/test
> use test
switched to db test
> db.master_slave.insert({"abc":123})
> db.master_slave.find()
{ "_id" : ObjectId("5323c1594678819d9c4323a2"), "abc" : 123 }
> exit
bye
XXXXX@XXXXX-asus:~$ mongo localhost:10001
MongoDB shell version: 2.2.4
connecting to: localhost:10001/test
> use test
switched to db test
> db.master_slave.find()
{ "_id" : ObjectId("5323c1594678819d9c4323a2"), "abc" : 123 }
> exit
bye
XXXXX@XXXXX-asus:~$ clear

先登陆master，并insert了一条数据，在slave上可以立刻看到这个改动。

mongodb的副本集是想在primary数据库启动的时候，有多个副本（secondary）在向primary同步数据，此时只有primary可用，所有secondary都不可用。在primary异常离线之后，副本集中立刻选取出一个primary来代替原有的primary继续工作。

配置副本集稍微麻烦一点，先要启动多个mongodb的副本集进程，如下

XXXXX@XXXXX-asus:~$ sudo mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --port 10000 --replSet testrep --nojournal --fork
forked process: 6862
all output going to: /var/log/mongodb/mongodb.log
log file [/var/log/mongodb/mongodb.log] exists; copied to temporary file [/var/log/mongodb/mongodb.log.2014-03-15T03-07-53]
child process started successfully, parent exiting
XXXXX@XXXXX-asus:~$ sudo mongod --dbpath /var/lib/mongodb1 --logpath /var/log/mongodb/mongodb1.log --port 10001 --replSet testrep --nojournal --fork
forked process: 6915
all output going to: /var/log/mongodb/mongodb1.log
child process started successfully, parent exiting
XXXXX@XXXXX-asus:~$ sudo mongod --dbpath /var/lib/mongodb2 --logpath /var/log/mongodb/mongodb2.log --port 10002 --replSet testrep --nojournal --fork
forked process: 6964
all output going to: /var/log/mongodb/mongodb2.log
child process started successfully, parent exiting
XXXXX@XXXXX-asus:~$ 

在上述步骤中，启动了3个mongodb进程，分别监听10000,10001,10002端口，组成副本集testrep，这里关键是要使用--replSet参数。

但是这样仍然不算已经完成配置，需要登陆mongodb初始化副本集才可以，如下

XXXXX@XXXXX-asus:~$ mongo localhost:10000
MongoDB shell version: 2.2.4
connecting to: localhost:10000/test
> rs.initiate({"_id":"testrep","members":[
... {"_id":1, "host":"localhost:10000"},
... {"_id":2, "host":"localhost:10001"},
... {"_id":3, "host":"localhost:10002"}
... ]})
{
    "info" : "Config now saved locally.  Should come online in about a minute.",
    "ok" : 1
}
> 

用rs.initiate初始化副本集，其中的参数要和启动mongodb时候的参数相符。初始化需要一点时间，完成之后，监听10000端口的mongodb变为primary，另外2个是secondary，可以看到log里面的状态变化，primary的log如下

Sat Mar 15 11:15:43 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Sat Mar 15 11:15:48 [initandlisten] connection accepted from 127.0.0.1:59140 #3 (1 connection now open)
Sat Mar 15 11:15:52 [conn3] replSet replSetInitiate admin command received from client
Sat Mar 15 11:15:52 [conn3] replSet replSetInitiate config object parses ok, 3 members specified
Sat Mar 15 11:15:52 [conn3] replSet replSetInitiate all members seem up
Sat Mar 15 11:15:52 [conn3] ******
Sat Mar 15 11:15:52 [conn3] creating replication oplog of size: 1810MB...
Sat Mar 15 11:15:52 [FileAllocator] allocating new datafile /var/lib/mongodb/local.1, filling with zeroes...
Sat Mar 15 11:15:52 [FileAllocator] creating directory /var/lib/mongodb/_tmp
Sat Mar 15 11:16:37 [FileAllocator] done allocating datafile /var/lib/mongodb/local.1, size: 2047MB,  took 44.99 secs
Sat Mar 15 11:16:41 [conn3] ******
Sat Mar 15 11:16:41 [conn3] replSet info saving a newer config version to local.system.replset
Sat Mar 15 11:16:41 [conn3] replSet saveConfigLocally done
Sat Mar 15 11:16:41 [conn3] replSet replSetInitiate config now saved locally.  Should come online in about a minute.
Sat Mar 15 11:16:41 [conn3] command admin.$cmd command: { replSetInitiate: { _id: "testrep", members: [ { _id: 1.0, host: "localhost:10000" }, { _id: 2.0, host: "localhost:10001" }, { _id: 3.0, host: "localhost:10002" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:49581556 reslen:112 49587ms
Sat Mar 15 11:16:41 [rsStart] replSet I am localhost:10000
Sat Mar 15 11:16:41 [rsStart] replSet STARTUP2
Sat Mar 15 11:16:41 [rsHealthPoll] replSet member localhost:10001 is up
Sat Mar 15 11:16:42 [rsSync] replSet SECONDARY
Sat Mar 15 11:16:43 [rsHealthPoll] replSet member localhost:10002 is up
Sat Mar 15 11:16:43 [rsMgr] replSet info electSelf 1
Sat Mar 15 11:16:43 [rsMgr] replSet couldn't elect self, only received 1 votes
Sat Mar 15 11:16:45 [initandlisten] connection accepted from 127.0.0.1:59173 #4 (2 connections now open)
Sat Mar 15 11:16:47 [rsHealthPoll] replSet member localhost:10002 is now in state STARTUP2
Sat Mar 15 11:16:47 [rsMgr] not electing self, localhost:10002 would veto
Sat Mar 15 11:16:49 [initandlisten] connection accepted from 127.0.0.1:59177 #5 (3 connections now open)
Sat Mar 15 11:17:01 [conn4] end connection 127.0.0.1:59173 (2 connections now open)
Sat Mar 15 11:17:01 [initandlisten] connection accepted from 127.0.0.1:59185 #6 (3 connections now open)
Sat Mar 15 11:17:01 [rsHealthPoll] DBClientCursor::init call() failed
Sat Mar 15 11:17:02 [rsHealthPoll] replSet info localhost:10001 is down (or slow to respond): DBClientBase::findN: transport error: localhost:10001 ns: admin.$cmd query: { replSetHeartbeat: "testrep", v: 1, pv: 1, checkEmpty: false, from: "localhost:10000", $auth: {} }
Sat Mar 15 11:17:02 [rsHealthPoll] replSet member localhost:10001 is now in state DOWN

secondary中的log如下

Sat Mar 15 11:16:39 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Sat Mar 15 11:16:47 [initandlisten] connection accepted from 127.0.0.1:59521 #2 (2 connections now open)
Sat Mar 15 11:16:49 [rsStart] trying to contact localhost:10000
Sat Mar 15 11:16:52 [rsStart] trying to contact localhost:10002
Sat Mar 15 11:17:02 [initandlisten] connection accepted from 127.0.0.1:59532 #3 (3 connections now open)
Sat Mar 15 11:17:03 [rsStart] DBClientCursor::init call() failed
Sat Mar 15 11:17:03 [conn2] command admin.$cmd command: { replSetHeartbeat: "testrep", v: 1, pv: 1, checkEmpty: false, from: "localhost:10002", $auth: {} } ntoreturn:1 keyUpdates:0  reslen:72 11356ms
Sat Mar 15 11:17:03 [conn1] command admin.$cmd command: { replSetHeartbeat: "testrep", v: 1, pv: 1, checkEmpty: false, from: "localhost:10000", $auth: {} } ntoreturn:1 keyUpdates:0  reslen:72 11177ms
Sat Mar 15 11:17:03 [conn1] end connection 127.0.0.1:59489 (2 connections now open)
Sat Mar 15 11:17:03 [rsStart] replSet I am localhost:10001
Sat Mar 15 11:17:03 [rsStart] replSet got config version 1 from a remote, saving locally
Sat Mar 15 11:17:03 [rsStart] replSet info saving a newer config version to local.system.replset
Sat Mar 15 11:17:03 [FileAllocator] allocating new datafile /var/lib/mongodb1/local.ns, filling with zeroes...
Sat Mar 15 11:17:03 [FileAllocator] creating directory /var/lib/mongodb1/_tmp
Sat Mar 15 11:17:04 [conn2] end connection 127.0.0.1:59521 (1 connection now open)
Sat Mar 15 11:17:04 [initandlisten] connection accepted from 127.0.0.1:59535 #4 (2 connections now open)
Sat Mar 15 11:17:06 [conn3] end connection 127.0.0.1:59532 (1 connection now open)
Sat Mar 15 11:17:06 [initandlisten] connection accepted from 127.0.0.1:59537 #5 (3 connections now open)
Sat Mar 15 11:17:07 [FileAllocator] done allocating datafile /var/lib/mongodb1/local.ns, size: 16MB,  took 1.438 secs
Sat Mar 15 11:17:07 [FileAllocator] allocating new datafile /var/lib/mongodb1/local.0, filling with zeroes...
Sat Mar 15 11:17:13 [FileAllocator] done allocating datafile /var/lib/mongodb1/local.0, size: 64MB,  took 5.174 secs
Sat Mar 15 11:17:13 [FileAllocator] allocating new datafile /var/lib/mongodb1/local.1, filling with zeroes...
Sat Mar 15 11:17:15 [rsStart] replSet saveConfigLocally done
Sat Mar 15 11:17:16 [rsStart] replSet STARTUP2
Sat Mar 15 11:17:16 [rsSync] ******
Sat Mar 15 11:17:16 [rsSync] creating replication oplog of size: 1631MB...
Sat Mar 15 11:17:17 [rsHealthPoll] replSet member localhost:10000 is up
Sat Mar 15 11:17:17 [rsHealthPoll] replSet member localhost:10000 is now in state SECONDARY
Sat Mar 15 11:17:17 [rsHealthPoll] replSet member localhost:10002 is up
Sat Mar 15 11:17:17 [rsHealthPoll] replSet member localhost:10002 is now in state STARTUP2
Sat Mar 15 11:17:21 [FileAllocator] done allocating datafile /var/lib/mongodb1/local.1, size: 128MB,  took 7.663 secs
Sat Mar 15 11:17:21 [FileAllocator] allocating new datafile /var/lib/mongodb1/local.2, filling with zeroes...
Sat Mar 15 11:17:26 [conn4] end connection 127.0.0.1:59535 (1 connection now open)
 

也可以直接登入primary去查看状态，用rs.status()

XXXXX@XXXXX-asus:~$ mongo localhost:10000
MongoDB shell version: 2.2.4
connecting to: localhost:10000/test
testrep:PRIMARY> rs.status()
{
    "set" : "testrep",
    "date" : ISODate("2014-03-15T05:18:36Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 1,
            "name" : "localhost:10000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 7843,
            "optime" : Timestamp(1394853401000, 1),
            "optimeDate" : ISODate("2014-03-15T03:16:41Z"),
            "self" : true
        },
        {
            "_id" : 2,
            "name" : "localhost:10001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 7292,
            "optime" : Timestamp(1394853401000, 1),
            "optimeDate" : ISODate("2014-03-15T03:16:41Z"),
            "lastHeartbeat" : ISODate("2014-03-15T05:18:35Z"),
            "pingMs" : 0
        },
        {
            "_id" : 3,
            "name" : "localhost:10002",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 7313,
            "optime" : Timestamp(1394853401000, 1),
            "optimeDate" : ISODate("2014-03-15T03:16:41Z"),
            "lastHeartbeat" : ISODate("2014-03-15T05:18:34Z"),
            "pingMs" : 0
        }
    ],
    "ok" : 1
}
testrep:PRIMARY> 

可以看到，再次登入primary的时候，提示符中的信息已经变为testrep:PRIMARY>，表示当前登陆的是副本集的primary节点。

如果此时对primary做读写操作，都是可以的，但是对secondary都不能读写，如下

XXXXX@XXXXX-asus:~$ mongo localhost:10000
MongoDB shell version: 2.2.4
connecting to: localhost:10000/test
testrep:PRIMARY> use test
switched to db test
testrep:PRIMARY> db.testrep.insert({"abcd":1234})
testrep:PRIMARY> db.testrep.find()
{ "_id" : ObjectId("5323e46faeb9bd3d8d02e2b6"), "abcd" : 1234 }
testrep:PRIMARY> exit
bye
XXXXX@XXXXX-asus:~$ mongo localhost:10001
MongoDB shell version: 2.2.4
connecting to: localhost:10001/test
testrep:SECONDARY> use test
switched to db test
testrep:SECONDARY> db.testrep.find()
error: { "$err" : "not master and slaveOk=false", "code" : 13435 }
testrep:SECONDARY> db.testrep.insert({"efgh":5678})
not master
testrep:SECONDARY> exit
bye
XXXXX@XXXXX-asus:~$ 

如果此时primary离线，2个secondary中会选举出一个成为新的primary，用kill -9 来模拟这个primary离线这个动作

XXXXX@XXXXX-asus:~$ ps -ef | grep mongo
root      6862     1  1 11:07 ?        00:01:30 mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --port 10000 --replSet testrep --nojournal --fork
root      6915     1  0 11:08 ?        00:01:26 mongod --dbpath /var/lib/mongodb1 --logpath /var/log/mongodb/mongodb1.log --port 10001 --replSet testrep --nojournal --fork
root      6964     1  1 11:08 ?        00:01:27 mongod --dbpath /var/lib/mongodb2 --logpath /var/log/mongodb/mongodb2.log --port 10002 --replSet testrep --nojournal --fork
XXXXX    13047  5380  0 13:33 pts/2    00:00:00 grep --color=auto mongo
XXXXX@XXXXX-asus:~$ sudo kill -9 6862
XXXXX@XXXXX-asus:~$ ps -ef | grep mongo
root      6915     1  0 11:08 ?        00:01:26 mongod --dbpath /var/lib/mongodb1 --logpath /var/log/mongodb/mongodb1.log --port 10001 --replSet testrep --nojournal --fork
root      6964     1  1 11:08 ?        00:01:27 mongod --dbpath /var/lib/mongodb2 --logpath /var/log/mongodb/mongodb2.log --port 10002 --replSet testrep --nojournal --fork
XXXXX    13058  5380  0 13:34 pts/2    00:00:00 grep --color=auto mongo
XXXXX@XXXXX-asus:~$ 

此时secondary的log会显示其已经代替原先的primary成为新的primary，新的primary是原先监听10001端口的mongodb进程

Sat Mar 15 13:33:59 [initandlisten] connection accepted from 127.0.0.1:35927 #556 (2 connections now open)
Sat Mar 15 13:34:00 [conn555] end connection 127.0.0.1:35925 (1 connection now open)
Sat Mar 15 13:34:00 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: localhost:10000
Sat Mar 15 13:34:01 [rsHealthPoll] DBClientCursor::init call() failed
Sat Mar 15 13:34:01 [rsHealthPoll] replSet info localhost:10000 is down (or slow to respond): DBClientBase::findN: transport error: localhost:10000 ns: admin.$cmd query: { replSetHeartbeat: "testrep", v: 1, pv: 1, checkEmpty: false, from: "localhost:10001", $auth: {} }
Sat Mar 15 13:34:01 [rsHealthPoll] replSet member localhost:10000 is now in state DOWN
Sat Mar 15 13:34:02 [rsMgr] replSet info electSelf 2
Sat Mar 15 13:34:02 [rsMgr] replSet PRIMARY
Sat Mar 15 13:34:03 [rsHealthPoll] couldn't connect to localhost:10000: couldn't connect to server localhost:10000
Sat Mar 15 13:34:03 [rsHealthPoll] couldn't connect to localhost:10000: couldn't connect to server localhost:10000
Sat Mar 15 13:34:05 [rsHealthPoll] couldn't connect to localhost:10000: couldn't connect to server localhost:10000
Sat Mar 15 13:34:07 [rsHealthPoll] couldn't connect to localhost:10000: couldn't connect to server localhost:10000
Sat Mar 15 13:34:09 [rsHealthPoll] couldn't connect to localhost:10000: couldn't connect to server localhost:10000
Sat Mar 15 13:34:10 [initandlisten] connection accepted from 127.0.0.1:35945 #557 (2 connections now open)
Sat Mar 15 13:34:11 [rsHealthPoll] couldn't connect to localhost:10000: couldn't connect to server localhost:10000
Sat Mar 15 13:34:13 [rsHealthPoll] couldn't connect to localhost:10000: couldn't connect to server localhost:10000

也可以直接登陆新的primary数据库查看副本集的状态

XXXXX@XXXXX-asus:~$ mongo localhost:10001
MongoDB shell version: 2.2.4
connecting to: localhost:10001/test
testrep:PRIMARY> rs.status()
{
    "set" : "testrep",
    "date" : ISODate("2014-03-15T05:39:24Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 1,
            "name" : "localhost:10000",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(1394861167000, 1),
            "optimeDate" : ISODate("2014-03-15T05:26:07Z"),
            "lastHeartbeat" : ISODate("2014-03-15T05:33:59Z"),
            "pingMs" : 0,
            "errmsg" : "socket exception [CONNECT_ERROR] for localhost:10000"
        },
        {
            "_id" : 2,
            "name" : "localhost:10001",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 9065,
            "optime" : Timestamp(1394861167000, 1),
            "optimeDate" : ISODate("2014-03-15T05:26:07Z"),
            "self" : true
        },
        {
            "_id" : 3,
            "name" : "localhost:10002",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 8527,
            "optime" : Timestamp(1394861167000, 1),
            "optimeDate" : ISODate("2014-03-15T05:26:07Z"),
            "lastHeartbeat" : ISODate("2014-03-15T05:39:23Z"),
            "pingMs" : 0
        }
    ],
    "ok" : 1
}
testrep:PRIMARY> 

可以看到，原先监听10000端口的副本状态是“(not reachable/healthy)”，此时新的primary数据库可以做查询新增动作

XXXXX@XXXXX-asus:~$ mongo localhost:10001
MongoDB shell version: 2.2.4
connecting to: localhost:10001/test
testrep:PRIMARY> use test
switched to db test
testrep:PRIMARY> db.testrep.find()
{ "_id" : ObjectId("5323e46faeb9bd3d8d02e2b6"), "abcd" : 1234 }
testrep:PRIMARY> db.testrep.insert({"efgh":5678})
testrep:PRIMARY> db.testrep.find()
{ "_id" : ObjectId("5323e46faeb9bd3d8d02e2b6"), "abcd" : 1234 }
{ "_id" : ObjectId("5323e8d10a61cc7b258aac5a"), "efgh" : 5678 }
testrep:PRIMARY> 

此时如果再次启动原先监听10000端口的进程，多半无法启动，因为数据库异常推出，锁没有释放，此时需要先对数据库做repair然后再启动

XXXXX@XXXXX-asus:~$ sudo mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --port 10000 --replSet testrep --nojournal --repair --fork
forked process: 16902
all output going to: /var/log/mongodb/mongodb.log
log file [/var/log/mongodb/mongodb.log] exists; copied to temporary file [/var/log/mongodb/mongodb.log.2014-03-15T06-00-10]
child process started successfully, parent exiting
XXXXX@XXXXX-asus:~$ sudo mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --port 10000 --replSet testrep --nojournal --fork
forked process: 17059
all output going to: /var/log/mongodb/mongodb.log
log file [/var/log/mongodb/mongodb.log] exists; copied to temporary file [/var/log/mongodb/mongodb.log.2014-03-15T06-02-04]
child process started successfully, parent exiting
XXXXX@XXXXX-asus:~$ ps -ef | grep mongo
root      6915     1  0 11:08 ?        00:01:44 mongod --dbpath /var/lib/mongodb1 --logpath /var/log/mongodb/mongodb1.log --port 10001 --replSet testrep --nojournal --fork
root      6964     1  1 11:08 ?        00:01:44 mongod --dbpath /var/lib/mongodb2 --logpath /var/log/mongodb/mongodb2.log --port 10002 --replSet testrep --nojournal --fork
XXXXX    13123  5712  0 13:34 pts/3    00:00:00 vim /var/log/mongodb/mongodb1.log
XXXXX    15728  5086  0 13:51 pts/1    00:00:00 mongo localhost:10001
root     17059     1  3 14:02 ?        00:00:00 mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --port 10000 --replSet testrep --nojournal --fork
XXXXX    17131  5380  0 14:02 pts/2    00:00:00 grep --color=auto mongo
XXXXX@XXXXX-asus:~$ 

此时，再次使用rs.status()就可以看到和最初状态几乎相同的输出了，唯一差别是10000端口的进程变为secondary，而10001端口的进程变为primary。

另外，mongodb官方已经开始推荐使用副本集而非主从复制来建立高可用，副本集常用的命令还有，rs.add("localhost:10000")，rs.remove("localhost:10000")分别用来增加或者删除副本集中的一个节点，在以后的文章中描述。