zoukankan      html  css  js  c++  java
  • PostgreSQL的 create index concurrently

    对于PostgreSQL的 "create index concurrently". 我个人认为其中存在一个bug。

    我的验证过程如下:

    我有两个表,tab01和 tab02,这两个表之间没有任何关联。

    我认为 对 tab02执行 "create index concurrently" 不会对 访问tab01的事务有任何影响,然而事实并非尽然。

    我第一程序的表现: 通过ecpg执行事务,再通过 "create index concurrently" 给tab02建立索引,成功。

    我第二程序的表现:通过ecpg执行事务,再通过 "create index concurrently" 给tab02建立索引,被阻塞。

    我第三个测试:      通过psql发起事务,  另一个psql客户端执行 "create index concurrently" 成功。

    我第四个测试:    通过psql发起事务 另一个psql客户端执行 "create index concurrently",被阻塞。

    无论 PostgreSQL9.1.2,还是PostgreSQL9.2.4,结果是一样的。

    数据准备:

    [postgres@server bin]$ ./psql -U tester -d tester
    psql (9.1.2)
    Type "help" for help.
    tester=> d tab01;
              Table "public.tab01"
     Column |         Type         | Modifiers
    --------+----------------------+-----------
     id     | integer              |
     cd     | character varying(4) |
    
    
    tester=> d tab02;
        Table "public.tab02"
     Column |  Type   | Modifiers
    --------+---------+-----------
     id     | integer |
     value  | integer |
    
    tester=> select * from tab01;
     id | cd
    ----+----
      1 | 14
      2 | 15
      3 | 14
    (3 rows)
    
    tester=> select * from tab02;
     id | value
    ----+-------
      1 |   100
      2 |   200
      3 |   300
    (3 rows)
    tester=>

    我的测试方法:

    对第一个程序和第二个程序:

    当我的eccp程序正在睡眠的时候,我另外开一个终端,执行: 

     "create index concurrently idx_tab02_id_new on tab02(id)"

    结果是:

    第一个程序执行中,我可成功建立索引。 
    第二个程序执行中,我无法建立索引,会被阻塞
    而我的tab01和tab02之间,没有任何关联。而且我也不认为我的ecpg程序会有潜在的可能去使用tab02的索引。

    事实上,如果我去看ecpg预编译后得到的c程序,我可以看到:

     { ECPGdo(__LINE__, 0, 1, "db_conn", 0, ECPGst_normal, "select count ( * ) from tab01 where cd = $1 ",
    
            ECPGt_char,(vcd),(long)4 + 1,(long)1,(4 + 1)*sizeof(char),
    
            ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EOIT,
    
            ECPGt_int,&(vCount),(long)1,(long)1,sizeof(int),
    
            ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EORT);}

    当我给$1加入引号后,我就可以成功地建立索引了。

     { ECPGdo(__LINE__, 0, 1, "db_conn", 0, ECPGst_normal, "select count ( * ) from tab01 where cd = '$1' ",
    
            ECPGt_char,(vcd),(long)4 + 1,(long)1,(4 + 1)*sizeof(char),
    
            ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EOIT,
    
            ECPGt_int,&(vCount),(long)1,(long)1,sizeof(int),
    
            ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EORT);}

    下面是我测试程序的源代码:

    第一个:

    [root@server soft]# cat ./test01/test01.pc
    int main()
    {
    
       EXEC SQL BEGIN DECLARE SECTION;
             int vCount;
             char vcd[4+1]; 
       EXEC SQL END DECLARE SECTION;
    EXEC SQL CONNECT TO
    'tester@127.0.0.1:5432' AS db_conn USER tester IDENTIFIED BY tester; EXEC SQL AT db_conn SELECT COUNT(*) INTO :vCount FROM tab01; fprintf(stderr,"count is:%d ",vCount); fprintf(stderr,"Before disconnect,sleep for 500 seconds "); sleep(500); EXEC SQL DISCONNECT db_conn; fprintf(stderr,"After disconnect,sleep for 600 seconds "); sleep(600); return 0; } [root@server soft]#

    第二个:

    [root@server soft]# cat ./test02/test02.pc
    
    int main()
    
    {
       EXEC SQL BEGIN DECLARE SECTION;
             int vCount;
             char vcd[4+1];                        
       EXEC SQL END DECLARE SECTION;
    
    
       EXEC SQL CONNECT TO 'tester@127.0.0.1:5432' AS db_conn
         USER tester IDENTIFIED BY tester;
    
       char *pCd="14";
       memset(vcd,'',5);
       strncpy(vcd, pCd,4);             
                                                                   
       EXEC SQL AT db_conn SELECT COUNT(*)            
            INTO :vCount FROM tab01 WHERE cd = :vcd;
                                                                           
       fprintf(stderr,"count is:%d
    ",vCount);
    fprintf(stderr,
    "Before disconnect,sleep for 500 seconds "); sleep(500); EXEC SQL DISCONNECT db_conn; fprintf(stderr,"After disconnect,sleep for 600 seconds "); sleep(600); return 0; } [root@server soft]#

    而且,通过 psql,还可以发现一个与 create index concurrently 相关的现象:

    我的第三个测试:

    客户端1:

    [postgres@server pgsql]$ ./bin/psql -d tester -U tester
    psql (9.1.2)
    Type "help" for help.
    
    tester=> begin;
    BEGIN
    tester=> select * from tab01 where cd = '14';
     id | cd
    ----+----
      1 | 14
      3 | 14
    (2 rows)
    tester=>

    客户端2: 

    [postgres@server pgsql]$ ./bin/psql -d tester -U tester
    psql (9.1.2)
    Type "help" for help.
    
    tester=> create index concurrently idx_tab02_id_new on tab02(id);
    

    可以很快就成功创建索引。

    我的第四个测试:

    客户端1:

    [postgres@server pgsql]$ ./bin/psql -d tester -U tester
    psql (9.1.2)
    Type "help" for help.
    tester=> begin;
    BEGIN
    tester=> select * from tab01 where cd = '14';
     id | cd
    ----+----
      1 | 14
      3 | 14
    (2 rows)
     
    
    tester=> select pg_sleep(500);
     pg_sleep
    ----------
    (1 row)
    tester=>

    客户端2:

    [postgres@server pgsql]$ ./bin/psql -d tester -U tester
    psql (9.1.2)
    Type "help" for help.
    
    tester=> create index concurrently idx_tab02_id_new on tab02(id);

    客户端2的创建索引会被阻塞

    根据我对PostgreSQL的源代码的跟踪,可以看到有如下的调用关系:

    PortalRunMulti--> PortalRunUtility-->Standard_ProcessUtility-->DefineIndex

    而我对DefineIndex作简化后,可以看到:

    {     
    … old_snapshots
    = GetCurrentVirtualXIDs(snapshot->xmin, true, false, PROC_IS_AUTOVACUUM | PROC_IN_VACUUM, &n_old_snapshots); for (i = 0; i < n_old_snapshots; i++) { … if (VirtualTransactionIdIsValid(old_snapshots[i])) VirtualXactLockTableWait(old_snapshots[i]); } … }

    对于我的第一个测试程序,GetCurrentVirtualXIDs 函数执行后,n_old_snapshots 的值为0 ,
    for (i = 0; i < n_old_snapshots; i++) 循环不会被执行,索引的生成不会被阻塞。

    对我的第二个测试程序,GetCurrentVirtualXIDs 函数执行后,n_old_snapshots 的值为1, 
    for (i = 0; i < n_old_snapshots; i++) 循环会被执行。
    VirtualXactLockTableWait(old_snapshots[i]) 的执行,导致等待一个锁,所以索引生成被阻塞。

    再往下分析:

    VirtualTransactionId *
    GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
                          bool allDbs, int excludeVacuum,
                          int *nvxids)
    {
        VirtualTransactionId *vxids;
        ProcArrayStruct *arrayP = procArray;
        int            count = 0;
        int            index;
    
        /* allocate what's certainly enough result space */
        vxids = (VirtualTransactionId *)
            palloc(sizeof(VirtualTransactionId) * arrayP->maxProcs);
    
        LWLockAcquire(ProcArrayLock, LW_SHARED);
    for (index = 0; index < arrayP->numProcs; index++)
        {
    volatile PGPROC *proc = arrayP->procs[index];
    
            if (proc == MyProc)
                continue;
    
            if (excludeVacuum & proc->vacuumFlags)
                continue;
    
            if (allDbs || proc->databaseId == MyDatabaseId)
            {
     /* Fetch xmin just once - might change on us */
                TransactionId pxmin = proc->xmin;
    if (excludeXmin0 && !TransactionIdIsValid(pxmin)) continue; /* * InvalidTransactionId precedes all other XIDs, so a proc that * hasn't set xmin yet will not be rejected by this test. */ if (!TransactionIdIsValid(limitXmin) || TransactionIdPrecedesOrEquals(pxmin, limitXmin)) { VirtualTransactionId vxid; GET_VXID_FROM_PGPROC(vxid, *proc);
    if (VirtualTransactionIdIsValid(vxid)) vxids[count++] = vxid; } } } LWLockRelease(ProcArrayLock); *nvxids = count; return vxids; }

    对于我的第一个程序,测试结果显示:pxmin 为零,TransactionIdIsValid(pxmin) 为假。所以如下代码导致跳过循环一次。

    if (excludeXmin0 && !TransactionIdIsValid(pxmin))
                    continue;

    没有机会执行 vxids[count++]=vxid 这一行。

    那么pxmin是如何来的?

    看这句: TransactionId pxmin = proc->xmin;

    而xmin的含义是:当我们执行程序中对数据进行增删改的时候,会将当前transaction id 赋予给 xmin。

    写记录的时候,把这个xmin写入该行记录头。

    如此,每个进程看来,它只关心 xmin 小于自己的transaction id的。PostgreSQL用这种方式来保证MVCC。

    但此处,proc->xmin为零是很不合理的。

    此时,

    if (allDbs || proc->databaseId == MyDatabaseId) 里的:
    if (excludeXmin0 && !TransactionIdIsValid(pxmin))就会成立,所以会直接continue调回循环开始处,也就没有机会去
    vxids[count++] = vxid;

    在我的第二个程序里,proc->xmin根本就不为零。故此说,这是一个bug。

    另外的佐证:对我的三个测试,运行下列SQL文:

    pgsql=# select l.pid, l.mode, sa.procpid, sa.current_query
    from pg_locks l
    inner join pg_stat_activity sa
            on l.pid = sa.procpid
    where l.mode like '%xclusive%';

    一开始在pg_sleep(100)执行期间,可以看到:

    pgsql=# select l.pid, l.mode, sa.procpid, sa.current_query
    from pg_locks l
    inner join pg_stat_activity sa
            on l.pid = sa.procpid
    where l.mode like '%xclusive%';
     pid  |     mode      | procpid |                   current_query                    
    ------+---------------+---------+----------------------------------------------------
     5356 | ExclusiveLock |    5356 | select l.pid, l.mode, sa.procpid, sa.current_query+
          |               |         | from pg_locks l                                   +
          |               |         | inner join pg_stat_activity sa                    +
          |               |         |         on l.pid = sa.procpid                     +
          |               |         | where l.mode like '%xclusive%';
     5517 | ExclusiveLock |    5517 | select pg_sleep(100);
    (2 rows)

    我开另外的终端,执行 "create index concurrently"的时候,再看:

    pgsql=# select l.pid, l.mode, sa.procpid, sa.current_query
    from pg_locks l
    inner join pg_stat_activity sa
            on l.pid = sa.procpid
    where l.mode like '%xclusive%';
     pid  |           mode           | procpid |                      current_query                       
    ------+--------------------------+---------+----------------------------------------------------------
     5356 | ExclusiveLock            |    5356 | select l.pid, l.mode, sa.procpid, sa.current_query      +
          |                          |         | from pg_locks l                                         +
          |                          |         | inner join pg_stat_activity sa                          +
          |                          |         |         on l.pid = sa.procpid                           +
          |                          |         | where l.mode like '%xclusive%';
     5517 | ExclusiveLock            |    5517 | select pg_sleep(100);
     5527 | ExclusiveLock            |    5527 | create index concurrently idx_tab02_id_new on tab02(id);
     5527 | RowExclusiveLock         |    5527 | create index concurrently idx_tab02_id_new on tab02(id);
     5527 | ShareUpdateExclusiveLock |    5527 | create index concurrently idx_tab02_id_new on tab02(id);
    (5 rows)

    等到 pg_sleep执行完毕的时候:

    pgsql=# select l.pid, l.mode, sa.procpid, sa.current_query
    from pg_locks l
    inner join pg_stat_activity sa
            on l.pid = sa.procpid
    where l.mode like '%xclusive%';
     pid  |           mode           | procpid |                      current_query                       
    ------+--------------------------+---------+----------------------------------------------------------
     5356 | ExclusiveLock            |    5356 | select l.pid, l.mode, sa.procpid, sa.current_query      +
          |                          |         | from pg_locks l                                         +
          |                          |         | inner join pg_stat_activity sa                          +
          |                          |         |         on l.pid = sa.procpid                           +
          |                          |         | where l.mode like '%xclusive%';
     5517 | ExclusiveLock            |    5517 | <IDLE> in transaction
     5527 | ExclusiveLock            |    5527 | create index concurrently idx_tab02_id_new on tab02(id);
     5527 | RowExclusiveLock         |    5527 | create index concurrently idx_tab02_id_new on tab02(id);
     5527 | ShareUpdateExclusiveLock |    5527 | create index concurrently idx_tab02_id_new on tab02(id);
    (5 rows)
  • 相关阅读:
    tar命令,vi编辑器
    Linux命令、权限
    Color Transfer between Images code实现
    利用Eclipse使用Java OpenCV(Using OpenCV Java with Eclipse)
    Matrix Factorization SVD 矩阵分解
    ZOJ Problem Set
    Machine Learning
    ZOJ Problem Set
    ZOJ Problem Set
    ZOJ Problem Set
  • 原文地址:https://www.cnblogs.com/gaojian/p/3154211.html
Copyright © 2011-2022 走看看