TraTraffic Server 进程模型

1.概述

         Traffic Server包括三个一起工作的进程来服务Traffic Server的请求，管理/控制/监控系统的健康状况。图1说明了三个进程的关系，三个进程将会在下面描述。

 

图1：进程之间的关系

1）traffic_server进程是 Traffic Server的事务处理引擎。它负责接收连接、处理协议请求以及从本地缓存或源服务器提供资源。

2）traffic_manager进程是用来命令和控制Traffic Server的工具，负责启动、监控以及重新配置traffic_server进程。traffic_manager进程同时负责代理自动配置端口、统计接口、集群管理以及vip故障转移。

         如果traffic_manager进程检测到traffic_server进程失败，它不仅会立即重启该进程，而且会为所有传入的请求维护一个连接队列。在traffic_server重启前的几秒内传入的所有连接将会被保存在连接队列中，并以FIFO的方式处理。这个连接队列接受任何server故障重启时的连接。
 

3）traffic_cop进程监控traffic_server和traffic_manager进程的健康状况。traffic_cop进程通过抓取合成web页面的心跳请求方式周期性的（每分钟若干次）查询traffic_server和traffic_manager进程。如果失败事件发生（如果在超时时间间隔内没有收到请求或者收到错误的请求），traffic_cop重启traffic_server和traffic_manager进程。系统这样设计的好处便是给traffic_server进程加上了来自traffic_manager和traffic_cop的双重保障，因为traffic_server进程是工作进程，必须保证它的正常运行。-

4）traffic server采用的是多线程异步事件处理模型：Traffic Server并不是为每个连接都建立一个线程，而是事先创建一组数量可配置的工作线程，每一个工作线程上都运行着独立的异步事件处理程序。traffic_server创建若干组Thread，并将Event按类型调度到相应的Thread的Event队列上，Thread通过执行Event对应的Continuation中的回调函数，来完成状态的迁移。从初始态到终止态的迁移代表了整个事件的执行过程，而Thread是永不退出的，等待着下一个事件的到来。

         本文重点在于分析traffic server中三个进程的关系以及实现，对于其多线程异步事件处理模型不作深入分析。进程模型图如下：

2.实现原理

基本原理：对traffic_manager进程和traffic_server进程分别配置对应的manager_lockfile和server_lockfile文件，traffic_cop通过两个lockfile文件来监控traffic_manager和traffic_server进程，同理traffic_manager进程通过server_lockfile来监控traffic_server进程。图2说明了这种关系：

图2：进程以及lockfile文件的关系

 

关键实现：

 

关键类 Lockfile

Lockfile::Open(pid_t * holding_pid)函数详解：

解释和说明：Lockfile::Open(pid_t * holding_pid)会有三种类型的返回值，close-on-exec:具体作用在于当开辟其他进程调用exec（）族函数时，在调用exec函数之前为exec族函数释放对应的文件描述符。

(1):返回1说明lockfile可以被打开，这也说明与lockfile关联的进程没有运行，如果关联的进程在运行，lockfile会被进程持有，就不会被打开；

(2):返回0说明检测到lockfile被某个进程持有，那么将持有lockfile的进程ID写入holding_pid返回，持有lockfile的进程ID是在对应进程运行的时候，由Get()函数写入到lockfile中的；

(3):返回负值一共有三种情况，一是打开fname失败，二是获取close-on-exec标识失败，三是设置clsoe-on-exec标识失败。

重要的kill进程的相关函数，简要说明如下:

// kill

//用于杀死指定pid的进程

//return: 0--okay，-1—error

1.int kill(pid_t pid, int sig);

//ink_killall     

//杀死程序名称为pname的所有进程
// return: 0--okay，-1—error
 2. ink_killall(const char *pname, int sig);

ink_killall调用:

　　3. ink_killall_get_pidv_xmalloc (pname, &pidv, &pidvcnt);
　　4. ink_killall_kill_pidv (pidv, pidvcnt, sig);

// ink_killall_get_pidv_xmalloc
//根据程序panme，获取程序运行的进程ID到pidv数组中，以及进程的个数到pidvcnt

//变量中

//return: -1 error (pidv: set to NULL; pidvcnt: set to 0); 0 okay (pidv: ats_malloc'd //pid vector; pidvcnt: number of pid's;if pidvcnt is set to 0, then pidv will //be set to NULL)

3.int ink_killall_get_pidv_xmalloc(const char *pname, pid_t ** pidv, int *pidvcnt);

 
// ink_killall_kill_pidv (pidv, pidvcnt, sig);
//将pidv中记录的进程ID逐个调用kill( pidv[i],sig)
// return: 0--okay，-1—error
4.int ink_killall_kill_pidv(pid_t * pidv, int pidvcnt,int sig);
ink_killall_kill_pidv调用:
　　

　　1.kill(pid_t pid, int sig);

 

// safe_kill
//用于安全的杀死程序名称为pname的所有进程，lockfile_name为进程需要关联的lockfile文件//group表明是否需要杀死pname进程创造的子进程，因为它们在同一个进程组;

//return: void

 5. static void safe_kill(const char *lockfile_name, const char *pname, bool group)；
static void safe_killd调用:

　　6. Lockfile::Kill(killsig, coresig, pname);

　　7. Lockfile::KillGroup(killsig, coresig, pname);

// Lockfile::Kill

//处理好对应的lockfile文件，杀死程序名为pname的所有进程，其中sig一般就是kill信号，//initial_sig默认为0，用于发送给init_pid进程的

//return：void

6. void Lockfile::Kill(int sig, int initial_sig, const char *pname);
Lockfile::Kill调用:

　　8.LockKill::lockfile_kill_internal(pid, initial_sig, pid, pname, sig);

// Lockfile::KillGroup

//处理好对应的lockfile文件，杀死程序名为pname的进程，以及该进程创建的子进程（当然也包括//子进程创建的线程），sig为kill信号

//信号

//initial_sig同上kill函数

//return :void

7.void Lockfile::KillGroup(int sig, int initial_sig, const char *pname);
Lockfile::KillGroup调用:
　　
　　8.LockKill::lockfile_kill_internal(pid, initial_sig, pid, pname, sig);

// LockKill::lockfile_kill_internal

//首先杀死init_pid进程，然后杀死程序名称为pname的所有进程

//return :void

8.static void lockfile_kill_internal(pid_t init_pid, int init_sig, pid_t pid, const char *pname, int sig);
lockfile_kill_internal调用:

　　1.kill(init_pid, init_sig);

　　3.ink_killall_get_pidv_xmalloc(pname, &pidv, &pidvcnt);
　　4.ink_killall_kill_pidv(pidv, pidvcnt, sig);

若想了解详细实现细节，请参见源代码.

2.     模拟traffic_cop对traffic_manager和traffic_server的监控

Traffic_cop启动以后进入main函数，main函数会调用一个check函数，在check里面会周期性的调用check_programs()函数来对traffic_manager和traffic_server进行监控。check_programs()函数有些复杂，流程图如下图。

3.模拟测试                                                                                   

根据原理，模仿了traffic_cop、traffic_manager和traffic_server三个进程，其中将traffic_cop实现为守护进程，traffic_manager进程对traffic-server进程的监控类似于traffic_cop对traffic_manager与traffic_server的监控，故不作重复说明。实验中，由于测试traffic_manager与traffic_server进程健康度的函数heartbear_manager()、server_up()与heartbeat_server()函数涉及到端口通信部分内容，由于其不妨碍原理部分的模拟，略写了它们的代码，而是让它们直接返回正常值。（程序运行的时候需要manage_lokfile和server_lockfile文件，读者应自己在可执行文件所在文件夹下加上这两个文件）

程序运行后，敲入命令 ps –axj|grep binary得到图如下：

前四个标识分别是：父进程ID/进程ID/进程组ID/会话ID

 

图中可以看出它们的正常关系。

当traffic_manager进程异常退出的时候，traffic_cop会重启traffic_manager进程，在日志文件中可以看出这一动作：（日志部分内容如下）

==============traffic_server is running, pid:'5443'!

----------------traffic_manager is running, pid:'5436'!

==============traffic_server is running, pid:'5443'!

---------------traffic_manager has a expcetion and eixt!

Entering check_programs()

traffic_manager not running, making sure traffic_server is dead

Entering safe_kill

Leaving safe_kill

Entering spwan_manager()!

Leaving spwan_manager()!

Leaving check_programs

----------------traffic_manager is running, pid:'5463'!

Entering spwan_server()!

Leaving spwan_server()!

==============traffic_server is running, pid:'5467'!

从日志中可以看出，某个时刻，traffic_manager进程ID是5436，traffic_server进程ID是5443；下一时刻中，traffic_manager进程出现了异常(---------------traffic_manager has a expcetion and eixt!)，然后traffic_cop在周期性的check_programs()中发现” traffic_manager not running”,然后它杀死了traffic_server进程（” making sure traffic_server is dead”）,然后重新创建了traffic_manager进程（” Entering spwan_manager()!”）,traffic_manager进程的ID已经变成了5463，traffic_manager正常运行后，发现traffic_server进程没有运行，随后它调用spwan_server()产生新的traffic_server进程，其ID号变成了5467。说明traffic_cop监控功能正常。

       当traffic_server进程异常退出的时候，traffic_manager进程会检测到这一行为，然后重启traffic_server进程，在日志文件中也可以看出这一动作：（日志部分内容如下）

==============traffic_server is running, pid:'7703'!

----------------traffic_manager is running, pid:'7699'!

=================traffic_server has a expcetion and exit!

Entering safe_kill

Leaving safe_kill

--------------Entering spwan_server()!

--------------Leaving spwan_server()!

 

----------------traffic_manager is running, pid:'7699'!

==============traffic_server is running, pid:'7712'!

从日志上可以看出，某时刻，traffic_manager进程ID为7699，traffic_server进程ID是7703，接下来traffic_server进程出现异常退出，traffic_manager进程则调用spwan_server()重新开启了一个traffic_server进程，ID号为7712，此时traffic_manager进程的ID号仍然是7699，说明traffic_manager进程没有改变。这说明traffic_manager起到了监控traffic_server进程的作用。

4.总结

     为什么设计了三个进程来工作，而不是采用两个进程：直接让traffic_manager进程来监管traffic_server进程。由于traffic_manager进程所负担的系统角色说明单独的两个进程是无法满足系统要求的。特别是当traffic_manager进程检测到traffic_server进程失败的时候，它会暂时将请求放入队列中，所以它也需要在端口上暂时监听请求，这样系统就无法保障该进程不会出现异常，这也意味着traffic_manager进程同样也会出现异常。为此系统设计了traffic_cop守护进程来监控，traffic_cop进程的角色就是纯粹的监控另外两个进程，理论上这个守护进程是不会异常结束的，这样的三层设计比两层设计更安全更可靠。当三个进程协同工作的时候，客户对于服务器的异常是透明的（设计上如此，但并非绝对，当traffic_manager与traffic_server同时异常结束的时候，traffic_cop在重启它们的几秒钟内，客户的请求会无法接收，小概率），客户是不会感知到自己的请求会出现问题的，可能会感觉延迟大了一些。从服务器的架构设计上可以看出，服务器的要求是尽可能的稳定安全，对于异常情况的考虑应周全。

 源代码:

1.lock_and_kill.h

#ifndef LOCK_AND_KILL_H
#define LOCK_AND_KILL_H
#include <sys/types.h>
#include <string.h>
#define PATH_NAME_MAX 4096

/*-------------------------------------------------------------------------
   ink_killall
   - Sends signal 'sig' to all processes with the name 'pname'
   - Returns: -1 error
               0 okay
  -------------------------------------------------------------------------*/
int ink_killall(const char *pname, int sig);

/*-------------------------------------------------------------------------
   ink_killall_get_pidv_xmalloc
   - Get all pid's named 'pname' and stores into ats_malloc'd
     pid_t array, 'pidv'
   - Returns: -1 error (pidv: set to NULL; pidvcnt: set to 0)
               0 okay (pidv: ats_malloc'd pid vector; pidvcnt: number of pid's;
                   if pidvcnt is set to 0, then pidv will be set to NULL)
  -------------------------------------------------------------------------*/
int ink_killall_get_pidv_xmalloc(const char *pname, pid_t ** pidv, int *pidvcnt);

/*-------------------------------------------------------------------------
   ink_killall_kill_pidv
   - Kills all pid's in 'pidv' with signal 'sig'
   - Returns: -1 error
               0 okay
  -------------------------------------------------------------------------*/
int ink_killall_kill_pidv(pid_t * pidv, int pidvcnt, int sig);



class Lockfile
{
public:
    
  Lockfile(void):fd(0)
  {
    fname[0] = '\0';
  }


  // coverity[uninit_member]
  Lockfile(const char *filename):fd(0)
  {
    strcpy(fname, filename);
  }


  ~Lockfile(void)
  {
  }

  void SetLockfileName(const char *filename)
  {
    strcpy(fname, filename);
  }

  const char *GetLockfileName(void)
  {
    return fname;
  }

  // Open() -----非常重要的函数
  //
  // Tries to open a lock file, returning:
  //   -errno on error
  //   0 if someone is holding the lock (with holding_pid set)
  //   1 if we now have a writable lock file
  int Open(pid_t * holding_pid);

  // Get()
  //
  // Gets write access to a lock file, and if successful, truncates
  // file, and writes the current process ID.  Returns:
  //   -errno on error
  //   0 if someone is holding the lock (with holding_pid set)
  //   1 if we now have a writable lock file
  int Get(pid_t * holding_pid);

  // Close()
  //
  // Closes the file handle on the opened Lockfile.
  void Close(void);

  // Kill()
  // KillGroup()
  //
  // Ensures no one is holding the lock. It tries to open the lock file
  // and if that does not succeed, it kills the process holding the lock.
  // If the lock file open succeeds, it closes the lock file releasing
  // the lock.
  //
  // The intial signal can be used to generate a core from the process while
  // still ensuring it dies.
  void Kill(int sig, int initial_sig = 0, const char *pname = NULL);
  void KillGroup(int sig, int initial_sig = 0, const char *pname = NULL);

private:
  char fname[PATH_NAME_MAX];
  int fd;
};


#endif

2.lock_and_kill.cpp

#include <stdio.h>
#include <stdlib.h>
#include <dirent.h>
#include<unistd.h> 
#include<sys/file.h>
#include <errno.h>
#include <signal.h>

#include "lock_and_kill.h"


#define PROC_BASE "/proc"
#define INITIAL_PIDVSIZE 32
#define LOCKFILE_BUF_LEN 16 
#define LINE_MAX 1024 //may be hava problem with it
int
ink_killall(const char *pname, int sig)
{
  int err;
  pid_t *pidv;
  int pidvcnt;
  
  if (ink_killall_get_pidv_xmalloc(pname, &pidv, &pidvcnt) < 0) {
    return -1;
  }

  if (pidvcnt == 0) {
    free(pidv);
    return 0;
  }

  err = ink_killall_kill_pidv(pidv, pidvcnt, sig);
  free(pidv);
  return err;
}

int
ink_killall_get_pidv_xmalloc(const char *pname, pid_t ** pidv, int *pidvcnt)
{
  DIR *dir;
  FILE *fp;
  struct dirent *de;
  pid_t pid, self;
  char buf[LINE_MAX], *p, *comm;
  int pidvsize = INITIAL_PIDVSIZE;

  if (!pname || !pidv || !pidvcnt)
    goto l_error;

  self = getpid();
  if (!(dir = opendir(PROC_BASE)))
    goto l_error;

  *pidvcnt = 0;
  *pidv = (pid_t *)malloc(pidvsize * sizeof(pid_t));

  while ((de = readdir(dir))) {
    if (!(pid = (pid_t) atoi(de->d_name)) || pid == self)
      continue;
    snprintf(buf, sizeof(buf), PROC_BASE "/%d/stat", pid);
    if ((fp = fopen(buf, "r"))) {
      if (fgets(buf, sizeof buf, fp) == 0)
        goto l_close;
      if ((p = strchr(buf, '('))) {
        comm = p + 1;
        if ((p = strchr(comm, ')')))
          *p = '\0';
        else
          goto l_close;
        if (strcmp(comm, pname) == 0) {
          if (*pidvcnt >= pidvsize) {
            pid_t *pidv_realloc;
            pidvsize *= 2;
            if (!(pidv_realloc = (pid_t *)realloc(*pidv, pidvsize * sizeof(pid_t)))) {
              free(*pidv);
              goto l_error;
            } else {
              *pidv = pidv_realloc;
            }
          }
          (*pidv)[*pidvcnt] = pid;
          (*pidvcnt)++;
        }
      }
    l_close:
      fclose(fp);
    }
  }
  closedir(dir);

  if (*pidvcnt == 0) {
    free(*pidv);
    *pidv = 0;
  }
  return 0;
l_error:
  *pidv = NULL;
  *pidvcnt = 0;
  return -1;
}

int
ink_killall_kill_pidv(pid_t * pidv, int pidvcnt, int sig)
{
  int err = 0;
  if (!pidv || (pidvcnt <= 0))
    return -1;
  while (pidvcnt > 0) {
    pidvcnt--;
    if (kill(pidv[pidvcnt], sig) < 0)
      err = -1;
  }
  return err;
}


////////////////////类函数的实现在下面//////////////////////////////////
////////////////////////////////////////////////////////////////////////
int
Lockfile::Open(pid_t * holding_pid)
{
  char buf[LOCKFILE_BUF_LEN];
  pid_t val;
  int err;
  *holding_pid = 0;

#define FAIL(x) \
{ \
  if (fd > 0) \
    close (fd); \
  return (x); \
}

  struct flock lock;
  char *t;
  int size;//开始的时候设置成无效的一个值

  // Try and open the Lockfile. Create it if it does not already
  // exist.
  do {
    fd = open(fname, O_RDWR | O_CREAT, 0644);
  } while ((fd < 0) && (errno == EINTR));

  if (fd < 0)
    return (-errno);

  // Lock it. Note that if we can't get the lock EAGAIN will be the
  // error we receive.
  lock.l_type = F_WRLCK;
  lock.l_start = 0;
  lock.l_whence = SEEK_SET;
  lock.l_len = 0;

  do {
    err = fcntl(fd, F_SETLK, &lock);
  } while ((err < 0) && (errno == EINTR));

  if (err < 0) {
    // We couldn't get the lock. Try and read the process id of the
    // process holding the lock from the lockfile.
    t = buf;

    for (size = 15; size > 0;) {
      do {
        err = read(fd, t, size);
      } while ((err < 0) && (errno == EINTR));

      if (err < 0)
        FAIL(-errno);
      if (err == 0)
        break;

      size -= err;
      t += err;
    }
    *t = '\0';

    // coverity[secure_coding]
    if (sscanf(buf, "%d\n", (int*)&val) != 1) {
      *holding_pid = 0;
    } else {
      *holding_pid = val;
    }
    FAIL(0);
    
  }
  // If we did get the lock, then set the close on exec flag so that
  // we don't accidently pass the file descriptor to a child process
  // when we do a fork/exec.
  do {
    err = fcntl(fd, F_GETFD, 0);
  } while ((err < 0) && (errno == EINTR));

  if (err < 0)
    FAIL(-errno);
  
  val = err | FD_CLOEXEC;

  do {
    err = fcntl(fd, F_SETFD, val);
  } while ((err < 0) && (errno == EINTR));

  if (err < 0)
    FAIL(-errno);

  // Return the file descriptor of the opened lockfile. When this file
  // descriptor is closed the lock will be released.
  return (1);                   // success
#undef FAIL
}

int
Lockfile::Get(pid_t * holding_pid)
{
  char buf[LOCKFILE_BUF_LEN];
  int err;
  *holding_pid = 0;

  fd = -1;

  // Open the Lockfile and get the lock. If we are successful, the
  // return value will be the file descriptor of the opened Lockfile.
  err = Open(holding_pid);
  if (err != 1)
    return err;

  if (fd < 0) {
    return -1;
  }

  // Truncate the Lockfile effectively erasing it.
  do {
    err = ftruncate(fd, 0);
  } while ((err < 0) && (errno == EINTR));

  if (err < 0) {
    close(fd);
    return (-errno);
  }

  // Write our process id to the Lockfile.
  snprintf(buf, sizeof(buf), "%d\n", (int) getpid());

  do {
    err = write(fd, buf, strlen(buf));
  } while ((err < 0) && (errno == EINTR));

  if (err != (int) strlen(buf)) {
    close(fd);
    return (-errno);
  }
  return (1);                   // success
}

void
Lockfile::Close(void)
{
  if (fd != -1) {
    close(fd);
  }
}

//-------------------------------------------------------------------------
// Lockfile::Kill() and Lockfile::KillAll()
//
// Open the lockfile. If we succeed it means there was no process
// holding the lock. We'll just close the file and release the lock
// in that case. If we don't succeed in getting the lock, the
// process id of the process holding the lock is returned. We
// repeatedly send the KILL signal to that process until doing so
// fails. That is, until kill says that the process id is no longer
// valid (we killed the process), or that we don't have permission
// to send a signal to that process id (the process holding the lock
// is dead and a new process has replaced it).
//
// INKqa11325 (Kevlar: linux machine hosed up if specific threads
// killed): Unfortunately, it's possible on Linux that the main PID of
// the process has been successfully killed (and is waiting to be
// reaped while in a defunct state), while some of the other threads
// of the process just don't want to go away.  Integrate ink_killall
// into Kill() and KillAll() just to make sure we really kill
// everything and so that we don't spin hard while trying to kill a
// defunct process.
//-------------------------------------------------------------------------


static void
lockfile_kill_internal(pid_t init_pid, int init_sig, pid_t pid, const char *pname, int sig)
{
  int err;

#if defined(linux)

  pid_t *pidv;
  int pidvcnt;

  // Need to grab pname's pid vector before we issue any kill signals.
  // Specifically, this prevents the race-condition in which
  // traffic_manager spawns a new traffic_server while we still think
  // we're killall'ing the old traffic_server.
  if (pname) {
      //这函数的功能是什么，将程序名为pname的进程都不给杀死，pidv是pid的数组指针，pidvcnt是进程个数
    ink_killall_get_pidv_xmalloc(pname, &pidv, &pidvcnt);
  }

  if (init_sig > 0) {
    kill(init_pid, init_sig);
    // sleep for a bit and give time for the first signal to be
    // delivered
    sleep(1);
  }

  do {
    if ((err = kill(pid, sig)) == 0) {
      sleep(1);
    }
    if (pname && (pidvcnt > 0)) {
      ink_killall_kill_pidv(pidv, pidvcnt, sig);
      sleep(1);
    }
  } while ((err == 0) || ((err < 0) && (errno == EINTR)));

  free(pidv);

#else

  if (init_sig > 0) {
    kill(init_pid, init_sig);
    // sleep for a bit and give time for the first signal to be
    // delivered
    sleep(1);
  }

  do {
    err = kill(pid, sig);
  } while ((err == 0) || ((err < 0) && (errno == EINTR)));

#endif  // linux check

}

/////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////
void
Lockfile::Kill(int sig, int initial_sig, const char *pname)
{
  int err;
  int pid;
  pid_t holding_pid;

  err = Open(&holding_pid);
  if (err == 1)                 // success getting the lock file,说明没有对应的server进程存在
  {
    Close();                    //因此不需要处理，关闭就行了
  } else if (err == 0)          // someone else has the lock
  {
    pid = holding_pid;          //获取持有锁进程的pid
    if (pid != 0) {             //当进程pid有效的时候，就去杀死这个进程
    
      lockfile_kill_internal(pid, initial_sig, pid, pname, sig);
    }
  }
}


/////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////
//没怎么明白这个函数!!
void
Lockfile::KillGroup(int sig, int initial_sig, const char *pname)
{
  int err;
  pid_t pid;
  pid_t holding_pid;

  err = Open(&holding_pid);
  if (err == 1)                 // success getting the lock file
  {
    Close();
  } else if (err == 0)          // someone else has the lock
  {
    do {
      pid = getpgid(holding_pid);//获得进程组识别码
    } while ((pid < 0) && (errno == EINTR));

    if ((pid < 0) || (pid == getpid()))
      pid = holding_pid;
    else
      pid = -pid;

    if (pid != 0) {
      // We kill the holding_pid instead of the process_group
      // initially since there is no point trying to get core files
      // from a group since the core file of one overwrites the core
      // file of another one
      lockfile_kill_internal(holding_pid, initial_sig, pid, pname, sig);
    }
  }
}

3.log.h

#ifndef LOG_H
#define LOG_H
#include <stdio.h>

void write_to_log(char* c){

    FILE* fd;
    fd = fopen("log.txt", "ab"); 
    if (fd)
      {
        fputs(c, fd); 
        fclose(fd);
      }
}

#endif

4.traffic_cop.cpp

#include "lock_and_kill.h"
#include "log.h"
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/sem.h>
#include <signal.h>
#include <sys/param.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <time.h>
#include <string.h>
#include <stdio.h>
#include <sys/stat.h> 


#define    NOWARN_UNUSED(x)    (void)(x)

static char cop_lockfile[PATH_NAME_MAX];
static char manager_lockfile[PATH_NAME_MAX];
static char server_lockfile[PATH_NAME_MAX];

static char manager_binary[PATH_NAME_MAX] = "traffic_manager";
static char server_binary[PATH_NAME_MAX] = "traffic_server";
static int killsig=SIGKILL;
static int coresig=0;
static int server_not_found = 0;
static int server_failures=0;
static int manager_failures =0;

static const int sleep_time = 10;       // 10 sec
static const int manager_timeout = 3 * 60;      //  3 min
static const int server_timeout = 3 * 60;       //  3 min
static const int kill_timeout = 1 * 60; //  1 min


static void sig_alarm_warn(int signum=0)
{
     alarm(kill_timeout);
}


static void sig_fatal(int signum)
{
    abort();
}


static void set_alarm_warn()
{
    struct sigaction action;
    action.sa_handler = sig_alarm_warn;
     sigemptyset(&action.sa_mask);
     action.sa_flags = 0;
    sigaction(SIGALRM, &action, NULL);
}

static void set_alarm_death()
{
    struct sigaction action;
    action.sa_handler = sig_fatal;
      sigemptyset(&action.sa_mask);
      action.sa_flags = 0;
    sigaction(SIGALRM, &action, NULL);
}

static void sig_child(int signum)
{
  NOWARN_UNUSED(signum);
  pid_t pid = 0;
  int status = 0;
  for (;;) {
    pid = waitpid(WAIT_ANY, &status, WNOHANG);

    if (pid <= 0) {
      break;
    }
    // TSqa03086 - We can not log the child status signal from
    //   the signal handler since syslog can deadlock.  Record
    //   the pid and the status in a global for logging
    //   next time through the event loop.  We will occasionally
    //   lose some information if we get two sig childs in rapid
    //   succession
   // child_pid = pid;
    //child_status = status;
  }
}


static void init_signals()
{
      struct sigaction action;
      write_to_log("Entering init_signals()\n");
      action.sa_handler = sig_child;
      sigemptyset(&action.sa_mask);
      action.sa_flags = 0;
      sigaction(SIGCHLD, &action, NULL);
      action.sa_handler = sig_fatal;
      sigemptyset(&action.sa_mask);
      action.sa_flags = 0;
      write_to_log("leaving init_signals()\n\n");
}


static void safe_kill(const char* lockfile_name,const char * pname,bool group)
{
    Lockfile lockfile(lockfile_name);
    write_to_log("Entering safe_kill\n");
    set_alarm_warn();
      alarm(kill_timeout);

      if (group == true) {
        lockfile.KillGroup(killsig, coresig, pname);
      } else {
        lockfile.Kill(killsig, coresig, pname);
      }
      alarm(0);
      set_alarm_death();
     write_to_log("Leaving safe_kill\n\n");

}


//为了简单化，直接返回0
static int server_up()
{
    return 1;

}


static int heartbeat_manager()
{
    //safe_kill(manager_lockfile, manager_binary, true);
    return 1;
}

static int heartbeat_server()
{
    //safe_kill(server_lockfile, server_binary, false);
    //server_failures = 0;
    return 1;
}



static void spawn_manager()
{
      int err;
      int key;
      err = fork();
  write_to_log("Entering spwan_manager()!\n\n");
  if (err == 0) {
    err = execv(manager_binary, NULL);
  write_to_log("somehow execv failed!\n");
    exit(1);
  } else if (err == -1) {
    write_to_log("unable to fork !\n");
    exit(1);
  } 
  
  manager_failures = 0;
  write_to_log("Leaving spwan_manager()!\n\n");
}


static void init_lockfiles()
{
 // Layout::relative_to(cop_lockfile, sizeof(cop_lockfile), Layout::get()->runtimedir, COP_LOCK);
 // Layout::relative_to(manager_lockfile, sizeof(manager_lockfile), Layout::get()->runtimedir,      MANAGER_LOCK);
 // Layout::relative_to(server_lockfile, sizeof(server_lockfile), Layout::get()->runtimedir, SERVER_LOCK);

 write_to_log("Entering init_lockfiles()\n");
 strcpy(cop_lockfile,"cop_lockfile");
 strcpy(manager_lockfile,"manager_lockfile");
 strcpy(server_lockfile,"server_lockfile");

 strcpy(manager_binary,"manager_binary");
 strcpy(server_binary,"server_binary");


 write_to_log("leaving init_lockfiles()\n\n");

 //manager_lockfile="manager_lockfile";
 //server_lockfile="server_lockfile";
 //manager_binary="manager_binary";
 //server_binary="server_binary";

}


static void check_lockfile()
{

  write_to_log("Entering check_lockfile()\n");
  int err;
  pid_t holding_pid;
  Lockfile cop_lf(cop_lockfile);
  err = cop_lf.Get(&holding_pid);


  if (err < 0) {
    write_to_log("leaving check_lockfile(),and err<0\n\n");
    exit(1);
  } else if (err == 0) {
    write_to_log("leaving check_lockfile(),and err==0\n\n");
    exit(1);
  }
    write_to_log("leaving check_lockfile()\n\n");

}



static void check_programs()
{
    int err;
    pid_t holding_pid;

    write_to_log("Entering check_programs()\n");
    printf("Entering check_programs()\n");
  //尝试去获取 manager的lockfile，如果成功，说明没有manager进程在运行
    Lockfile manager_lf(manager_lockfile);
        err = manager_lf.Open(&holding_pid);

   //通过检测err的值来判断manager进程的运行情况
   if(err==0){
        write_to_log("in check_programs(),manager_lockfile,err==0\n");

        printf("in check_programs(),manager_lockfile,err==0\n");
        
        if(kill(holding_pid,0)==-1){
          
           printf("holding_pid is %d,and invalid\n",holding_pid);

                ink_killall(manager_binary, killsig);
                sleep(1);                 // give signals a chance to be received 
                 err = manager_lf.Open(&holding_pid);
            }

   }


    if(err>0){//说明可以获得manager lockfile
        // 'lockfile_open' returns the file descriptor of the opened
        // lockfile.  We need to close this before spawning the
        // manager so that the manager can grab the lock. 
            manager_lf.Close(); 
            // Make sure we don't have a stray traffic server running.

            write_to_log("traffic_manager not running, making sure traffic_server is dead\n");
            safe_kill(server_lockfile,server_binary,false);
            spawn_manager();
    }
    else
    {

            
            

            //err<0,Open中返回负值，说明可能是加锁成功，但是设置lockfile的文件信息失败
            // If there is a manager running we want to heartbeat it to
            // make sure it hasn't wedged. If the manager test succeeds we
            // check to see if the server is up. (That is, it hasn't been
            // brought down via the UI).  If the manager thinks the server
            // is up, we make sure there is actually a server process
            // running. If there is we test it.

                alarm(2*manager_timeout);
                err=heartbeat_manager();//?
                alarm(0);

                if(err<0){//???what case
                    return ;

                    }

                
                if(server_up()<=0){//???what case
                    return;//err>0 ,manager is running ,if server is down  we think manager can create a new server ,so return
                    }

                Lockfile server_lf(server_lockfile);
                err=server_lf.Open(&holding_pid);

                if(err==0){
                    if(kill(holding_pid,0)==-1){
                        ink_killall(server_binary,killsig);
                        sleep(1);// give signals a chance to be received
                        err=server_lf.Open(&holding_pid);
                        }
                    }

                if(err>0){
                    server_lf.Close();
                    server_not_found += 1;

                    if(server_not_found>1){


                        server_not_found=0;
                        safe_kill(manager_lockfile, manager_binary, true);
                        }
                }else{
                          alarm(2 * server_timeout);
                                heartbeat_server();//?
                              alarm(0);

                        }
                
    }
   printf("Leaving check_programs\n\n");
   write_to_log("Leaving check_programs\n\n");
}


static void init()
{    
    write_to_log("Entering init()\n");
    init_signals();
        init_lockfiles();
        check_lockfile();
    write_to_log("Leaving init()\n\n");
}

static void millisleep(int ms)
{
  struct timespec ts;
  ts.tv_sec = ms / 1000;
  ts.tv_nsec = (ms - ts.tv_sec * 1000) * 1000 * 1000;
  nanosleep(&ts, NULL);
}

// Changed function from taking no argument and returning void
// to taking a void* and returning a void*. The change was made
// so that we can call ink_thread_create() on this function
// in the case of running cop as a win32 service.

static void* check(void* arg)
{
    //bool mgmt_init=false;
    write_to_log("Entering check()\n\n");
    for(;;){
        
        // problems with the ownership of this file as root Make sure it is
        // owned by the admin user
        
        alarm(2 * (sleep_time + manager_timeout * 2 + server_timeout));

        check_programs();
        millisleep(sleep_time * 1000);
        }
    write_to_log("Leaveing check()\n\n");
    return arg;
}

void init_daemon(void) 
{ 
    int i; 
    pid_t pid;
    struct rlimit rl;
    struct sigaction sa;
    //printf("------------------------------\n");
    //umask(0);
    if(getrlimit(RLIMIT_NOFILE,&rl)<0){
        exit(1);
    }


    if((pid=fork())<0){
        exit(1);//fork失败，退出 
    }else if(pid> 0){ 
        exit(0);//是父进程，结束父进程 
        }

    //是第一子进程，后台继续执行 
    setsid();//第一子进程成为新的会话组长和进程组长 
    //并与控制终端分离 
    sa.sa_handler=SIG_IGN;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags=0;

    if(sigaction(SIGHUP,&sa,NULL)<0){
        exit(1);
    }

    if((pid=fork())<0){
        exit(1);//fork失败，退出 
    }else if(pid> 0){ 
        exit(0);//是父进程，结束父进程 
        }
    //是第二子进程，继续 
    //第二子进程不再是会话组长 
    umask(0);
    if (rl.rlim_max==RLIM_INFINITY){
        rl.rlim_max=1024;

    }

    for(i=0;i< rl.rlim_max;++i)//关闭打开的文件描述符 
         {         
        close(i);
          } 

    //chdir("/tmp");//改变工作目录到/tmp 
    return; 
} 


int main()
{

    init_daemon();//守护进程初始化函数
      write_to_log("Entering main()\n");
      signal(SIGHUP, SIG_IGN);
      signal(SIGTSTP, SIG_IGN);
      signal(SIGTTOU, SIG_IGN);
      signal(SIGTTIN, SIG_IGN);
      //setsid(); 
      init();
        check(NULL);
      write_to_log("leaving main()\n\n");
       return 0;
}

5.traffic_manager.cpp

#include "lock_and_kill.h"
#include "log.h"
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/sem.h>
#include <signal.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <time.h>
#include <string.h>
#include <stdio.h>

#define    NOWARN_UNUSED(x)    (void)(x)
static char manager_lockfile[4096]="manager_lockfile";
static char server_lockfile[4096]="server_lockfile";
static int server_failures=0;
static int killsig=SIGKILL;
static int coresig=0;
static char server_binary[4096] = "server_binary";
static const int sleep_time = 10;       // 10 sec
static const int manager_timeout = 3 * 60;      //  3 min
static const int server_timeout = 3 * 60;       //  3 min
static const int kill_timeout = 1 * 60; //  1 min

static void sig_alarm_warn(int signum=0)
{
     alarm(kill_timeout);
}


static void sig_fatal(int signum)
{
    abort();
}


static void set_alarm_warn()
{
    struct sigaction action;
    action.sa_handler = sig_alarm_warn;
     sigemptyset(&action.sa_mask);
     action.sa_flags = 0;
    sigaction(SIGALRM, &action, NULL);
}

static void set_alarm_death()
{
    struct sigaction action;
    action.sa_handler = sig_fatal;
      sigemptyset(&action.sa_mask);
      action.sa_flags = 0;
    sigaction(SIGALRM, &action, NULL);
}

static void sig_child(int signum)
{
  NOWARN_UNUSED(signum);
  pid_t pid = 0;
  int status = 0;
  for (;;) {
    pid = waitpid(WAIT_ANY, &status, WNOHANG);

    if (pid <= 0) {
      break;
    }
    // TSqa03086 - We can not log the child status signal from
    //   the signal handler since syslog can deadlock.  Record
    //   the pid and the status in a global for logging
    //   next time through the event loop.  We will occasionally
    //   lose some information if we get two sig childs in rapid
    //   succession
   // child_pid = pid;
    //child_status = status;
  }
}

static void safe_kill(const char* lockfile_name,const char * pname,bool group)
{
    Lockfile lockfile(lockfile_name);
    write_to_log("Entering safe_kill\n");
    set_alarm_warn();
      alarm(kill_timeout);

      if (group == true) {
        lockfile.KillGroup(killsig, coresig, pname);
      } else {
        lockfile.Kill(killsig, coresig, pname);
      }
      alarm(0);
      set_alarm_death();
      write_to_log("Leaving safe_kill\n\n");

}

static void spawn_server()
{
      int err;
      int key;
  write_to_log("--------------Entering spwan_server()!\n\n");
      err = fork();
  if (err == 0) {
    err = execv(server_binary, NULL);
    
    write_to_log("--------------somehow execv failed!\n");
       exit(1);
  } else if (err == -1) {
        write_to_log("--------------unable to fork server !\n");
       exit(1);
  } 
  
  server_failures = 0;
  write_to_log("--------------Leaving spwan_server()!\n\n");
}


void check_server()
{
    int err;
    pid_t holding_pid;
    Lockfile server_lf(server_lockfile);
    err=server_lf.Get(&holding_pid);

    if(err==0){
        if(kill(holding_pid,0)==-1){
            ink_killall(server_binary,killsig);
            sleep(1);
            err=server_lf.Open(&holding_pid);
            }

        }

    if(err>0){
        server_lf.Close();
        safe_kill(server_lockfile,server_binary,false);
        spawn_server();

        }

}




int main()
{
    pid_t holding_pid=0;
    Lockfile manager_lf(manager_lockfile);
    manager_lf.Get(&holding_pid);

    while(1){

        char buf[100];
        sprintf(buf,"----------------traffic_manager is running, pid:'%d'!\n",getpid());
        write_to_log(buf);
        
        printf("----------------traffic_manager is running,pidID: %d\n",getpid());

        sleep(5);
        int c=rand()%10;
        
        if(c==1){//模拟manager进程出现状况
            write_to_log("----------------traffic_manager has a expcetion and eixt!\n");
            exit(1);
        }else{//对server进程进行检查
            check_server();
        }
        }
}

6.traffic_server.cpp

#include "log.h"
#include "lock_and_kill.h"
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>


static char server_lockfile[4096]="server_lockfile";

int main()
{

        pid_t holding_pid=0;
        Lockfile server_lf(server_lockfile);
        server_lf.Get(&holding_pid);

        while(1){

            char buf[100];
        sprintf(buf,"==============traffic_server is running, pid:'%d'!\n",getpid());
        write_to_log(buf);
            sleep(5);
            int c=rand()%100;
            
            if(c<30){//模拟server进程出现状况
                write_to_log("=================traffic_server has a expcetion and exit!\n");
                exit(1);
            }
        }
        return 0;

}

以上文档为以前研究时所写，希望能给感兴趣的同学一点帮助,同时也请大家指点。我这里时简要的分析了traffic进程控制的问题，测试中许多是简化的，比如心跳测试之类的，代码中有说明。