非阻塞模式下网络读写接口返回值的处理

Cyassl库

https://wolfssl.com/wolfSSL/Docs-wolfssl-manual-17-8-wolfssl-api-connection-session-io.html

int CyaSSL_read(CYASSL* ssl, void* data, int sz);

Return Values:
> 0 - the number of bytes read upon success.
0 - will be returned upon failure.  This may be caused by a either a clean (close notify alert) shutdown or just that the peer closed the connection.  Call CyaSSL_get_error() for the specific error code.
SSL_FATAL_ERROR - will be returned upon failure when either an error occurred or, when using non-blocking sockets, the SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE error was received and and the application needs to call CyaSSL_read() again.  Use CyaSSL_get_error() to get a specific error code.

int CyaSSL_write(CYASSL* ssl, const void* data, int sz);

Return Values:
> 0 - the number of bytes written upon success.
0 - will be returned upon failure.  Call CyaSSL_get_error() for the specific error code.
SSL_FATAL_ERROR - will be returned upon failure when either an error occurred or, when using non-blocking sockets, the SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE error was received and and the application needs to call CyaSSL_write() again.  Use CyaSSL_get_error() to get a specific error code.

Openssl库

SSL_read

RETURN VALUES
       The following return values can occur:

       >0  The read operation was successful; the return value is the number of bytes actually read from the TLS/SSL connection.

       0   The read operation was not successful. The reason may either be a clean shutdown due to a "close notify" alert sent by the peer (in which case the SSL_RECEIVED_SHUTDOWN
           flag in the ssl shutdown state is set (see SSL_shutdown(3), SSL_set_shutdown(3)). It is also possible, that the peer simply shut down the underlying transport and the
           shutdown is incomplete. Call SSL_get_error() with the return value ret to find out, whether an error occurred or the connection was shut down cleanly
           (SSL_ERROR_ZERO_RETURN).

           SSLv2 (deprecated) does not support a shutdown alert protocol, so it can only be detected, whether the underlying connection was closed. It cannot be checked, whether
           the closure was initiated by the peer or by something else.

       <0  The read operation was not successful, because either an error occurred or action must be taken by the calling process. Call SSL_get_error() with the return value ret to
           find out the reason.

WARNING
       When an SSL_read() operation has to be repeated because of SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, it must be repeated with the same arguments.

SSL_write

https://www.openssl.org/docs/manmaster/ssl/SSL_write.html

RETURN VALUES
       The following return values can occur:

       >0  The write operation was successful, the return value is the number of bytes actually written to the TLS/SSL connection.

       0   The write operation was not successful. Probably the underlying connection was closed. Call SSL_get_error() with the return value ret to find out, whether an error
           occurred or the connection was shut down cleanly (SSL_ERROR_ZERO_RETURN).

           SSLv2 (deprecated) does not support a shutdown alert protocol, so it can only be detected, whether the underlying connection was closed. It cannot be checked, why the
           closure happened.

       <0  The write operation was not successful, because either an error occurred or action must be taken by the calling process. Call SSL_get_error() with the return value ret
           to find out the reason.

----------------- 同时考虑 SSL_ERROR_WANT_WRITE SSL_ERROR_WANT_READ ---------

http://stackoverflow.com/questions/5589508/do-i-need-to-handle-ssl-error-want-write-when-ssl-read

The tutorial doc says I need to handle SSL_ERROR_WANT_WRITE(plus SSL_ERROR_WANT_READ of course) on SSL_read because SSL rehandshaking could take place anytime.
And SSL_ERROR_WANT_READ on SSL_write for the same reason.

Rehandshaking can be triggered by either side at any point during the connection. It doesn't really have anything directly to do with session resumption.

So yes, if you want your application to be reliable, you should be prepared to handle both SSL_WANT_WRITE and SSL_WANT_READ no matter whether you are currently reading or writing.

-------

http://jmarshall.com/stuff/handling-nbio-errors-in-openssl.html

Handling Non-Blocking I/O Errors in OpenSSL

SSL_accept()

After result of:	Select on:	Try this next:
<0, SSL_ERROR_WANT_READ	R	SSL_accept()
<0, SSL_ERROR_WANT_WRITE	W	SSL_accept()
0, SSL_ERROR_SYSCALL		nothing (illegal EOF)
<0, SSL_ERROR_SYSCALL		see errno

SSL_connect()

After result of:	Select on:	Try this next:
<0, SSL_ERROR_WANT_READ	R	SSL_connect()
<0, SSL_ERROR_WANT_WRITE	W	SSL_connect()
0, SSL_ERROR_SYSCALL		nothing (illegal EOF)
<0, SSL_ERROR_SYSCALL		see errno

SSL_read()

After result of:	Select on:	Try this next:
<0, SSL_ERROR_WANT_READ	R	SSL_read()
<0, SSL_ERROR_WANT_WRITE	W	SSL_read()
0, SSL_ERROR_ZERO_RETURN		nothing (socket is shut down)

SSL_write()

After result of:	Select on:	Try this next:
<0, SSL_ERROR_WANT_READ	R	SSL_write()
<0, SSL_ERROR_WANT_WRITE	W	SSL_write()
0, SSL_ERROR_ZERO_RETURN		nothing (socket is shut down)

SSL_shutdown()

After result of:	Select on:	Try this next:
-1, SSL_ERROR_WANT_READ	R	SSL_shutdown()
-1, SSL_ERROR_WANT_WRITE	W	SSL_shutdown()
0	R	SSL_shutdown()

参考代码

http://stackoverflow.com/questions/31171396/openssl-non-blocking-socket-ssl-read-unpredictable

char *sslRead (connection *c)
{
    const int readSize = 1024;
    char *rc = NULL;
    int received, count = 0;
    int TotalReceived = 0;
    fd_set fds;
    struct timeval timeout;
    char buffer[1024];

    if (c)
    {
        while (1)
        {
            received = SSL_read (c->sslHandle, buffer, readSize);
            if (received > 0)
            {
                TotalReceived += received;
                printf("Buffsize - %i - %.*s 
", received, received, buffer);
            }
            else
            {
                count++;

                //printf(" received equal to or less than 0
")
                int err = SSL_get_error(c->sslHandle, received);
                switch (err)
                {
                    case SSL_ERROR_NONE:
                    {
                        // no real error, just try again...
                        printf("SSL_ERROR_NONE %i
", count);
                        continue;
                    }   

                    case SSL_ERROR_ZERO_RETURN: 
                    {
                        // peer disconnected...
                        printf("SSL_ERROR_ZERO_RETURN %i
", count);
                        break;
                    }   

                    case SSL_ERROR_WANT_READ: 
                    {
                        // no data available right now, wait a few seconds in case new data arrives...
                        printf("SSL_ERROR_WANT_READ %i
", count);

                        int sock = SSL_get_rfd(c->sslHandle);
                        FD_ZERO(&fds);
                        FD_SET(sock, &fds);

                        timeout.tv_sec = 5;
                        timeou.tv_nsec = 0;

                        err = select(sock+1, &fds, NULL, NULL, &timeout);
                        if (err > 0)
                            continue; // more data to read...

                        if (err == 0) {
                            // timeout...
                        } else {
                            // error...
                        }

                        break;
                    }

                    case SSL_ERROR_WANT_WRITE: 
                    {
                        // socket not writable right now, wait a few seconds and try again...
                        printf("SSL_ERROR_WANT_WRITE %i
", count);

                        int sock = SSL_get_wfd(c->sslHandle);
                        FD_ZERO(&fds);
                        FD_SET(sock, &fds);

                        timeout.tv_sec = 5;
                        timeou.tv_nsec = 0;

                        err = select(sock+1, NULL, &fds, NULL, &timeout);
                        if (err > 0)
                            continue; // can write more data now...

                        if (err == 0) {
                            // timeout...
                        } else {
                            // error...
                        }

                        break;
                    }

                    default:
                    {
                        printf("error %i:%i
", received, err); 
                        break;
                    }
                }     

                break;
            }
        }
    }

    return rc;
}

ngx_ssl_write

ssize_t
ngx_ssl_write(ngx_connection_t *c, u_char *data, size_t size)
{
    int        n, sslerr;
    ngx_err_t  err;

    ngx_ssl_clear_error(c->log);

    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL to write: %uz", size);

    n = SSL_write(c->ssl->connection, data, size);

    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_write: %d", n);

    if (n > 0) {

        if (c->ssl->saved_read_handler) {

            c->read->handler = c->ssl->saved_read_handler;
            c->ssl->saved_read_handler = NULL;
            c->read->ready = 1;

            if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
                return NGX_ERROR;
            }

            ngx_post_event(c->read, &ngx_posted_events);
        }

        c->sent += n;

        return n;
    }

    sslerr = SSL_get_error(c->ssl->connection, n);

    err = (sslerr == SSL_ERROR_SYSCALL) ? ngx_errno : 0;

    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_get_error: %d", sslerr);

    if (sslerr == SSL_ERROR_WANT_WRITE) {
        c->write->ready = 0;
        return NGX_AGAIN;
    }

    if (sslerr == SSL_ERROR_WANT_READ) {

        ngx_log_error(NGX_LOG_INFO, c->log, 0,
                      "peer started SSL renegotiation");

        c->read->ready = 0;

        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
            return NGX_ERROR;
        }

        /*
         * we do not set the timer because there is already
         * the write event timer
         */

        if (c->ssl->saved_read_handler == NULL) {
            c->ssl->saved_read_handler = c->read->handler;
            c->read->handler = ngx_ssl_read_handler;
        }

        return NGX_AGAIN;
    }

    c->ssl->no_wait_shutdown = 1;
    c->ssl->no_send_shutdown = 1;
    c->write->error = 1;

    ngx_ssl_connection_error(c, sslerr, err, "SSL_write() failed");

    return NGX_ERROR;
}

ngx_ssl_recv

ssize_t
ngx_ssl_recv(ngx_connection_t *c, u_char *buf, size_t size)
{
    int  n, bytes;

    if (c->ssl->last == NGX_ERROR) {
        c->read->error = 1;
        return NGX_ERROR;
    }

    if (c->ssl->last == NGX_DONE) {
        c->read->ready = 0;
        c->read->eof = 1;
        return 0;
    }

    bytes = 0;

    ngx_ssl_clear_error(c->log);

    /*
     * SSL_read() may return data in parts, so try to read
     * until SSL_read() would return no data
     */

    for ( ;; ) {

        n = SSL_read(c->ssl->connection, buf, size);

        ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_read: %d", n);

        if (n > 0) {
            bytes += n;
        }

        c->ssl->last = ngx_ssl_handle_recv(c, n);

        if (c->ssl->last == NGX_OK) {

            size -= n;

            if (size == 0) {
                c->read->ready = 1;
                return bytes;
            }

            buf += n;

            continue;
        }

        if (bytes) {
            if (c->ssl->last != NGX_AGAIN) {
                c->read->ready = 1;
            }

            return bytes;
        }

        switch (c->ssl->last) {

        case NGX_DONE:
            c->read->ready = 0;
            c->read->eof = 1;
            return 0;

        case NGX_ERROR:
            c->read->error = 1;

            /* fall through */

        case NGX_AGAIN:
            return c->ssl->last;
        }
    }
}

ngx_ssl_handle_recv

static ngx_int_t
ngx_ssl_handle_recv(ngx_connection_t *c, int n)
{
    int        sslerr;
    ngx_err_t  err;

    if (c->ssl->renegotiation) {
        /*
         * disable renegotiation (CVE-2009-3555):
         * OpenSSL (at least up to 0.9.8l) does not handle disabled
         * renegotiation gracefully, so drop connection here
         */

        ngx_log_error(NGX_LOG_NOTICE, c->log, 0, "SSL renegotiation disabled");

        while (ERR_peek_error()) {
            ngx_ssl_error(NGX_LOG_DEBUG, c->log, 0,
                          "ignoring stale global SSL error");
        }

        ERR_clear_error();

        c->ssl->no_wait_shutdown = 1;
        c->ssl->no_send_shutdown = 1;

        return NGX_ERROR;
    }

    if (n > 0) {

        if (c->ssl->saved_write_handler) {

            c->write->handler = c->ssl->saved_write_handler;
            c->ssl->saved_write_handler = NULL;
            c->write->ready = 1;

            if (ngx_handle_write_event(c->write, 0) != NGX_OK) {
                return NGX_ERROR;
            }

            ngx_post_event(c->write, &ngx_posted_events);
        }

        return NGX_OK;
    }

    sslerr = SSL_get_error(c->ssl->connection, n);

    err = (sslerr == SSL_ERROR_SYSCALL) ? ngx_errno : 0;

    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_get_error: %d", sslerr);

    if (sslerr == SSL_ERROR_WANT_READ) {
        c->read->ready = 0;
        return NGX_AGAIN;
    }

    if (sslerr == SSL_ERROR_WANT_WRITE) {

        ngx_log_error(NGX_LOG_INFO, c->log, 0,
                      "peer started SSL renegotiation");

        c->write->ready = 0;

        if (ngx_handle_write_event(c->write, 0) != NGX_OK) {
            return NGX_ERROR;
        }

        /*
         * we do not set the timer because there is already the read event timer
         */

        if (c->ssl->saved_write_handler == NULL) {
            c->ssl->saved_write_handler = c->write->handler;
            c->write->handler = ngx_ssl_write_handler;
        }

        return NGX_AGAIN;
    }

    c->ssl->no_wait_shutdown = 1;
    c->ssl->no_send_shutdown = 1;

    if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0) {
        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
                       "peer shutdown SSL cleanly");
        return NGX_DONE;
    }

    ngx_ssl_connection_error(c, sslerr, err, "SSL_read() failed");

    return NGX_ERROR;
}

非加密

recv

RETURN VALUE
       These calls return the number of bytes received, or -1 if an error occurred. The return value will be 0 when the peer has performed an orderly shutdown.

http://www.cnblogs.com/tzhangofseu/archive/2011/12/09/2282466.html

recv返回值

返回值为n

1.n>0，正常读取n个字节

2.n==0，socket对方节点正常shutdown

3.n==-1，未能正常读取数据，有以下几种情况，errno被置为以下几种：

EAGAIN or EWOULDBLOCK
The socket is marked nonblocking and the receive operation would block, or a receive timeout had been set and the timeout expired before data was received.

这种情况要注意，前者是非阻塞方式下的一种正常返回，后者是超时的返回情况。

ERRORS
       These are some standard errors generated by the socket layer. Additional errors may be generated and returned from the underlying protocol modules; see their manual pages.

       EAGAIN The socket is marked non-blocking and the receive operation would block, or a receive timeout had been set and the timeout expired before data was received.

       EBADF  The argument s is an invalid descriptor.

       ECONNREFUSED
              A remote host refused to allow the network connection (typically because it is not running the requested service).

       EFAULT The receive buffer pointer(s) point outside the process’s address space.

       EINTR  The receive was interrupted by delivery of a signal before any data were available.

       EINVAL Invalid argument passed.

       ENOMEM Could not allocate memory for recvmsg().

       ENOTCONN
              The socket is associated with a connection-oriented protocol and has not been connected (see connect(2) and accept(2)).

       ENOTSOCK
              The argument s does not refer to a socket.

 

send

RETURN VALUE
       On success, these calls return the number of characters sent.  On error, -1 is returned, and errno is set appropriately.

ERRORS
       These are some standard errors generated by the socket layer. Additional errors may be generated and returned from the underlying protocol modules; see their respective man-
       ual pages.

       EACCES (For Unix domain sockets, which are identified by pathname) Write permission is denied on the destination socket file, or search permission is denied for one  of  the
              directories the path prefix. (See path_resolution(2).)

       EAGAIN or EWOULDBLOCK
              The socket is marked non-blocking and the requested operation would block.

       EBADF  An invalid descriptor was specified.

       ECONNRESET
              Connection reset by peer.

       EDESTADDRREQ
              The socket is not connection-mode, and no peer address is set.

       EFAULT An invalid user space address was specified for a parameter.

       EINTR  A signal occurred before any data was transmitted.

       EINVAL Invalid argument passed.

       EISCONN
              The connection-mode socket was connected already but a recipient was specified.  (Now either this error is returned, or the recipient specification is ignored.)

       EMSGSIZE
              The socket type requires that message be sent atomically, and the size of the message to be sent made this impossible.

       ENOBUFS
              The output queue for a network interface was full.  This generally indicates that the interface has stopped sending, but may be caused by transient congestion.  (Nor-
              mally, this does not occur in Linux. Packets are just silently dropped when a device queue overflows.)

       ENOMEM No memory available.

       ENOTCONN
              The socket is not connected, and no target has been given.

       ENOTSOCK
              The argument s is not a socket.

       EOPNOTSUPP
              Some bit in the flags argument is inappropriate for the socket type.

       EPIPE  The local end has been shut down on a connection oriented socket.  In this case the process will also receive a SIGPIPE unless MSG_NOSIGNAL is set.

write

http://man7.org/linux/man-pages/man2/write.2.html

       #include <unistd.h>

       ssize_t write(int fd, const void *buf, size_t count);

RETURN VALUE         top

       On success, the number of bytes written is returned (zero indicates
       nothing was written).  It is not an error if this number is smaller
       than the number of bytes requested; this may happen for example
       because the disk device was filled.  See also NOTES.

       On error, -1 is returned, and errno is set appropriately.

       If count is zero and fd refers to a regular file, then write() may
       return a failure status if one of the errors below is detected.  If
       no errors are detected, or error detection is not performed, 0 will
       be returned without causing any other effect.  If count is zero and
       fd refers to a file other than a regular file, the results are not
       specified.

read

http://man7.org/linux/man-pages/man2/read.2.html

       #include <unistd.h>

       ssize_t read(int fd, void *buf, size_t count);

RETURN VALUE         top

       On success, the number of bytes read is returned (zero indicates end
       of file), and the file position is advanced by this number.  It is
       not an error if this number is smaller than the number of bytes
       requested; this may happen for example because fewer bytes are
       actually available right now (maybe because we were close to end-of-
       file, or because we are reading from a pipe, or from a terminal), or
       because read() was interrupted by a signal.  See also NOTES.

       On error, -1 is returned, and errno is set appropriately.  In this
       case, it is left unspecified whether the file position (if any)
       changes.

recv 和 send 与 read 和 write 关系

http://www.cnblogs.com/heiyue/archive/2011/08/03/2126622.html

1、recv和send
　　recv和send函数提供了和read和write差不多的功能.但是他们提供了第四个参数来控制读写操作。
int recv(int sockfd,void *buf,int len,int flags)
int send(int sockfd,void *buf,int len,int flags)
前面的三个参数和read,write相同,第四个参数能够是0或是以下的组合
_______________________________________________________________
 MSG_DONTROUTE：不查找路由表
 MSG_OOB：接受或发送带外数据
 MSG_PEEK：查看数据,并不从系统缓冲区移走数据
 MSG_WAITALL ：等待任何数据
————————————————————–
MSG_DONTROUTE：是send函数使用的标志.这个标志告诉IP协议.目的主机在本地网络上面,没有必要查找路由表.这个标志一般用网络诊断和路由程式里面。
MSG_OOB：表示能够接收和发送带外的数据.关于带外数据我们以后会解释的.
MSG_PEEK：是recv函数的使用标志，表示只是从系统缓冲区中读取内容，而不清除系统缓冲区的内容。这样下次读的时候,仍然是相同的内容。一般在有多个进程读写数据时能够使用这个标志。
MSG_WAITALL：是recv函数的使用标志，表示等到任何的信息到达时才返回。使用这个标志的时候recv会一直阻塞，直到指定的条件满足，或是发生了错误。 1)当读到了指定的字节时，函数正常返回，返回值等于len 2)当读到了文档的结尾时,函数正常返回.返回值小于len 3)当操作发生错误时,返回-1,且配置错误为相应的错误号(errno)
　　假如flags为0，则和read,write相同的操作，更有其他的几个选项,但是我们实际上用的很少,能够查看 Linux Programmer’s Manual得到周详解释。

参考代码

thttpd--- read -----

-------------------------handle_read

    /* Read some more bytes. */
    sz = read(
    hc->conn_fd, &(hc->read_buf[hc->read_idx]),
    hc->read_size - hc->read_idx );
    if ( sz == 0 )  // 读报文头读不到
    {
    httpd_send_err( hc, 400, httpd_err400title, "", httpd_err400form, "" );
    finish_connection( c, tvP );
    return;
    }
    if ( sz < 0 )
    {
    /* Ignore EINTR and EAGAIN.  Also ignore EWOULDBLOCK.  At first glance
    ** you would think that connections returned by fdwatch as readable
    ** should never give an EWOULDBLOCK; however, this apparently can
    ** happen if a packet gets garbled.
    */
    if ( errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK )
        return;
    httpd_send_err(
        hc, 400, httpd_err400title, "", httpd_err400form, "" );
    finish_connection( c, tvP );
    return;
    }

------------------------- interpose_input

/* This routine is used only for POST requests.  It reads the data
** from the request and sends it to the child process.  The only reason
** we need to do it this way instead of just letting the child read
** directly is that we have already read part of the data into our
** buffer.
*/
static void
cgi_interpose_input( httpd_conn* hc, int wfd )
    {
    size_t c;
    ssize_t r;
    char buf[1024];

    c = hc->read_idx - hc->checked_idx;
    if ( c > 0 )
    {
    if ( httpd_write_fully( wfd, &(hc->read_buf[hc->checked_idx]), c ) != c )
        return;
    }
    while ( c < hc->contentlength )
    {
    r = read( hc->conn_fd, buf, MIN( sizeof(buf), hc->contentlength - c ) );
    if ( r < 0 && ( errno == EINTR || errno == EAGAIN ) )
        {
        sleep( 1 );
        continue;
        }
    if ( r <= 0 )
        return;
    if ( httpd_write_fully( wfd, buf, r ) != r )
        return;
    c += r;
    }
    post_post_garbage_hack( hc );
    }

thttpd--- write -----

handle_send

    /* Do we need to write the headers first? */
    if ( hc->responselen == 0 )
    {
    /* No, just write the file. */
    sz = write(
        hc->conn_fd, &(hc->file_address[c->next_byte_index]),
        MIN( c->end_byte_index - c->next_byte_index, max_bytes ) );
    }
    else
    {
    /* Yes.  We'll combine headers and file into a single writev(),
    ** hoping that this generates a single packet.
    */
    struct iovec iv[2];

    iv[0].iov_base = hc->response;
    iv[0].iov_len = hc->responselen;
    iv[1].iov_base = &(hc->file_address[c->next_byte_index]);
    iv[1].iov_len = MIN( c->end_byte_index - c->next_byte_index, max_bytes );
    sz = writev( hc->conn_fd, iv, 2 );
    }

    if ( sz < 0 && errno == EINTR )
    return;

    if ( sz == 0 ||
     ( sz < 0 && ( errno == EWOULDBLOCK || errno == EAGAIN ) ) )
    {
    /* This shouldn't happen, but some kernels, e.g.
    ** SunOS 4.1.x, are broken and select() says that
    ** O_NDELAY sockets are always writable even when
    ** they're actually not.
    **
    ** Current workaround is to block sending on this
    ** socket for a brief adaptively-tuned period.
    ** Fortunately we already have all the necessary
    ** blocking code, for use with throttling.
    */
    c->wouldblock_delay += MIN_WOULDBLOCK_DELAY;
    c->conn_state = CNST_PAUSING;
    fdwatch_del_fd( hc->conn_fd );
    client_data.p = c;
    if ( c->wakeup_timer != (Timer*) 0 )
        syslog( LOG_ERR, "replacing non-null wakeup_timer!" );
    c->wakeup_timer = tmr_create(
        tvP, wakeup_connection, client_data, c->wouldblock_delay, 0 );
    if ( c->wakeup_timer == (Timer*) 0 )
        {
        syslog( LOG_CRIT, "tmr_create(wakeup_connection) failed" );
        exit( 1 );
        }
    return;
    }

    if ( sz < 0 )
    {
    /* Something went wrong, close this connection.
    **
    ** If it's just an EPIPE, don't bother logging, that
    ** just means the client hung up on us.
    **
    ** On some systems, write() occasionally gives an EINVAL.
    ** Dunno why, something to do with the socket going
    ** bad.  Anyway, we don't log those either.
    **
    ** And ECONNRESET isn't interesting either.
    */
    if ( errno != EPIPE && errno != EINVAL && errno != ECONNRESET )
        syslog( LOG_ERR, "write - %m sending %.80s", hc->encodedurl );
    clear_connection( c, tvP );
    return;
    }

http_write_fully

/* Write the requested buffer completely, accounting for interruptions. */
int
httpd_write_fully( int fd, const char* buf, size_t nbytes )
    {
    int nwritten;

    nwritten = 0;
    while ( nwritten < nbytes )
    {
    int r;

    r = write( fd, buf + nwritten, nbytes - nwritten );
    if ( r < 0 && ( errno == EINTR || errno == EAGAIN ) )
        {
        sleep( 1 );
        continue;
        }
    if ( r < 0 )
        return r;
    if ( r == 0 )
        break;
    nwritten += r;
    }

    return nwritten;
    }

------------ IBM recv send -----------------

https://www.ibm.com/support/knowledgecenter/ssw_i5_54/rzab6/xnonblock.htm

               printf("  Descriptor %d is readable
", i);
               close_conn = FALSE;
               /*************************************************/
               /* Receive all incoming data on this socket      */
               /* before we loop back and call select again.    */
               /*************************************************/
               do
               {
                  /**********************************************/
                  /* Receive data on this connection until the  */
                  /* recv fails with EWOULDBLOCK.  If any other */
                  /* failure occurs, we will close the          */
                  /* connection.                                */
                  /**********************************************/
                  rc = recv(i, buffer, sizeof(buffer), 0);
                  if (rc < 0)
                  {
                     if (errno != EWOULDBLOCK)
                     {
                        perror("  recv() failed");
                        close_conn = TRUE;
                     }
                     break;
                  }

                  /**********************************************/
                  /* Check to see if the connection has been    */
                  /* closed by the client                       */
                  /**********************************************/
                  if (rc == 0)
                  {
                     printf("  Connection closed
");
                     close_conn = TRUE;
                     break;
                  }

                  /**********************************************/
                  /* Data was received                          */
                  /**********************************************/
                  len = rc;
                  printf("  %d bytes received
", len);

                  /**********************************************/
                  /* Echo the data back to the client           */
                  /**********************************************/
                  rc = send(i, buffer, len, 0);
                  if (rc < 0)
                  {
                     perror("  send() failed");
                     close_conn = TRUE;
                     break;
                  }

               } while (TRUE);

------  ngx_http_upstream_check_broken_connection

    n = recv(c->fd, buf, 1, MSG_PEEK);

    err = ngx_socket_errno;

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, ev->log, err,
                   "http upstream recv(): %d", n);

    if (ev->write && (n >= 0 || err == NGX_EAGAIN)) {
        return;
    }

    if ((ngx_event_flags & NGX_USE_LEVEL_EVENT) && ev->active) {

        event = ev->write ? NGX_WRITE_EVENT : NGX_READ_EVENT;

        if (ngx_del_event(ev, event, 0) != NGX_OK) {
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
            return;
        }
    }

    if (n > 0) {
        return;
    }

    if (n == -1) {
        if (err == NGX_EAGAIN) {
            return;
        }

        ev->error = 1;

    } else { /* n == 0 */
        err = 0;
    }

--------ngx_unix_recv ----------------

    do {
        n = recv(c->fd, buf, size, 0);

        ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0,
                       "recv: fd:%d %z of %uz", c->fd, n, size);

        if (n == 0) {
            rev->ready = 0;
            rev->eof = 1;

#if (NGX_HAVE_KQUEUE)

            /*
             * on FreeBSD recv() may return 0 on closed socket
             * even if kqueue reported about available data
             */

            if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) {
                rev->available = 0;
            }

#endif

            return 0;
        }

        if (n > 0) {

#if (NGX_HAVE_KQUEUE)

            if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) {
                rev->available -= n;

                /*
                 * rev->available may be negative here because some additional
                 * bytes may be received between kevent() and recv()
                 */

                if (rev->available <= 0) {
                    if (!rev->pending_eof) {
                        rev->ready = 0;
                    }

                    rev->available = 0;
                }

                return n;
            }

#endif

            if ((size_t) n < size
                && !(ngx_event_flags & NGX_USE_GREEDY_EVENT))
            {
                rev->ready = 0;
            }

            return n;
        }

        err = ngx_socket_errno;

        if (err == NGX_EAGAIN || err == NGX_EINTR) {
            ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, err,
                           "recv() not ready");
            n = NGX_AGAIN;

        } else {
            n = ngx_connection_error(c, err, "recv() failed");
            break;
        }

    } while (err == NGX_EINTR);

    rev->ready = 0;

    if (n == NGX_ERROR) {
        rev->error = 1;
    }

------------ ngx_unix_send

    for ( ;; ) {
        n = send(c->fd, buf, size, 0);

        ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0,
                       "send: fd:%d %z of %uz", c->fd, n, size);

        if (n > 0) {
            if (n < (ssize_t) size) {
                wev->ready = 0;
            }

            c->sent += n;

            return n;
        }

        err = ngx_socket_errno;

        if (n == 0) {
            ngx_log_error(NGX_LOG_ALERT, c->log, err, "send() returned zero");
            wev->ready = 0;
            return n;
        }

        if (err == NGX_EAGAIN || err == NGX_EINTR) {
            wev->ready = 0;

            ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, err,
                           "send() not ready");

            if (err == NGX_EAGAIN) {
                return NGX_AGAIN;
            }

        } else {
            wev->error = 1;
            (void) ngx_connection_error(c, err, "send() failed");
            return NGX_ERROR;
        }
    }