Heritrix 3.1.0 源码解析（十七）

我们接下来分析与与BdbFrontier对象void finished(CrawlURI cURI)方法相关的方法 

/**
     * Note that the previously emitted CrawlURI has completed
     * its processing (for now).
     *
     * The CrawlURI may be scheduled to retry, if appropriate,
     * and other related URIs may become eligible for release
     * via the next next() call, as a result of finished().
     *
     * TODO: make as many decisions about what happens to the CrawlURI
     * (success, failure, retry) and queue (retire, snooze, ready) as 
     * possible elsewhere, such as in DispositionProcessor. Then, break
     * this into simple branches or focused methods for each case. 
     *  
     * @see org.archive.crawler.framework.Frontier#finished(org.archive.modules.CrawlURI)
     */
    protected void processFinish(CrawlURI curi) {
//        assert Thread.currentThread() == managerThread;        
        long now = System.currentTimeMillis();
        //尝试次数
        curi.incrementFetchAttempts();
        logNonfatalErrors(curi);
        
        WorkQueue wq = (WorkQueue) curi.getHolder();
        // always refresh budgeting values from current curi
        // (whose overlay settings should be active here)
        wq.setSessionBudget(getBalanceReplenishAmount());
        wq.setTotalBudget(getQueueTotalBudget());
        
        assert (wq.peek(this) == curi) : "unexpected peek " + wq;

        int holderCost = curi.getHolderCost();
        //是否需要重新处理
        if (needsReenqueuing(curi)) {
            // codes/errors which don't consume the URI, leaving it atop queue
            if(curi.getFetchStatus()!=S_DEFERRED) {
                wq.expend(holderCost); // all retries but DEFERRED cost
            }
            //延时时间
            long delay_ms = retryDelayFor(curi) * 1000;
            curi.processingCleanup(); // lose state that shouldn't burden retry
            wq.unpeek(curi);
            //更新到WorkQueue wq
            wq.update(this, curi); // rewrite any changes
            //重新归队
            handleQueue(wq,curi.includesRetireDirective(),now,delay_ms);
            appCtx.publishEvent(new CrawlURIDispositionEvent(this,curi,DEFERRED_FOR_RETRY));
            doJournalReenqueued(curi);
            wq.makeDirty();
            return; // no further dequeueing, logging, rescheduling to occur
        }

        // Curi will definitely be disposed of without retry, so remove from queue
        //从WorkQueue wq中移除该CrawlURI curi对象
        wq.dequeue(this,curi);
        decrementQueuedCount(1);
        largestQueues.update(wq.getClassKey(), wq.getCount());
        log(curi);
        
        if (curi.isSuccess()) {
            // codes deemed 'success' 
            incrementSucceededFetchCount();
            totalProcessedBytes.addAndGet(curi.getRecordedSize());
            appCtx.publishEvent(new CrawlURIDispositionEvent(this,curi,SUCCEEDED));
            doJournalFinishedSuccess(curi);
           
        } else if (isDisregarded(curi)) {
            // codes meaning 'undo' (even though URI was enqueued, 
            // we now want to disregard it from normal success/failure tallies)
            // (eg robots-excluded, operator-changed-scope, etc)
            incrementDisregardedUriCount();
            appCtx.publishEvent(new CrawlURIDispositionEvent(this,curi,DISREGARDED));
            holderCost = 0; // no charge for disregarded URIs
            // TODO: consider reinstating forget-URI capability, so URI could be
            // re-enqueued if discovered again
            doJournalDisregarded(curi);
            
        } else {
            // codes meaning 'failure'
            incrementFailedFetchCount();
            appCtx.publishEvent(new CrawlURIDispositionEvent(this,curi,FAILED));
            // if exception, also send to crawlErrors
            if (curi.getFetchStatus() == S_RUNTIME_EXCEPTION) {
                Object[] array = { curi };
                loggerModule.getRuntimeErrors().log(Level.WARNING, curi.getUURI()
                        .toString(), array);
            }        
            // charge queue any extra error penalty
            wq.noteError(getErrorPenaltyAmount());
            doJournalFinishedFailure(curi);
            
        }

        wq.expend(holderCost); // successes & failures charge cost to queue
        //延时时间
        long delay_ms = curi.getPolitenessDelay();
        //long delay_ms = 0;
        //重新归队
        handleQueue(wq,curi.includesRetireDirective(),now,delay_ms);
        wq.makeDirty();
        
        if(curi.getRescheduleTime()>0) {
            // marked up for forced-revisit at a set time
            curi.processingCleanup();
            curi.resetForRescheduling(); 
            futureUris.put(curi.getRescheduleTime(),curi);
            futureUriCount.incrementAndGet(); 
        } else {
            curi.stripToMinimal();
            curi.processingCleanup();
        }
    }

首先判断CrawlURI curi对象是否需要重新放入队列，方法如下

/**
     * Checks if a recently processed CrawlURI that did not finish successfully
     * needs to be reenqueued (and thus possibly, processed again after some 
     * time elapses)
     * 
     * @param curi
     *            The CrawlURI to check
     * @return True if we need to retry.
     */
    protected boolean needsReenqueuing(CrawlURI curi) {
        //是否超过最大的尝试次数，默认为30次
        if (overMaxRetries(curi)) {
            return false;
        }
        //根据状态判断
        switch (curi.getFetchStatus()) {
        case HttpStatus.SC_UNAUTHORIZED:
            // We can get here though usually a positive status code is
            // a success. We get here if there is rfc2617 credential data
            // loaded and we're supposed to go around again. See if any
            // rfc2617 credential present and if there, assume it got
            // loaded in FetchHTTP on expectation that we're to go around
            // again. If no rfc2617 loaded, we should not be here.
            boolean loaded = curi.hasRfc2617Credential();
            if (!loaded && logger.isLoggable(Level.FINE)) {
                logger.fine("Have 401 but no creds loaded " + curi);
            }
            return loaded;
        case S_DEFERRED:
        case S_CONNECT_FAILED:
        case S_CONNECT_LOST:
        case S_DOMAIN_UNRESOLVABLE:
            // these are all worth a retry
            // TODO: consider if any others (S_TIMEOUT in some cases?) deserve
            // retry
            return true;
        case S_UNATTEMPTED:
            if(curi.includesRetireDirective()) {
                return true;
            } // otherwise, fall-through: no status is an error without queue-directive
        default:
            return false;
        }
    }

 long retryDelayFor(CrawlURI curi)方法为设置WorkQueue wq延时时间 

/**
     * Return a suitable value to wait before retrying the given URI.
     * 
     * @param curi
     *            CrawlURI to be retried
     * @return millisecond delay before retry
     */
    protected long retryDelayFor(CrawlURI curi) {
        int status = curi.getFetchStatus();
        return (status == S_CONNECT_FAILED || status == S_CONNECT_LOST ||
                status == S_DOMAIN_UNRESOLVABLE)? getRetryDelaySeconds() : 0;
                // no delay for most
    }

getRetryDelaySeconds()的值默认为900秒（15分）

后面为将CrawlURI curi对象更新到WorkQueue wq，最后 重置WorkQueue wq的队列归属（放入不再激活的队列或休眠队列或reenqueueQueue(wq)进一步处理）

/**
     * 重置WorkQueue wq的队列归属
     * Send an active queue to its next state, based on the supplied 
     * parameters.
     * 
     * @param wq
     * @param forceRetire
     * @param now
     * @param delay_ms
     */
    protected void handleQueue(WorkQueue wq, boolean forceRetire, long now, long delay_ms) {
        
        inProcessQueues.remove(wq);
        if(forceRetire) {
            retireQueue(wq);
        } else if (delay_ms > 0) {
            snoozeQueue(wq, now, delay_ms);
        } else {
            //Enqueue the given queue to either readyClassQueues or inactiveQueues,as appropriate
            reenqueueQueue(wq);
        }
    }

接下来看后面的方法wq.dequeue(this,curi)为将CrawlURI curi对象从WorkQueue wq中移除

最后重置WorkQueue wq的队列归属

 long delay_ms = curi.getPolitenessDelay(); 
 handleQueue(wq,curi.includesRetireDirective(),now,delay_ms);

handleQueue方法在上面部分

---------------------------------------------------------------------------

本系列Heritrix 3.1.0 源码解析系本人原创

转载请注明出处 博客园 刺猬的温驯

本文链接 http://www.cnblogs.com/chenying99/archive/2013/04/21/3033520.html