gocrawl 分析

1. gocrawl 类结构

 

// The crawler itself, the master of the whole process
type Crawler struct {
    Options *Options

    // Internal fields
    logFunc         func(LogFlags, string, ...interface{})
    push            chan *workerResponse
    enqueue         chan interface{}
    stop            chan struct{}
    wg              *sync.WaitGroup
    pushPopRefCount int
    visits          int

    // keep lookups in maps, O(1) access time vs O(n) for slice. The empty struct value
    // is of no use, but this is the smallest type possible - it uses no memory at all.
    visited map[string]struct{}
    hosts   map[string]struct{}
    workers map[string]*worker
}

// The Options available to control and customize the crawling process.
type Options struct {
    UserAgent             string
    RobotUserAgent        string
    MaxVisits             int
    EnqueueChanBuffer     int
    HostBufferFactor      int
    CrawlDelay            time.Duration // Applied per host
    WorkerIdleTTL         time.Duration
    SameHostOnly          bool
    HeadBeforeGet         bool
    URLNormalizationFlags purell.NormalizationFlags
    LogFlags              LogFlags
    Extender              Extender
}

// Extension methods required to provide an extender instance.
type Extender interface {
    // Start, End, Error and Log are not related to a specific URL, so they don't
    // receive a URLContext struct.
    Start(interface{}) interface{}
    End(error)
    Error(*CrawlError)
    Log(LogFlags, LogFlags, string)

    // ComputeDelay is related to a Host only, not to a URLContext, although the FetchInfo
    // is related to a URLContext (holds a ctx field).
    ComputeDelay(string, *DelayInfo, *FetchInfo) time.Duration

    // All other extender methods are executed in the context of an URL, and thus
    // receive an URLContext struct as first argument.
    Fetch(*URLContext, string, bool) (*http.Response, error)
    RequestGet(*URLContext, *http.Response) bool
    RequestRobots(*URLContext, string) ([]byte, bool)
    FetchedRobots(*URLContext, *http.Response)
    Filter(*URLContext, bool) bool
    Enqueued(*URLContext)
    Visit(*URLContext, *http.Response, *goquery.Document) (interface{}, bool)
    Visited(*URLContext, interface{})
    Disallowed(*URLContext)
}

entry point:

func main() {
    ext := &Ext{&gocrawl.DefaultExtender{}}
    // Set custom options
    opts := gocrawl.NewOptions(ext)
    opts.CrawlDelay = 1 * time.Second
    opts.LogFlags = gocrawl.LogError
    opts.SameHostOnly = false
    opts.MaxVisits = 10

    c := gocrawl.NewCrawlerWithOptions(opts)
    c.Run("http://0value.com")
}

3 steps:  in main

1) get a Extender

2) create Options with given Extender

3) create gocrawel

as it is commented, go crawel contols the whole process, Option supplies some configuration info and Extender does the real work.

2. other key structs

worker, workResponse and sync.WaitGroup

// Communication from worker to the master crawler, about the crawling of a URL
type workerResponse struct {
    ctx           *URLContext
    visited       bool
    harvestedURLs interface{}
    host          string
    idleDeath     bool
}

// The worker is dedicated to fetching and visiting a given host, respecting
// this host's robots.txt crawling policies.
type worker struct {
    // Worker identification
    host  string
    index int

    // Communication channels and sync
    push    chan<- *workerResponse
    pop     popChannel
    stop    chan struct{}
    enqueue chan<- interface{}
    wg      *sync.WaitGroup

    // Robots validation
    robotsGroup *robotstxt.Group

    // Logging
    logFunc func(LogFlags, string, ...interface{})

    // Implementation fields
    wait           <-chan time.Time
    lastFetch      *FetchInfo
    lastCrawlDelay time.Duration
    opts           *Options
}

for info about sync.WaitGroup, please visit http://mindfsck.net/example-golang-makes-concurrent-programming-easy-awesome/ and http://soniacodes.wordpress.com/2011/02/28/channels-vs-sync-package/

3. I will give a whole workflow of gocrawl in a few days.(6/20/2014)