Scrapy - Stats Collection

Description

Stats Collector is a facipty provided by Scrapy to collect the stats in the form of key/values and it is accessed using the Crawler API (Crawler provides access to all Scrapy core components). The stats collector provides one stats table per spider in which the stats collector opens automatically when spider is opening and closes the stats collector when spider is closed.

Common Stats Collector Uses

The following code accesses the stats collector using stats attribute.

class ExtensionThatAccessStats(object): 
   def __init__(self, stats): 
      self.stats = stats  
   
   @classmethod 
   def from_crawler(cls, crawler): 
      return cls(crawler.stats)

The following table shows various options can be used with stats collector −

stats.set_value( hostname , socket.gethostname())

stats.inc_value( customized_count )

stats.max_value( max_items_scraped , value)

stats.min_value( min_free_memory_percent , value)

stats.get_value( customized_count )

stats.get_stats() { custom_count : 1,  start_time : 
datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} 

Available Stats Collectors

Scrapy provides different types of stats collector which can be accessed using the STATS_CLASS setting.

MemoryStatsCollector

It is the default Stats collector that maintains the stats of every spider which was used for scraping and the data will be stored in the memory.

class scrapy.statscollectors.MemoryStatsCollector

DummyStatsCollector

This stats collector is very efficient which does nothing. This can be set using the STATS_CLASS setting and can be used to disable the stats collection in order to improve the performance.

class scrapy.statscollectors.DummyStatsCollector