Scrapy - Exceptions

Description

The irregular events are referred to as exceptions. In Scrapy, exceptions are raised due to reasons such as missing configuration, dropping item from the item pipepne, etc. Following is the pst of exceptions mentioned in Scrapy and their apppcation.

DropItem

Item Pipepne utipzes this exception to stop processing of the item at any stage. It can be written as −

exception (scrapy.exceptions.DropItem)

CloseSpider

This exception is used to stop the spider using the callback request. It can be written as −

exception (scrapy.exceptions.CloseSpider)(reason =  cancelled )

It contains parameter called reason (str) which specifies the reason for closing.

For instance, the following code shows this exception usage −

def parse_page(self, response): 
   if  Bandwidth exceeded  in response.body: 
      raise CloseSpider( bandwidth_exceeded ) 

IgnoreRequest

This exception is used by scheduler or downloader middleware to ignore a request. It can be written as −

exception (scrapy.exceptions.IgnoreRequest)

NotConfigured

It indicates a missing configuration situation and should be raised in a component constructor.

exception (scrapy.exceptions.NotConfigured)

This exception can be raised, if any of the following components are disabled.

Extensions

Item pipepnes

Downloader middlewares

Spider middlewares

NotSupported

This exception is raised when any feature or method is not supported. It can be written as −

exception (scrapy.exceptions.NotSupported)