Scrapy Tutorial
Scrapy Basic Concepts
Scrapy Live Project
Scrapy Built In Services
Scrapy Useful Resources
Selected Reading
Scrapy Basic Concepts
- Scrapy - Exceptions
- Scrapy - Settings
- Scrapy - Link Extractors
- Scrapy - Requests & Responses
- Scrapy - Feed exports
- Scrapy - Item Pipeline
- Scrapy - Shell
- Scrapy - Item Loaders
- Scrapy - Items
- Scrapy - Selectors
- Scrapy - Spiders
- Scrapy - Command Line Tools
- Scrapy - Environment
- Scrapy - Overview
Scrapy Live Project
- Scrapy - Scraped Data
- Scrapy - Following Links
- Scrapy - Using an Item
- Scrapy - Extracting Items
- Scrapy - Crawling
- Scrapy - First Spider
- Scrapy - Define an Item
- Scrapy - Create a Project
Scrapy Built In Services
- Scrapy - Web Services
- Scrapy - Telnet Console
- Scrapy - Sending an E-mail
- Scrapy - Stats Collection
- Scrapy - Logging
Scrapy Useful Resources
Selected Reading
- Who is Who
- Computer Glossary
- HR Interview Questions
- Effective Resume Writing
- Questions and Answers
- UPSC IAS Exams Notes
Scrapy - Web Services
Scrapy - Web Services
Description
A running Scrapy web crawler can be controlled via JSON-RPC. It is enabled by JSONRPC_ENABLED setting. This service provides access to the main crawler object via protocol. The endpoint for accessing the crawler object is −
http://localhost:6080/crawler
The following table contains some of the settings which show the behavior of web service −
| Sr.No | Setting & Description | Default Value |
|---|---|---|
| 1 | JSONRPC_ENABLED This refers to the boolean, which decides the web service along with its extension will be enabled or not. |
True |
| 2 | JSONRPC_LOGFILE This refers to the file used for logging HTTP requests made to the web service. If it is not set the standard Scrapy log will be used. |
None |
| 3 | JSONRPC_PORT This refers to the port range for the web service. If it is set to none, then the port will be dynamically assigned. |
[6080, 7030] |
| 4 | JSONRPC_HOST This refers to the interface the web service should psten on. |
127.0.0.1 |