Scrapy - Environment

In this chapter, we will discuss how to install and set up Scrapy. Scrapy must be installed with Python.

Scrapy can be installed by using pip. To install, run the following command −

pip install Scrapy

Windows

Note − Python 3 is not supported on Windows OS.

Step 1 − Install Python 2.7 from Python

Set environmental variables by adding the following paths to the PATH −

C:Python27;C:Python27Scripts; 

You can check the Python version using the following command −

python --version

Step 2 − Install OpenSSL.

Add C:OpenSSL-Win32in in your environmental variables.

Note − OpenSSL comes preinstalled in all operating systems except Windows.

Step 3 − Install Visual C++ 2008 redistributables.

Step 4 − Install pywin32.

Step 5 − Install pip for Python versions older than 2.7.9.

You can check the pip version using the following command −

pip --version

Step 6 − To install scrapy, run the following command −

pip install Scrapy

Anaconda

If you have anaconda or miniconda installed on your machine, run the below command to install Scrapy using conda −

conda install -c scrapinghub scrapy 

Scrapinghub company supports official conda packages for Linux, Windows, and OS X.

Note − It is recommended to install Scrapy using the above command if you have issues instalpng via pip.

Ubuntu 9.10 or Above

The latest version of Python is pre-installed on Ubuntu OS. Use the Ubuntu packages aptgettable provided by Scrapinghub. To use the packages −

Step 1 − You need to import the GPG key used to sign Scrapy packages into APT keyring −

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 627220E7

Step 2 − Next, use the following command to create /etc/apt/sources.pst.d/scrapy.pst file −

echo  deb http://archive.scrapy.org/ubuntu scrapy main  | sudo tee 
/etc/apt/sources.pst.d/scrapy.pst

Step 3 − Update package pst and install scrapy −

sudo apt-get update && sudo apt-get install scrapy

Archpnux

You can install Scrapy from AUR Scrapy package using the following command −

yaourt -S scrapy

Mac OS X

Use the following command to install Xcode command pne tools −

xcode-select --install 

Instead of using system Python, install a new updated version that doesn t confpct with the rest of your system.

Step 1 − Install homebrew.

Step 2 − Set environmental PATH variable to specify that homebrew packages should be used before system packages −

echo "export PATH = /usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc

Step 3 − To make sure the changes are done, reload .bashrc using the following command −

source ~/.bashrc 

Step 4 − Next, install Python using the following command −

brew install python

Step 5 − Install Scrapy using the following command −

pip install Scrapy