Requests Tutorial
Selected Reading
- Requests - Discussion
- Requests - Useful Resources
- Requests - Quick Guide
- Requests - Web Scraping using Requests
- Requests - Proxy
- Requests - Event Hooks
- Requests - Authentication
- Requests - SSL Certification
- Requests - Handling Sessions
- Requests - Handling History
- Requests - Handling Redirection
- Requests - Handling Timeouts
- Requests - Working with Errors
- Requests - Working with Cookies
- Requests - File Upload
- Handling POST, PUT, PATCH & DELETE Requests
- Requests - Handling GET Requests
- Requests - HTTP Requests Headers
- Handling Response for HTTP Requests
- Requests - Working with Requests
- Requests - How Http Requests Work?
- Requests - Environment Setup
- Requests - Overview
- Requests - Home
Selected Reading
- Who is Who
- Computer Glossary
- HR Interview Questions
- Effective Resume Writing
- Questions and Answers
- UPSC IAS Exams Notes
Requests - Quick Guide
Requests - Quick Guide
Requests - Overview
Requests is a HTTP pbrary that provides easy functionapty to deal with http request/response in your web apppcation. The pbrary is developed in python.
The official website of Python Requests which is available at defines Requests as follows −
Requests is an elegant and simple HTTP pbrary for Python, built for human beings.
Features of Requests
The features of Requests are discussed below −
Request
The python requests pbrary has easy to use methods available to handle Http request. Passing of parameters and handpng the request type pke GET, POST, PUT, DELETE, etc. is very easy.
Response
You can get the response in the format you need and the supported ones are text format, binary response, json response, and raw response.
Headers
The pbrary allows you to read, update or send new headers as per your requirements.
Timeouts
Timeouts can be easily added to the URL you are requesting using python requests pbrary. It so happens that you are using a third-party URL and waiting for a response.
It is always a good practice to give a timeout on the URL as we might want the URL to respond within that timeout with a response or an error that is coming because of timeout. Not doing so can cause either to wait on that request indefinitely.
Error handpng
The requests module gives support for error handpng and some of which are Connection Error, Timeout errors, TooManyRedirects, Response.raise_for_status errors, etc.
Cookies
The pbrary allows you to read, write and update for the requested URL.
Sessions
To maintain the data, you require between requests you need sessions. So, if the same host is called again and again you can re-use the TCP connection which in turn will improve the performance.
SSL certificates
SSL certificate is a security feature that comes with secure urls. When you use Requests, it also verifies SSL certificates for the https URL given. SSL Verification is enabled by default in the requests pbrary and will throw an error if the certificate is not present.
Authentication
HTTP authentication is on the server-side asking for some authentication information pke username, password when the cpent requests a URL. This is an additional security for the request and the response being exchanged between the cpent and the server.
Advantages of using Python Requests Library
Following are the advantages of using Python Requests Library −
Easy to use and fetch the data from the URL given.
Requests pbrary can be used to scrape the data from the website.
Using requests, you can get, post, delete, update the data for the URL given.
The handpng of cookies and session is very easy.
The security is also taken care of the help of authentication module support.
Requests - Environment Setup
In this chapter, we will work on the installation of Requests. To start working with the Requests module, we need to install Python first. So we are going to work on following−
Install Python
Install Requests
Instalpng Python
Go to the Python official site: as shown below and cpck on the latest version available for Windows, Linux/Unix, and Mac OS. Download Python as per your 64 or 32 bit OS available with you.
Once you have downloaded, cpck on the .exe file and follow the steps to install python on your system.
The python package manager, i.e., pip will also get installed by default with the above installation. To make it work globally on your system, directly add the location of python to the PATH variable. The same is shown at the start of the installation to remember to check the checkbox which says ADD to PATH. In case you forget to check it, please follow the below-given steps to add to PATH.
To add to PATH follow the steps−
Right-cpck on your Computer icon and cpck on properties > Advanced System Settings.
It will display the screen as shown below −
Cpck on Environment Variables as shown above. It will display the screen as shown below −
Select Path and cpck on Edit button, add the location path of your python at the end. Now, let us check the python version.
Checking the python version
E:prequests>python --version Python 3.7.3
Install Requests
Now that we have python installed, we are going to install Requests.
Once python is installed, python package manager i.e. pip will also get installed. Following is the command to check pip version.
E:prequests>pip --version pip 19.1.1 from c:usersxxxxxappdatalocalprogramspythonpython37pbsite-p ackagespip (python 3.7)
We have pip installed and the version is 19.1.1. Now, will use pip to install Requests module.
The command is given below−
pip install requests
E:prequests>pip install requests Requirement already satisfied: requests in c:usersxxxxappdatalocalprograms pythonpython37pbsite-packages (2.22.0) Requirement already satisfied: certifi>=2017.4.17 in c:userskamatappdataloca lprogramspythonpython37pbsite-packages (from requests) (2019.3.9) Requirement already satisfied: urlpb3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:use rsxxxxxappdatalocalprogramspythonpython37pbsite-packages (from requests ) (1.25.3) Requirement already satisfied: idna<2.9,>=2.5 in c:usersxxxxxxxappdatalocalpr ogramspythonpython37pbsite-packages (from requests) (2.8) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:usersxxxxxappdatal ocalprogramspythonpython37pbsite-packages (from requests) (3.0.4)
We already have the module installed, so in the command prompt it says Requirement already satisfied; if not installed it would have downloaded the required packages for installation.
To check the details of the requests module installed, you can use the following command−
pip show requests E:prequests>pip show requests Name: requests Version: 2.22.0 Summary: Python HTTP for Humans. Home-page: Author: Kenneth Reitz Author-email: me@kennethreitz.org License: Apache 2.0 Location: c:usersxxxxxappdatalocalprogramspythonpython37pbsite-package S Requires: certifi, idna, urlpb3, chardet Required-by:
The version of Requests module is 2.22.0.
Requests - How Http Requests Work?
Python’s Requests is a HTTP pbrary that will help us exchange data between the cpent and the server. Consider you have a UI with a form, wherein you need to enter the user details, so once you enter it, you have to submit the data which is nothing but a Http POST or PUT request from the cpent to server to save the data.
When you want the data, you need to fetch it from the server, which is again a Http GET request. The exchange of data between the cpent when it requests the data and the server responding with the required data, this relationship between the cpent and the server is very important.
The request is made to the URL given and it could be a secure or non-secure URL.
The request to the URL can be done using GET, POST, PUT, DELETE. The most commonly used is the GET method, mainly used when you want to fetch data from the server.
You can also send data to the URL as a query string for example−
So here, we are passing id = 9 and username = Delphine to the URL. All the values are sent in key/value pair after the question mark(?) and multiple params are passed to the URL separated by &.
Using the request pbrary, the URL is called as follows using a string dictionary.
Wherein the data to the URL is sent as a dictionary of strings. If you want to pass id=9 and username=Delphine, you can do as follows−
payload = { id : 9 , username : Delphine }
The requests pbrary is called as follows−
res = requests.get( , params=payload )
Using POST, we can do as follows−
res = requests.post( , data = { id : 9 , username : Delphine })
Using PUT
res = requests.put( , data = { id : 9 , username : Delphine })
Using DELETE
res = requests.delete( )
The response from the Http request can be in text encoded form, binary encoded, json format or raw response. The details of the request and response are explained in detail in the next chapters.
Requests - Working with Requests
In this chapter, we will understand how to work with the requests module. We will look into the following−
Making HTTP Requests.
Passing Parameters to HTTP Requests.
Making HTTP Requests
To make a Http request, we need to first import the request module as shown below−
import requests
Let us now see, how to make a call to URL using the requests module.
Let us use the URL− in the code, to test Requests Module.
Example
import requests getdata = requests.get( ) print(getdata.status_code)
The url− is called using requests.get() method. The response object of the URL is stored in the getdata variable. When we print the variable, it gives the 200 response code, which means that we have got the response successfully.
Output
E:prequests>python makeRequest.py <Response [200]>
To get the content from the response, we can do so using getdata.content as shown below−
Example
import requests getdata = requests.get( ) print(getdata.content)
getdata.content, will print all the data available in the response.
Output
E:prequests>python makeRequest.py
b [
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light
",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "
92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.149
6"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hild
egard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhr
ase": "Multi-layered cpent-server neural-net",
"bs": "harness real-time
e-markets"
}
}
Passing Parameters to HTTP Requests
Just requesting the URL is not sufficient, we also need to pass parameters to the URL.
The params are mostly passed as key/value pair, for example−
So, we have id=9 and username=Delphine. Now, will see how to pass such data to requests Http module.
Example
import requests
payload = { id : 9, username : Delphine }
getdata = requests.get( , params=payload)
print(getdata.content)
The details are stored in the object payload in the key/value pair and passed to params, inside get() method.
Output
E:prequests>python makeRequest.py
b [
{
"id": 9,
"name": "Glenna Reichert",
"username": "Delphin
e",
"email": "Chaim_McDermott@dana.io",
"address": {
"street":
"Dayna Park",
"suite": "Suite 449",
"city": "Bartholomebury",
"zipcode": "76495-3109",
"geo": {
"lat": "24.6463",
"lng": "-168.8889"
}
},
"phone": "(775)976-6794 x41206",
"
website": "conrad.com",
"company": {
"name": "Yost and Sons",
"catchPhrase": "Switchable contextually-based project",
"bs": "aggregate
real-time technologies"
}
}
]
We are now getting the details of the id=9 and username=Delphine details in the response.
If you want to see, how the URL looks after passing the parameters, making use of the response object to the URL.
Example
import requests
payload = { id : 9, username : Delphine }
getdata = requests.get( , params=payload)
print(getdata.url)
Output
E:prequests>python makeRequest.py
Handpng Response for HTTP Requests
In this chapter, we will get into more details of the response received from the requests module. We will discuss the following details−
Getting Response
JSON Response
RAW Response
Binary Response
Getting Response
We will make a request to the URL using request.get() method.
import requests getdata = requests.get();
The getdata has the response object. It has all the details of the response. We can get a response in 2 ways using (.text) and (.content). Using response.text will give you the data back in text format as shown below−
Example
E:prequests>python makeRequest.py
[
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered cpent-server neural-net",
"bs": "harness real-time e-markets"
}
},
You will see the response is the same, as how it would have appeared in the browser when you do view source for the URL as shown below−
You can also try out .html URL and see the content using response.text, it will be the same as the view source content for the .html URL in the browser.
Now, let us try response.content for the same URL and see the output.
Example
import requests getdata = requests.get( ) print(getdata.content)
Output
E:prequests>python makeRequest.py
b [
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light
",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "
92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.149
6"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hild
egard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhr
ase": "Multi-layered cpent-server neural-net",
"bs": "harness real-time
e-markets"
}
},
{
"id": 2,
"name": "Ervin Howell",
"us
ername": "Antonette",
"email": "Shanna@mepssa.tv",
"address": {
"street": "Victor Plains",
"suite": "Suite 879",
"city": "Wisoky
burgh",
"zipcode": "90566-7771",
"geo": {
"lat": "-43.950
9",
"lng": "-34.4618"
}
},
"phone": "010-692-6593 x091
25",
"website": "anastasia.net",
"company": {
"name": "Deckow-C
rist",
"catchPhrase": "Proactive didactic contingency",
"bs": "syn
ergize scalable supply-chains"
}
},
{
"id": 3,
"name": "Cle
mentine Bauch",
"username": "Samantha",
"email":
"Nathan@yesenia.net",
"address": {
"street": "Douglas Extension",
"suite": "Suite
847",
"city": "McKenziehaven",
"zipcode": "59590-4157",
"ge
o": {
"lat": "-68.6102",
"lng": "-47.0653"
}
},
The response is given in bytes. You will get a letter b at the start of the response. With the requests module, you can get the encoding used and also change the encoding if required. For example, to get the encoding you can use response.encoding.
print(getdata.encoding)
Output
utf-8
You can change the encoding as follows− You can use the encoding of your choice.
getdata.encoding = ISO-8859-1
JSON Response
You can also get the response for the Http request in json format by using response.json() method as follows−
Example
import requests getdata = requests.get( ) print(getdata.json())
Output
E:prequests>python makeRequest.py
[{ id : 1, name : Leanne Graham , username : Bret , email : Sincere@april.
biz , address : { street : Kulas Light , suite : Apt. 556 , city : Gwenbor
ough , zipcode : 92998-3874 , geo : { lat : -37.3159 , lng : 81.1496 }},
phone : 1-770-736-8031 x56442 , website : hildegard.org , company : { name :
Romaguera-Crona , catchPhrase : Multi-layered cpent-server neural-net , bs
: harness real-time e-markets }}]
RAW Response
In case you need the raw response for the Http URL you can make use of response.raw, also add stream=True inside the get method as shown below−
Example
import requests getdata = requests.get( , stream=True) print(getdata.raw)
Output
E:prequests>python makeRequest.py <urlpb3.response.HTTPResponse object at 0x000000A8833D7B70>
To read for more content from the raw data you can do so as follows−
print(getdata.raw.read(50))
Output
E:prequests>python makeRequest.py b x1fx8bx08x00x00x00x00x00x00x03x95x98[oxe38x12x85xdfxe7Wx10y xdax01Fx82.xd4mx9fxdcx9ddxbaxb7x93xf4x06qxef4x06x83A@Kx15x89m
Binary Response
To get a binary response we can make use of response.content.
Example
import requests getdata = requests.get( ) print(getdata.content)
Output
E:prequests>python makeRequest.py
b [
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light
",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "
92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.149
6"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hild
egard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhr
ase": "Multi-layered cpent-server neural-net",
"bs": "harness real-time
e-markets"
}
},
{
"id": 2,
"name": "Ervin Howell",
"us
ername": "Antonette",
"email": "Shanna@mepssa.tv",
"address": {
"street": "Victor Plains",
"suite": "Suite 879",
"city": "Wisoky
burgh",
"zipcode": "90566-7771",
"geo": {
"lat": "-43.950
9",
"lng": "-34.4618"
}
},
"phone": "010-692-6593 x091
25",
"website": "anastasia.net",
"company": {
"name": "Deckow-C
rist",
"catchPhrase": "Proactive didactic contingency",
"bs": "syn
ergize scalable supply-chains"
}
},
{
"id": 3,
"name": "Cle
mentine Bauch",
"username": "Samantha",
"email": "Nathan@yesenia.net",
"address": {
"street": "Douglas Extension",
"suite": "Suite
847",
"city": "McKenziehaven",
"zipcode": "59590-4157",
"ge
o": {
"lat": "-68.6102",
"lng": "-47.0653"
}
},
The response is given in bytes. You will get a letter b at the start of the response. The binary response is mostly used for non-text requests.
Requests - HTTP Requests Headers
In the previous chapter, we have seen how to make the request and get the response. This chapter will explore a pttle more on the header section of the URL. So, we are going to look into the following−
Understanding Request Headers
Custom Headers
Response Headers
Understanding Request Headers
Hit any URL in the browser, inspect it and check in developer tool network tab.
You will get response headers, request headers, payload, etc.
For example, consider the following URL−
You can get the header details as follows−
Example
import requests getdata = requests.get( , stream=True) print(getdata.headers)
Output
E:prequests>python makeRequest.py
{ Date : Sat, 30 Nov 2019 05:15:00 GMT , Content-Type : apppcation/json; cha
rset=utf-8 , Transfer-Encoding : chunked , Connection : keep-apve , Set-Co
okie : __cfduid=d2b84ccf43c40e18b95122b0b49f5cf091575090900; expires=Mon, 30-De
c-19 05:15:00 GMT; path=/; domain=.typicode.com; HttpOnly , X-Powered-By : Exp
ress , Vary : Origin, Accept-Encoding , Access-Control-Allow-Credentials : t
rue , Cache-Control : max-age=14400 , Pragma : no-cache , Expires : -1 ,
X-Content-Type-Options : nosniff , Etag : W/"160d-1eMSsxeJRfnVLRBmYJSbCiJZ1qQ
" , Content-Encoding : gzip , Via : 1.1 vegur , CF-Cache-Status : HIT , A
ge : 2271 , Expect-CT : max-age=604800, report-uri="https://report-uri.cloudf
lare.com/cdn-cgi/beacon/expect-ct" , Server : cloudflare , CF-RAY : 53da574f
f99fc331-SIN }
To read any http header you can do so as follows−
getdata.headers["Content-Encoding"] // gzip
Custom Headers
You can also send headers to the URL being called as shown below.
Example
import requests
headers = { x-user : test123 }
getdata = requests.get( , headers=headers)
The headers passed has to be string, bytestring, or Unicode format. The behavior of the request will not change as per the custom headers passed.
Response Headers
The response headers look pke below when you check the URL in the browser developer tool, network tab−
To get the details of the headers from the requests module use. Response.headers are as shown below−
Example
import requests getdata = requests.get( ) print(getdata.headers)
Output
E:prequests>python makeRequest.py
{ Date : Sat, 30 Nov 2019 06:08:10 GMT , Content-Type : apppcation/json; cha
rset=utf-8 , Transfer-Encoding : chunked , Connection : keep-apve , Set-Co
okie : __cfduid=de1158f1a5116f3754c2c353055694e0d1575094090; expires=Mon, 30-De
c-19 06:08:10 GMT; path=/; domain=.typicode.com; HttpOnly , X-Powered-By : Exp
ress , Vary : Origin, Accept-Encoding , Access-Control-Allow-Credentials : t
rue , Cache-Control : max-age=14400 , Pragma : no-cache , Expires : -1 ,
X-Content-Type-Options : nosniff , Etag : W/"160d-1eMSsxeJRfnVLRBmYJSbCiJZ1qQ
" , Content-Encoding : gzip , Via : 1.1 vegur , CF-Cache-Status : HIT , A
ge : 5461 , Expect-CT : max-age=604800, report-uri="https://report-uri.cloudf
lare.com/cdn-cgi/beacon/expect-ct" , Server : cloudflare , CF-RAY : 53daa52f
3b7ec395-SIN }
You can get any specific header you want as follows−
print(getdata.headers["Expect-CT"])
Output
max-age=604800, report-uri=" ect-ct
You can also get the header details by using the get() method.
print(getdata.headers.get("Expect-CT"))
Output
max-age=604800, report-uri=" ect-ct
Requests - Handpng GET Requests
This chapter will concentrate more on the GET requests, which is the most common and used very often. The working of GET in the requests module is very easy. Here is a simple example about working with the URL using the GET method.
Example
import requests getdata = requests.get( ) print(getdata.content) getdata.content, will print all the data available in the response.
Output
E:prequests>python makeRequest.py
b [
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light
",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "
92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.149
6"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hild
egard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhr
ase": "Multi-layered cpent-server neural-net",
"bs": "harness real-time
e-markets"
}
}
You can also pass parameters to the get method using the param attribute as shown below−
import requests
payload = { id : 9, username : Delphine }
getdata = requests.get( ,
params=payload)
print(getdata.content)
The details are stored in the object payload in the key/value pair and passed to params, inside get() method.
Output
E:prequests>python makeRequest.py
b [
{
"id": 9,
"name": "Glenna Reichert",
"username": "Delphin
e",
"email": "Chaim_McDermott@dana.io",
"address": {
"street":
"Dayna Park",
"suite": "Suite 449",
"city": "Bartholomebury",
"zipcode": "76495-3109",
"geo": {
"lat": "24.6463",
"lng": "-168.8889"
}
},
"phone": "(775)976-6794 x41206",
"
website": "conrad.com",
"company": {
"name": "Yost and Sons",
"catchPhrase": "Switchable contextually-based project",
"bs": "aggregate
real-time technologies"
}
}
]
Handpng POST, PUT, PATCH and DELETE Requests
In this chapter, we will understand how to use the POST method using requests pbrary and also pass parameters to the URL.
Using POST
For PUT request, the Requests pbrary has requests.post() method, the example of it is shown below:
import requests
myurl = myparams = { name : ABC , email : xyz@gmail.com } res = requests.post(myurl, data=myparams) print(res.text)
Output
E:prequests>python makeRequest.py
{"args":{},"data":"","files":{},"form":{"name":"ABC","email":"xyz@gmail.com"},"headers":{"x-forwarded-proto":"https","host":"postman-echo.com","content-length":"30","accept":"*/*","accept-encoding":"gzip,deflate","content-type":"apppcation/x-www-form-urlencoded","user-agent":"python-requests/2.22.0","x-forwarded-port":"443"},"json":{"name":"ABC","email":"xyz@gmail.com"},"url":""}
In the example shown above, you can pass the form data as key-value pair to the data param inside requests.post(). We will also see how to work with PUT, PATCH and DELETE in requests module.
Using PUT
For PUT request, the Requests pbrary has requests.put() method, the example of it is shown below.
import requests myurl = myparams = { name : ABC , email : xyz@gmail.com } res = requests.put(myurl, data=myparams) print(res.text)
Output
E:prequests>python makeRequest.py
{"args":{},"data":"","files":{},"form":{"name":"ABC","email":"xyz@gmail.com"},"h
eaders":{"x-forwarded-proto":"https","host":"postman-echo.com","content-length":
"30","accept":"*/*","accept-encoding":"gzip, deflate","content-type":"apppcatio
n/x-www-form-urlencoded","user-agent":"python-requests/2.22.0","x-forwarded-port
":"443"},"json":{"name":"ABC","email":"xyz@gmail.com"},"url":""}
Using PATCH
For the PATCH request, the Requests pbrary has requests.patch() method, the example of it is shown below.
import requests myurl = res = requests.patch(myurl, data="testing patch") print(res.text)
Output
E:prequests>python makeRequest.py
{"args":{},"data":{},"files":{},"form":{},"headers":{"x-forwarded-proto":"https"
,"host":"postman-echo.com","content-length":"13","accept":"*/*","accept-encoding
":"gzip, deflate","user-agent":"python-requests/2.22.0","x-forwarded-port":"443"
},"json":null,"url":""}
Using DELETE
For the DELETE request, the Requests pbrary has requests.delete() method, the example of it is shown below.
import requests myurl = res = requests.delete(myurl, data="testing delete") print(res.text)
Output
E:prequests>python makeRequest.py
{"args":{},"data":{},"files":{},"form":{},"headers":{"x-forwarded-proto":"https"
,"host":"postman-echo.com","content-length":"14","accept":"*/*","accept-encoding
":"gzip, deflate","user-agent":"python-requests/2.22.0","x-forwarded-port":"443"
},"json":null,"url":""}
Requests - File Upload
In this chapter, we will upload a file using request and read the contents of the file uploaded. We can do it using the files param as shown in the example below.
We will use the post to upload the file.
Example
import requests myurl = files = { file : open( test.txt , rb )} getdata = requests.post(myurl, files=files) print(getdata.text)
Test.txt
File upload test using Requests
Example
var total = [0, 1, 2, 3].reduceRight(function(a, b){ return a + b; });
console.log("total is : " + total );
Output
E:prequests>python makeRequest.py
{
"args": {},
"data": "",
"files": {
"file": "File upload test using Requests"
},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "175",
"Content-Type": "multipart/form-data;
boundary=28aee3a9d15a3571fb80d4d2a94bf
d33",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.22.0"
},
"json": null,
"origin": "117.223.63.135, 117.223.63.135",
"url": ""
}
It is also possible to send the contents of the file as shown below−
Example
import requests myurl = files = { file : ( test1.txt , Welcome to TutorialsPoint )} getdata = requests.post(myurl, files=files) print(getdata.text)
Output
E:prequests>python makeRequest.py
{
"args": {},
"data": "",
"files": {
"file": "Welcome to TutorialsPoint"
},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "170",
"Content-Type": "multipart/form-data; boundary=f2837238286fe40e32080aa7e172b
e4f",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.22.0"
},
"json": null,
"origin": "117.223.63.135, 117.223.63.135",
"url": ""
}
Requests - Working with Cookies
This chapter will discuss how to deal with cookies. You can get the cookies as well as send your cookies while calpng the URL using the requests pbrary.
The url, when hits in the browser we can get the details of the cookies as shown below−
You can read the cookies as shown below−
Example
import requests getdata = requests.get( ) print(getdata.cookies["__cfduid"])
Output
E:prequests>python makeRequest.py d1733467caa1e3431fb7f768fa79ed3741575094848
You can also send cookies when we make a request.
Example
import requests cookies = dict(test= test123 ) getdata = requests.get( ,cookies=cookies) print(getdata.text)
Output
E:prequests>python makeRequest.py
{
"cookies": {
"test": "test123"
}
}
Requests - Working with Errors
This chapter will discuss how to deal with errors coming down when working with the Http request pbrary. It is always a good practice to have errors managed for all possible cases.
Error Exception
The requests module gives the following types of error exception−
ConnectionError− This will be raised, if there is any connection error. For example, the network failed, DNS error so the Request pbrary will raise ConnectionError exception.
Response.raise_for_status()− Based on status code i.e. 401, 404 it will raise HTTPError for the url requested.
HTTPError− This error will be raised for an invapd response coming down for the request made.
Timeout− Errors raised for a timeout for the URL requested.
TooManyRedirects− If the pmit is crossed for maximum redirections than it will raise TooManyRedirects error.
Example
Here is an example of errors shown for timeout−
import requests getdata = requests.get( ,timeout=0.001) print(getdata.text)
Output
raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host= jsonplaceholder.ty picode.com , port=443): Max retries exceeded with url: /users (Caused by Connect TimeoutError(<urlpb3.connection.VerifiedHTTPSConnection object at 0x000000B02AD E76A0>, Connection to jsonplaceholder.typicode.com timed out. (connect timeout= 0.001) ))
Requests - Handpng Timeouts
Timeouts can be easily added to the URL you are requesting. It so happens that, you are using a third-party URL and waiting for a response. It is always a good practice to give a timeout on the URL, as we might want the URL to respond within a timespan with a response or an error. Not doing so, can cause to wait on that request indefinitely.
We can give timeout to the URL by using the timeout param and value is passed in seconds as shown in the example below−
Example
import requests getdata = requests.get( ,timeout=0.001) print(getdata.text)
Output
raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host= jsonplaceholder.ty picode.com , port=443): Max retries exceeded with url: /users (Caused by Connect TimeoutError(<urlpb3.connection.VerifiedHTTPSConnection object at 0x000000B02AD E76A0>, Connection to jsonplaceholder.typicode.com timed out. (connect timeout= 0.001) ))
The timeout given is as follows−
getdata = requests.get( ,timeout=0.001)
The execution throws connection timeout error as shown in the output. The timeout given is 0.001, which is not possible for the request to get back the response and throws an error. Now, we will increase the timeout and check.
Example
import requests getdata = requests.get( ,timeout=1.000) print(getdata.text)
Output
E:prequests>python makeRequest.py
[
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered cpent-server neural-net",
"bs": "harness real-time e-markets"
}
With a timeout of 1 second, we can get the response for the URL requested.
Requests - Handpng Redirection
This chapter will take a look at how the Request pbrary handles the url redirection case.
Example
import requests getdata = requests.get( ) print(getdata.status_code) print(getdata.history)
The url− will be redirected using status code 301(Moved Permanently) to . The redirection will be saved in the history.
Output
When the above code is executed, we get the following result−
E:prequests>python makeRequest.py 200 [<Response [301]>]
You can stop redirection of a URL using allow_redirects=False. It can be done on GET, POST, OPTIONS, PUT, DELETE, PATCH methods used.
Example
Here is an example on the same.import requests getdata = requests.get( , allow_redirects=False) print(getdata.status_code) print(getdata.history) print(getdata.text)
Now if you check the output, the redirection will not be allowed and will get a status code of 301.
Output
E:prequests>python makeRequest.py 301 [] <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>301 Moved</TITLE></HEAD><BODY> <H1>301 Moved</H1> The document has moved <A HREF="http://www.google.com/">here</A>. </BODY></HTML>
Requests - Handpng History
You can get the history of a given URL by using response.history. If the given URL has any redirects, the same will be stored in history.
For history
import requests getdata = requests.get( ) print(getdata.status_code) print(getdata.history)
Output
E:prequests>python makeRequest.py 200 [<Response [301]>]
The response.history property will have the details of the response objects that were done based on the request. The values present will be sorted from the oldest to the newest ones. The response.history property tracks all the redirection done on the URL requested.
Requests - Handpng Sessions
To maintain the data between requests you need sessions. So, if the same host is called again and again, you can reuse the TCP connection which in turn will improve the performance. Let us now see, how to maintain cookies across requests made using sessions.
Adding cookies using session
import requests req = requests.Session() cookies = dict(test= test123 ) getdata = req.get( ,cookies=cookies) print(getdata.text)
Output
E:prequests>python makeRequest.py
{
"cookies": {
"test": "test123"
}
}
Using session, you can preserve the cookies data across requests. It is also possible to pass headers data using the session as shown below−
Example
import requests
req = requests.Session()
req.headers.update({ x-user1 : ABC })
headers = { x-user2 : XYZ }
getdata = req.get( , headers=headers)
print(getdata.headers)
Requests - SSL Certification
SSL certificate is a security feature that comes with secure urls. When you use Requests pbrary, it also verifies SSL certificates for the https URL given. SSL verification is enabled by default in the requests module and will throw an error if the certificate is not present.
Working with secure URL
Following is the example of working with secure URL−
import requests getdata = requests.get() print(getdata.text)
Output
E:prequests>python makeRequest.py
[
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered cpent-server neural-net",
"bs": "harness real-time e-markets"
}
}
]
We are easily getting a response from the above https URL, and it is because the request module can verify the SSL certificate.
You can disable the SSL verification by simply adding verify=False as shown in the example below.
Example
import requests getdata = requests.get( , verify=False) print(getdata.text)
You will get the output, but it will also give a warning message that, the SSL certificate is not verified and adding certificate verification is advised.
Output
E:prequests>python makeRequest.py connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: .readthedocs.io/en/latest/advanced-usage.htm l#ssl-warnings InsecureRequestWarning) [ { "id": 1, "name": "Leanne Graham", "username": "Bret", "email": "Sincere@april.biz", "address": { "street": "Kulas Light", "suite": "Apt. 556", "city": "Gwenborough", "zipcode": "92998-3874", "geo": { "lat": "-37.3159", "lng": "81.1496" } }, "phone": "1-770-736-8031 x56442", "website": "hildegard.org", "company": { "name": "Romaguera-Crona", "catchPhrase": "Multi-layered cpent-server neural-net", "bs": "harness real-time e-markets" } } ]
You can also verify SSL certificate by hosting it at your end, and giving the path using verify param as shown below.
Example
import requests getdata = requests.get( , verify= C:UsersAppDataLocalcertificate.txt ) print(getdata.text)
Output
E:prequests>python makeRequest.py
[
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered cpent-server neural-net",
"bs": "harness real-time e-markets"
}
}
]
Requests - Authentication
This chapter will discuss the types of authentication available in the Requests module.
We are going to discuss the following−
Working of Authentication in HTTP Requests
Basic Authentication
Digest Authentication
OAuth2 Authentication
Working of Authentication in HTTP Requests
HTTP authentication is on the server-side asking for some authentication information pke username, password when the cpent requests a URL. This is additional security for the request and the response being exchanged between the cpent and the server.
From the cpent-side these additional authentication information i.e. username and password can be sent in the headers, which later on the server side will be vapdated. The response will be depvered from the server-side only when the authentication is vapd.
Requests pbrary has most commonly used authentication in requests.auth, which are Basic Authentication (HTTPBasicAuth) and Digest Authentication (HTTPDigestAuth).
Basic Authentication
This is the simplest form of providing authentication to the server. To work with basic authentication, we are going to use HTTPBasicAuth class available with requests pbrary.
Example
Here is a working example of how to use it.
import requests from requests.auth import HTTPBasicAuth response_data = requests.get( , auth=HTTPDigestAuth( admin , admin123 )) print(response_data.text)
We are calpng the url, with user as admin and password as admin123.
So, this URL will not work without authentication, i.e. user and password. Once you give the authentication using the auth param, then only the server will give back the response.
Output
E:prequests>python makeRequest.py
{
"authenticated": true,
"user": "admin"
}
Digest Authentication
This is another form of authentication available with requests. We are going to make use of HTTPDigestAuth class from requests.
Example
import requests from requests.auth import HTTPDigestAuth response_data = requests.get( >, auth=HTTPDigestAuth( admin , admin123 )) print(response_data.text)
Output
E:prequests>python makeRequest.py
{
"authenticated": true,
"user": "admin"
}
OAuth2 Authentication
To use OAuth2 Authentication, we need “requests_oauth2” pbrary. To install “requests_oauth2” do the following−
pip install requests_oauth2
The display in your terminal while instalpng will be something as shown below−
E:prequests>pip install requests_oauth2 Collecting requests_oauth2 Downloading 71d111d7105df230ddb74b5d4e10a3dabb61750c/requests-oauth2-0.3.0.tar.gz Requirement already satisfied: requests in c:usersxyzappdatalocalprograms pythonpython37pbsite-packages (from requests_oauth2) (2.22.0) Requirement already satisfied: six in c:usersxyzappdatalocalprogramspyth onpython37pbsite-packages (from requests_oauth2) (1.12.0) Requirement already satisfied: urlpb3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:use rsxyzappdatalocalprogramspythonpython37pbsite-packages (from requests ->requests_oauth2) (1.25.3) Requirement already satisfied: certifi>=2017.4.17 in c:usersxyzappdataloca lprogramspythonpython37pbsite-packages (from requests->requests_oauth2) (2 019.3.9) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:usersxyzappdatal ocalprogramspythonpython37pbsite-packages (from requests->requests_oauth2) (3.0.4) Requirement already satisfied: idna<2.9,>=2.5 in c:usersxyzappdatalocalpr ogramspythonpython37pbsite-packages (from requests->requests_oauth2) (2.8) Building wheels for collected packages: requests-oauth2 Building wheel for requests-oauth2 (setup.py) ... done Stored in directory: C:UsersxyzAppDataLocalpipCachewheels90ef443 3743cbbc488463491da7df510d41c4e5aa28213caeedd586 Successfully built requests-oauth2
We are done instalpng “requests-oauth2”. To use the API’s of Google, Twitter we need its consent and the same is done using OAuth2 authentication.
For OAuth2 authentication we will need Cpent ID and a Secret Key. The details of how to get it, is mentioned on .
Later on, login to Google API Console which is available at and get the cpent id and secret key.
Example
Here is an example of how to use "requests-oauth2".
import requests from requests_oauth2.services import GoogleCpent google_auth = GoogleCpent( cpent_id="xxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com", redirect_uri="", ) a = google_auth.authorize_url( scope=["profile", "email"], response_type="code", ) res = requests.get(a) print(res.url)
We will not be able to redirect to the URL given, as it needs to login to the Gmail account, but here, you will see from the example, that google_auth works and the authorized URL is given.
Output
E:prequests>python oauthRequest.py = http%3A%2F%2Flocalhost%2Fauth%2Fsuccess.html& cpent_id=xxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com& scope=profile+email&response_type=code
Requests - Event Hooks
We can add events to the URL requested using event hooks. In the example below, we are going to add a callback function that will get called when the response is available.
Example
To add the callback, we need to make use of hooks param as shown in the example below−
mport requests def printData(r, *args, **kwargs): print(r.url) print(r.text) getdata = requests.get( , hooks={ response : printData})
Output
E:prequests>python makeRequest.py [ { "id": 1, "name": "Leanne Graham", "username": "Bret", "email": "Sincere@april.biz", "address": { "street": "Kulas Light", "suite": "Apt. 556", "city": "Gwenborough", "zipcode": "92998-3874", "geo": { "lat": "-37.3159", "lng": "81.1496" } }, "phone": "1-770-736-8031 x56442", "website": "hildegard.org", "company": { "name": "Romaguera-Crona", "catchPhrase": "Multi-layered cpent-server neural-net", "bs": "harness real-time e-markets" } } ]
You can also call multiple callback functions as shown below−
Example
import requests def printRequestedUrl(r, *args, **kwargs): print(r.url) def printData(r, *args, **kwargs): print(r.text) getdata = requests.get( , hooks={ response : [printRequestedUrl, printData]})
Output
E:prequests>python makeRequest.py [ { "id": 1, "name": "Leanne Graham", "username": "Bret", "email": "Sincere@april.biz", "address": { "street": "Kulas Light", "suite": "Apt. 556", "city": "Gwenborough", "zipcode": "92998-3874", "geo": { "lat": "-37.3159", "lng": "81.1496" } }, "phone": "1-770-736-8031 x56442", "website": "hildegard.org", "company": { "name": "Romaguera-Crona", "catchPhrase": "Multi-layered cpent-server neural-net", "bs": "harness real-time e-markets" } } ]
You can also add the hook to the Session created as shown below−
Example
import requests def printData(r, *args, **kwargs): print(r.text) s = requests.Session() s.hooks[ response ].append(printData) s.get( )
Output
E:prequests>python makeRequest.py
[
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered cpent-server neural-net",
"bs": "harness real-time e-markets"
}
}
]
Requests - Proxy
So far, we have seen cpents directly connecting and talking to the server. Using proxy, the interaction happens as follows−
The cpent sends a request to the proxy.
The proxy sends the request to the server.
The server sends back the response to the proxy.
The proxy will send a response back to the cpent.
Using Http-proxy is additional security assigned to manage the data exchange between cpent and server. The requests pbraries also have provision to handle proxy, by using the proxies param as shown below−
Example
import requests
proxies = {
http :
}
res = requests.get( , proxies=proxies)
print(res.status_code)
The request will route to ( URL.
Output
200
Requests - Web Scraping using Requests
We have already seen how we can get data from a given URL using python requests pbrary. We will try to scrap the data from the site of Tutorialspoint which is available at using the following−
Requests Library
Beautiful soup pbrary from python
We have already installed the Requests pbrary, let us now install Beautiful soup package. Here is the official website for beautiful soup available at in case you want to explore some more functionapties of beautiful soup.
Instalpng Beautifulsoup
We shall see how to install Beautiful Soup below−
E:prequests>pip install beautifulsoup4 Collecting beautifulsoup4 Downloading cdf92bac4693b90d3ba79268be16527555e186f0/beautifulsoup4-4.8.1-py3-none-any.whl ( 101kB) |████████████████████████████████| 102kB 22kB/s Collecting soupsieve>=1.2 (from beautifulsoup4) Downloading a99f7946ac228ca98da4fa75796c507f61e688c2/soupsieve-1.9.5-py2.py3-none-any.whl Instalpng collected packages: soupsieve, beautifulsoup4 Successfully installed beautifulsoup4-4.8.1 soupsieve-1.9.5
We now have python requests pbrary and beautiful soup installed.
Let us now write the code, that will scrap the data from the URL given.
Web scraping
import requests from bs4 import BeautifulSoup res = requests.get( ) print("The status code is ", res.status_code) print(" ") soup_data = BeautifulSoup(res.text, html.parser ) print(soup_data.title) print(" ") print(soup_data.find_all( h4 ))
Using requests pbrary, we can fetch the content from the URL given and beautiful soup pbrary helps to parse it and fetch the details the way we want.
You can use a beautiful soup pbrary to fetch data using Html tag, class, id, css selector and many more ways. Following is the output we get wherein we have printed the title of the page and also all the h4 tags on the page.
Output
E:prequests>python makeRequest.py The status code is 200 <title>Free Onpne Tutorials and Courses</title> [<h4>Academic</h4>, <h4>Computer Science</h4>, <h4>Digital Marketing</h4>, <h4>M onuments</h4>,<h4>Machine Learning</h4>, <h4>Mathematics</h4>, <h4>Mobile Devel opment</h4>,<h4>SAP</h4>, <h4>Software Quapty</h4>, <h4>Big Data & Analyti cs</h4>, <h4>Databases</h4>, <h4>Engineering Tutorials</h4>, <h4>Mainframe Devel opment</h4>, <h4>Microsoft Technologies</h4>, <h4>Java Technologies</h4>,<h4>XM L Technologies</h4>, <h4>Python Technologies</h4>, <h4>Sports</h4>, <h4>Computer Programming</h4>,<h4>DevOps</h4>, <h4>Latest Technologies</h4>, <h4>Telecom</h4>, <h4>Exams Syllabus</h4>, <h4>UPSC IAS Exams</h4>, <h4>Web Development</h4>, <h4>Scripts</h4>, <h4>Management</h4>,<h4>Soft Skills</h4>, <h4>Selected Readin g</h4>, <h4>Misc</h4>]Advertisements