HTTP requests in Python3 with urllib.request module

python
HTTP requests in Python3 with urllib.request module
Page content

In this article, I introduced how to use urllib.request module to request with HTTP in Python.

Environment

Runtime environment is as follows.

  • python 3.7

Ordinal HTTP request

HTTP GET request with an Request object

Use Request object included in urllib.request module.

Create Request object instance as follows.

1from urllib.request import *
2
3req = Request(url='https://docs.python.org/ja/', headers={}, method='GET')

Access to Request object property. You can see the host information and path etc..

 1print(req.full_url)
 2> https://docs.python.org/ja/
 3
 4print(req.type)
 5> https
 6
 7print(req.host)
 8> docs.python.org
 9
10print(req.origin_req_host)
11> docs.python.org
12
13print(req.selector)
14> /ja/
15
16print(req.data)
17> None
18
19print(req.unverifiable)
20> False
21
22print(req.method)
23> GET

Then, open the request using urlopen function.

1with urlopen(req) as res:
2    body = res.read().decode()
3    print(body)

HTTP POST request with an Request object

Next, request with HTTP POST.

You can give request body with data option when creating Request object instance.

1from urllib.request import *
2import json
3
4req = Request(url='https://docs.python.org/ja/', data=json.dumps({'param': 'piyo'}).encode(), headers={}, method='POST')
5
6with urlopen(req) as res:
7    body = res.read().decode()
8    print(body)

If data object is not encoded, the error message TypeError: can't concat str to bytes occued. In above case, convert dictionary type to json with json.dumps({'param': 'piyo'}).encode().

HTTP request wit only urlopen function

Instead of Request object, you can request with only urlopen function in simple request case. The sample code is as follows.

1from urllib.request import *
2
3with urlopen('https://docs.python.org/ja/') as res:
4    body = res.read().decode()
5    print(body)

References