Use Cases¶
Handling Streaming Responses¶
In addition to receiving responses
with IResponse.deliverBody()
, treq provides a helper function
treq.collect()
which takes a
response
and a single argument function which will be called with all new
data available from the response. Much like IProtocol.dataReceived()
,
treq.collect()
knows nothing about the framing of your data and will
simply call your collector function with any data that is currently available.
Here is an example which simply a file object’s write method to
treq.collect()
to save the response body to a file.
1def download_file(reactor, url, destination_filename):
2 destination = open(destination_filename, 'wb')
3 d = treq.get(url, unbuffered=True)
4 d.addCallback(treq.collect, destination.write)
5 d.addBoth(lambda _: destination.close())
6 return d
Full example: download_file.py
URLs, URIs, and Hyperlinks¶
The url argument to HTTPClient.request()
accepts three URL representations:
High-level:
hyperlink.DecodedURL
Mid-level
str
(unicode
on Python 2)Low-level: ASCII
bytes
orhyperlink.URL
The high-level DecodedURL
form is useful when programatically generating URLs.
Here is an example that builds a URL that contains a &
character, which is automatically escaped properly.
1def main(reactor):
2 url = (
3 DecodedURL.from_text(u"https://httpbin.org")
4 .child(u"get") # add path /get
5 .add(u"foo", u"&") # add query ?foo=%26
6 )
7 print(url.to_text())
8 return treq.get(url).addCallback(print_response)
Full example: basic_url.py
Query Parameters¶
treq.HTTPClient.request()
supports a params
keyword argument which will
be URL-encoded and added to the url
argument in addition to any query
parameters that may already exist.
The params
argument may be either a dict
or a list
of
(key, value)
tuples.
If it is a dict
then the values in the dict may either be scalar values or a list
or tuple
thereof.
Scalar values means str
, bytes
, or anything else — even None
— which will be coerced to str
.
Strings are UTF-8 encoded.
1@inlineCallbacks
2def main(reactor):
3 print('List of tuples')
4 resp = yield treq.get('https://httpbin.org/get',
5 params=[('foo', 'bar'), ('baz', 'bax')])
6 content = yield resp.text()
7 print(content)
8
9 print('Single value dictionary')
10 resp = yield treq.get('https://httpbin.org/get',
11 params={'foo': 'bar', 'baz': 'bax'})
12 content = yield resp.text()
13 print(content)
14
15 print('Multi value dictionary')
16 resp = yield treq.get('https://httpbin.org/get',
17 params={b'foo': [b'bar', b'baz', b'bax']})
18 content = yield resp.text()
19 print(content)
20
21 print('Mixed value dictionary')
22 resp = yield treq.get('https://httpbin.org/get',
23 params={'foo': [1, 2, 3], 'bax': b'quux', b'bar': 'foo'})
24 content = yield resp.text()
25 print(content)
26
27 print('Preserved query parameters')
28 resp = yield treq.get('https://httpbin.org/get?foo=bar',
29 params={'baz': 'bax'})
30 content = yield resp.text()
31 print(content)
Full example: query_params.py
If you prefer a strictly-typed API, try hyperlink.DecodedURL
.
Use its add()
and set()
methods to add query parameters without risk of accidental type coercion.
JSON¶
HTTPClient.request()
supports a json keyword argument that gives a data structure to serialize as JSON (using json.dumps()
).
This also implies a Content-Type: application/json
request header.
The json parameter is mutually-exclusive with data.
The _Response.json()
method decodes a JSON response body.
It buffers the whole response and decodes it with json.loads()
.
1@defer.inlineCallbacks
2def main(reactor):
3 response = yield treq.post(
4 'https://httpbin.org/post',
5 json={"msg": "Hello!"},
6 )
7 data = yield response.json()
8 pprint(data)
Full example: json_post.py
Auth¶
HTTP Basic authentication as specified in RFC 2617 is easily supported by
passing an auth
keyword argument to any of the request functions.
The auth
argument should be a tuple of the form ('username', 'password')
.
1def main(reactor, *args):
2 d = treq.get(
3 'https://httpbin.org/basic-auth/treq/treq',
4 auth=('treq', 'treq')
5 )
6 d.addCallback(print_response)
7 return d
8
9react(main, [])
Full example: basic_auth.py
Redirects¶
treq handles redirects by default.
The following will print a 200 OK response.
1def main(reactor, *args):
2 d = treq.get('https://httpbin.org/redirect/1')
3 d.addCallback(print_response)
4 return d
5
6react(main, [])
Full example: redirects.py
You can easily disable redirects by simply passing allow_redirects=False to any of the request methods.
1def main(reactor, *args):
2 d = treq.get('https://httpbin.org/redirect/1', allow_redirects=False)
3 d.addCallback(print_response)
4 return d
5
6react(main, [])
Full example: disable_redirects.py
You can even access the complete history of treq response objects by calling
the history()
method on the response.
1def main(reactor, *args):
2 d = treq.get('https://httpbin.org/redirect/1')
3
4 def cb(response):
5 print('Response history:')
6 print(response.history())
7 return print_response(response)
8
9 d.addCallback(cb)
Full example: response_history.py
Customizing the Twisted Agent¶
The main treq
module has helper functions that automatically instantiate
an instance of treq.client.HTTPClient
. You can create an instance
of HTTPClient
directly in order to customize the
parameters used to initialize it.
Internally, the HTTPClient
wraps an instance of
twisted.web.client.Agent
. When you create an instance of
HTTPClient
, you must initialize it with an instance of
Agent
. This allows you to customize its
behavior.
1def make_custom_agent(reactor):
2 return Agent(reactor, connectTimeout=42)
3
4def main(reactor, *args):
5 agent = make_custom_agent(reactor)
6 http_client = HTTPClient(agent)
7 d = http_client.get(
8 'https://secure.example.net/area51',
9 auth=('admin', "you'll never guess!"))
10 d.addCallback(print_response)
11 return d
12
13react(main, [])
14
Full example: custom_agent.py