Use Cases

Handling Streaming Responses

In addition to receiving responses with IResponse.deliverBody(), treq provides a helper function treq.collect() which takes a response and a single argument function which will be called with all new data available from the response. Much like IProtocol.dataReceived(), treq.collect() knows nothing about the framing of your data and will simply call your collector function with any data that is currently available.

Here is an example which simply a file object’s write method to treq.collect() to save the response body to a file.

def download_file(reactor, url, destination_filename):
    destination = open(destination_filename, 'wb')
    d = treq.get(url, unbuffered=True)
    d.addCallback(treq.collect, destination.write)
    d.addBoth(lambda _: destination.close())
    return d

Full example: download_file.py

URLs, URIs, and Hyperlinks

The url argument to HTTPClient.request() accepts three URL representations:

• High-level: hyperlink.DecodedURL
• Mid-level str (unicode on Python 2)
• Low-level: ASCII bytes or hyperlink.URL

The high-level DecodedURL form is useful when programatically generating URLs. Here is an example that builds a URL that contains a & character, which is automatically escaped properly.

def main(reactor):
    url = (
        DecodedURL.from_text(u"https://httpbin.org")
        .child(u"get")      # add path /get
        .add(u"foo", u"&")  # add query ?foo=%26
    )
    print(url.to_text())
    return treq.get(url).addCallback(print_response)

Full example: basic_url.py

Query Parameters

treq.HTTPClient.request() supports a params keyword argument which will be URL-encoded and added to the url argument in addition to any query parameters that may already exist.

The params argument may be either a dict or a list of (key, value) tuples.

If it is a dict then the values in the dict may either be scalar values or a list or tuple thereof. Scalar values means str, bytes, or anything else — even None — which will be coerced to str. Strings are UTF-8 encoded.

@inlineCallbacks
def main(reactor):
    print('List of tuples')
    resp = yield treq.get('https://httpbin.org/get',
                          params=[('foo', 'bar'), ('baz', 'bax')])
    content = yield resp.text()
    print(content)

    print('Single value dictionary')
    resp = yield treq.get('https://httpbin.org/get',
                          params={'foo': 'bar', 'baz': 'bax'})
    content = yield resp.text()
    print(content)

    print('Multi value dictionary')
    resp = yield treq.get('https://httpbin.org/get',
                          params={b'foo': [b'bar', b'baz', b'bax']})
    content = yield resp.text()
    print(content)

    print('Mixed value dictionary')
    resp = yield treq.get('https://httpbin.org/get',
                          params={'foo': [1, 2, 3], 'bax': b'quux', b'bar': 'foo'})
    content = yield resp.text()
    print(content)

    print('Preserved query parameters')
    resp = yield treq.get('https://httpbin.org/get?foo=bar',
                          params={'baz': 'bax'})
    content = yield resp.text()
    print(content)

Full example: query_params.py

If you prefer a strictly-typed API, try hyperlink.DecodedURL. Use its add() and set() methods to add query parameters without risk of accidental type coercion.

JSON

HTTPClient.request() supports a json keyword argument that gives a data structure to serialize as JSON (using json.dumps()). This also implies a Content-Type: application/json request header. The json parameter is mutually-exclusive with data.

The _Response.json() method decodes a JSON response body. It buffers the whole response and decodes it with json.loads().

@defer.inlineCallbacks
def main(reactor):
    response = yield treq.post(
        'https://httpbin.org/post',
        json={"msg": "Hello!"},
    )
    data = yield response.json()
    pprint(data)

Full example: json_post.py

Auth

HTTP Basic authentication as specified in RFC 2617 is easily supported by passing an auth keyword argument to any of the request functions.

The auth argument should be a tuple of the form ('username', 'password').

def main(reactor, *args):
    d = treq.get(
        'https://httpbin.org/basic-auth/treq/treq',
        auth=('treq', 'treq')
    )
    d.addCallback(print_response)
    return d

react(main, [])

Full example: basic_auth.py

Redirects

treq handles redirects by default.

The following will print a 200 OK response.

def main(reactor, *args):
    d = treq.get('https://httpbin.org/redirect/1')
    d.addCallback(print_response)
    return d

react(main, [])

Full example: redirects.py

You can easily disable redirects by simply passing allow_redirects=False to any of the request methods.

def main(reactor, *args):
    d = treq.get('https://httpbin.org/redirect/1', allow_redirects=False)
    d.addCallback(print_response)
    return d

react(main, [])

Full example: disable_redirects.py

You can even access the complete history of treq response objects by calling the history() method on the response.

def main(reactor, *args):
    d = treq.get('https://httpbin.org/redirect/1')

    def cb(response):
        print('Response history:')
        print(response.history())
        return print_response(response)

    d.addCallback(cb)

Full example: response_history.py

Cookies

Cookies can be set by passing a dict or cookielib.CookieJar instance via the cookies keyword argument. Later cookies set by the server can be retrieved using the cookies() method of the response.

The object returned by cookies() supports the same key/value access as requests cookies.

def main(reactor, *args):
    d = treq.get('https://httpbin.org/cookies/set?hello=world')

    def _get_jar(resp):
        jar = resp.cookies()

        print('The server set our hello cookie to: {}'.format(jar['hello']))

        return treq.get('https://httpbin.org/cookies', cookies=jar)

    d.addCallback(_get_jar)
    d.addCallback(print_response)

    return d

Full example: using_cookies.py

Customizing the Twisted Agent

The main treq module has helper functions that automatically instantiate an instance of treq.client.HTTPClient. You can create an instance of HTTPClient directly in order to customize the paramaters used to initialize it. Internally, the HTTPClient wraps an instance of twisted.web.client.Agent. When you create an instance of HTTPClient, you must initialize it with an instance of Agent. This allows you to customize its behavior.

def make_custom_agent(reactor):
    return Agent(reactor, connectTimeout=42)

def main(reactor, *args):
    agent = make_custom_agent(reactor)
    http_client = HTTPClient(agent)
    d = http_client.get(
        'https://secure.example.net/area51',
        auth=('admin', "you'll never guess!"))
    d.addCallback(print_response)
    return d

react(main, [])

Full example: custom_agent.py