API Reference¶
This page lists all of the interfaces exposed by the treq package.
Making Requests¶
The treq
module provides several convenience functions for making requests.
These functions all create a default treq.client.HTTPClient
instance and pass their arguments to the appropriate HTTPClient
method.
- treq.request(method, url, **kwargs)[source]¶
Make an HTTP request.
- Parameters:
method (str) – HTTP method. Example:
'GET'
,'HEAD'
.'PUT'
,'POST'
.url (
hyperlink.DecodedURL
, str, bytes, orhyperlink.EncodedURL
) – http or https URL, which may include query arguments.headers (
Headers
or None) – Optional HTTP Headers to send with this request.params (dict w/ str or list/tuple of str values, list of 2-tuples, or None.) – Optional parameters to be append to the URL query string. Any query string parameters in the url will be preserved.
data (bytes, typing.BinaryIO, IBodyProducer, or None) –
Arbitrary request body data.
If files is also passed this must be a
dict
, atuple
orlist
of field tuples as accepted byMultiPartProducer
. The request is assigned a Content-Type ofmultipart/form-data
.If a
dict
,list
, ortuple
it is URL-encoded and the request assigned a Content-Type ofapplication/x-www-form-urlencoded
.Otherwise, any non-
None
value is passed to the client’s data_to_body_producer callable (by default,IBodyProducer
), which acceptsbytes
and binary files like returned byopen(..., "rb")
.files –
Files to include in the request body, in any of the several formats:
[("fieldname", binary_file)]
[("fieldname", "filename", binary_file)]
[("fieldname, "filename', "content-type", binary_file)]
Or a mapping:
{"fieldname": binary_file}
{"fieldname": ("filename", binary_file)}
{"fieldname": ("filename", "content-type", binary_file)}
Each
binary_file
is a file-like object open in binary mode (like returned byopen("filename", "rb")
). The filename is taken from the file’sname
attribute if not specified. The Content-Type is guessed based on the filename usingmimetypes.guess_type()
if not specified, falling back toapplication/octet-stream
.While uploading Treq will measure the length of seekable files to populate the Content-Length header of the file part.
If files is given the request is assigned a Content-Type of
multipart/form-data
. Additional fields may be given in the data argument.json (dict, list, tuple, int, str, bool, or None) – Optional JSON-serializable content for the request body. Mutually exclusive with data and files.
auth (tuple of
('username', 'password')
) – HTTP Basic Authentication information — seetreq.auth.add_auth()
.cookies (
dict
orcookielib.CookieJar
) – Cookies to send with this request. The HTTP kind, not the tasty kind.timeout (int) – Request timeout seconds. If a response is not received within this timeframe, a connection is aborted with
CancelledError
.allow_redirects (bool) – Follow HTTP redirects. Default:
True
browser_like_redirects (bool) – Follow redirects like a web browser: When a 301 or 302 redirect is received in response to a POST request convert the method to GET. See 7231 and
BrowserLikeRedirectAgent
). Default:False
unbuffered (bool) – Pass
True
to to disable response buffering. By default treq buffers the entire response body in memory.reactor – Optional Twisted reactor.
persistent (bool) – Use persistent HTTP connections. Default:
True
agent (twisted.web.iweb.IAgent) – Provide your own custom agent. Use this to override things like
connectTimeout
orBrowserLikePolicyForHTTPS
. By default, treq will create its own Agent with reasonable defaults.
- Return type:
Deferred that fires with an
IResponse
Changed in version treq: 20.9.0
The url param now accepts
hyperlink.DecodedURL
andhyperlink.EncodedURL
objects.
- treq.get(url, headers=None, **kwargs)[source]¶
Make a
GET
request.See
treq.request()
- treq.head(url, **kwargs)[source]¶
Make a
HEAD
request.See
treq.request()
- treq.post(url, data=None, **kwargs)[source]¶
Make a
POST
request.See
treq.request()
- treq.put(url, data=None, **kwargs)[source]¶
Make a
PUT
request.See
treq.request()
- treq.patch(url, data=None, **kwargs)[source]¶
Make a
PATCH
request.See
treq.request()
- treq.delete(url, **kwargs)[source]¶
Make a
DELETE
request.See
treq.request()
Accessing Content¶
- treq.collect(response: <InterfaceClass twisted.web.iweb.IResponse>, collector: ~typing.Callable[[bytes], None]) Deferred[None] [source]¶
Incrementally collect the body of the response.
This function may only be called once for a given response.
If the
collector
raises an exception, it will be set as the error value on responseDeferred
returned from this function, and the underlying HTTP transport will be closed.- Parameters:
response (IResponse) – The HTTP response to collect the body from.
collector (single argument callable) – A callable to be called each time data is available from the response body.
- Return type:
Deferred that fires with None when the entire body has been read.
- treq.content(response: <InterfaceClass twisted.web.iweb.IResponse>) Deferred[bytes] [source]¶
Read the contents of an HTTP response.
This function may be called multiple times for a response, it uses a
WeakKeyDictionary
to cache the contents of the response.- Parameters:
response (IResponse) – The HTTP Response to get the contents of.
- Return type:
Deferred that fires with the content as a str.
- treq.text_content(response: <InterfaceClass twisted.web.iweb.IResponse>, encoding: str = 'ISO-8859-1') Deferred[str] [source]¶
Read the contents of an HTTP response and decode it with an appropriate charset, which may be guessed from the
Content-Type
header.- Parameters:
response (IResponse) – The HTTP Response to get the contents of.
encoding (str) – A charset, such as
UTF-8
orISO-8859-1
, used if the response does not specify an encoding.
- Return type:
Deferred that fires with a unicode string.
- treq.json_content(response: <InterfaceClass twisted.web.iweb.IResponse>, **kwargs: ~typing.Any) Deferred[Any] [source]¶
Read the contents of an HTTP response and attempt to decode it as JSON.
This function relies on
content()
and so may be called more than once for a given response.- Parameters:
response (IResponse) – The HTTP Response to get the contents of.
kwargs – Any keyword arguments accepted by
json.loads()
- Return type:
Deferred that fires with the decoded JSON.
The HTTP Client¶
treq.client.HTTPClient
has methods that match the signatures of the convenience request functions in the treq
module.
- class treq.client.HTTPClient(agent, cookiejar=None, data_to_body_producer=IBodyProducer)[source]¶
- request(method: str, url: str | bytes | ~hyperlink._url.URL | ~hyperlink._url.DecodedURL, *, params: ~typing.Mapping[str, str | ~typing.Tuple[str, ...] | ~typing.List[str]] | ~typing.List[~typing.Tuple[str, str]] | None = None, headers: ~twisted.web.http_headers.Headers | ~typing.Dict[bytes | str, bytes | str] | ~typing.Dict[bytes | str, ~typing.List[bytes | str]] | None = None, data: bytes | ~_io.BytesIO | ~_io.BufferedReader | <InterfaceClass twisted.web.iweb.IBodyProducer> | ~typing.Dict[str, str] | ~typing.List[~typing.Tuple[str, str]] | None = None, files: ~typing.Mapping[str, str | bytes | ~typing.Tuple[str, str, <InterfaceClass twisted.web.iweb.IBodyProducer>]] | ~typing.Iterable[~typing.Tuple[str, str | bytes | ~typing.Tuple[str, str, <InterfaceClass twisted.web.iweb.IBodyProducer>]]] | None = None, json: ~typing.Any | ~treq.client._Nothing = <treq.client._Nothing object>, auth: ~typing.Tuple[bytes | str, bytes | str] | None = None, cookies: ~http.cookiejar.CookieJar | ~typing.Mapping[str, str] | None = None, allow_redirects: bool = True, browser_like_redirects: bool = False, unbuffered: bool = False, reactor: <InterfaceClass treq._types._ITreqReactor> | None = None, timeout: float | None = None, _stacklevel: int = 2) Deferred[_Response] [source]¶
See
treq.request()
.
- get(url: str | bytes | URL | DecodedURL, **kwargs: Any) Deferred[_Response] [source]¶
See
treq.get()
.
- head(url: str | bytes | URL | DecodedURL, **kwargs: Any) Deferred[_Response] [source]¶
See
treq.head()
.
- post(url: str | bytes | ~hyperlink._url.URL | ~hyperlink._url.DecodedURL, data: bytes | ~_io.BytesIO | ~_io.BufferedReader | <InterfaceClass twisted.web.iweb.IBodyProducer> | ~typing.Dict[str, str] | ~typing.List[~typing.Tuple[str, str]] | None = None, **kwargs: ~typing.Any) Deferred[_Response] [source]¶
See
treq.post()
.
- put(url: str | bytes | ~hyperlink._url.URL | ~hyperlink._url.DecodedURL, data: bytes | ~_io.BytesIO | ~_io.BufferedReader | <InterfaceClass twisted.web.iweb.IBodyProducer> | ~typing.Dict[str, str] | ~typing.List[~typing.Tuple[str, str]] | None = None, **kwargs: ~typing.Any) Deferred[_Response] [source]¶
See
treq.put()
.
- patch(url: str | bytes | ~hyperlink._url.URL | ~hyperlink._url.DecodedURL, data: bytes | ~_io.BytesIO | ~_io.BufferedReader | <InterfaceClass twisted.web.iweb.IBodyProducer> | ~typing.Dict[str, str] | ~typing.List[~typing.Tuple[str, str]] | None = None, **kwargs: ~typing.Any) Deferred[_Response] [source]¶
See
treq.patch()
.
Augmented Response Objects¶
treq.request()
, treq.get()
, etc. return an object which provides twisted.web.iweb.IResponse
, plus a few additional convenience methods:
- class treq.response._Response[source]¶
- collect(collector)[source]¶
Incrementally collect the body of the response, per
treq.collect()
.- Parameters:
collector – A single argument callable that will be called with chunks of body data as it is received.
- Returns:
A Deferred that fires when the entire body has been received.
- content()[source]¶
Read the entire body all at once, per
treq.content()
.- Returns:
A Deferred that fires with a bytes object when the entire body has been received.
- json(**kwargs)[source]¶
Collect the response body as JSON per
treq.json_content()
.- Parameters:
kwargs – Any keyword arguments accepted by
json.loads()
- Return type:
Deferred that fires with the decoded JSON when the entire body has been read.
- text(encoding='ISO-8859-1')[source]¶
Read the entire body all at once as text, per
treq.text_content()
.- Return type:
A Deferred that fires with a unicode string when the entire body has been received.
- history()[source]¶
Get a list of all responses that (such as intermediate redirects), that ultimately ended in the current response. The responses are ordered chronologically.
- Returns:
A list of
_Response
objects
- cookies()[source]¶
Get a copy of this response’s cookies.
- Return type:
requests.cookies.RequestsCookieJar
Inherited from
twisted.web.iweb.IResponse
:- Variables:
version – See
IResponse.version
code – See
IResponse.code
phrase – See
IResponse.phrase
headers – See
IResponse.headers
length – See
IResponse.length
request – See
IResponse.request
previousResponse – See
IResponse.previousResponse
- deliverBody(protocol)¶
- setPreviousResponse(response)¶
Authentication¶
- treq.auth.add_auth(agent, auth_config)[source]¶
Wrap an agent to perform authentication
- Parameters:
agent – Agent to wrap.
auth_config – A
('username', 'password')
tuple — seeadd_basic_auth()
.
- Returns:
- Raises:
UnknownAuthConfig – When the format auth_config isn’t supported.
- treq.auth.add_basic_auth(agent: <InterfaceClass twisted.web.iweb.IAgent>, username: str | bytes, password: str | bytes) <InterfaceClass twisted.web.iweb.IAgent> [source]¶
Wrap an agent to add HTTP basic authentication
The returned agent sets the Authorization request header according to the basic authentication scheme described in RFC 7617. This header contains the given username and password in plaintext, and thus should only be used over an encrypted transport (HTTPS).
Note that the colon (
:
) is used as a delimiter between the username and password, so if either parameter includes a colon the interpretation of the Authorization header is server-defined.- Parameters:
agent – Agent to wrap.
username – The username.
password – The password.
- Returns:
Test Helpers¶
The treq.testing
module contains tools for in-memory testing of HTTP clients and servers.
StubTreq Objects¶
- class treq.testing.StubTreq(resource)¶
StubTreq
implements the same interface as thetreq
module or theHTTPClient
class, with the limitation that it does not support thefiles
argument.- flush()¶
Flush all data between pending client/server pairs.
This is only necessary if a
Resource
under test returnsNOT_DONE_YET
from itsrender
method, making a response asynchronous. In that case, after each write from the server,flush()
must be called so the client can see it.
As the methods on
treq.client.HTTPClient
:- request()¶
See
treq.request()
.
- get()¶
See
treq.get()
.
- head()¶
See
treq.head()
.
- post()¶
See
treq.post()
.
- put()¶
See
treq.put()
.
- patch()¶
See
treq.patch()
.
- delete()¶
See
treq.delete()
.
RequestTraversalAgent Objects¶
- class treq.testing.RequestTraversalAgent(rootResource)[source]¶
IAgent
implementation that issues an in-memory request rather than going out to a real network socket.
RequestSequence Objects¶
- class treq.testing.RequestSequence(sequence, async_failure_reporter=None)[source]¶
For an example usage, see
RequestSequence.consume()
.Takes a sequence of:
[((method, url, params, headers, data), (code, headers, body)), ...]
Expects the requests to arrive in sequence order. If there are no more responses, or the request’s parameters do not match the next item’s expected request parameters, calls sync_failure_reporter or async_failure_reporter.
For the expected request tuples:
method
should bebytes
normalized to lowercase.url
should be a str normalized as per the transformations in that (usually) preserve semantics. A URL to http://something-that-looks-like-a-directory would be normalized to http://something-that-looks-like-a-directory/ and a URL to http://something-that-looks-like-a-page/page.html remains unchanged.headers
is a dictionary mappingbytes
tolist
ofbytes
– note thattwisted.web.client.Agent
may add its own headers which are not guaranteed to be present (for instance, user-agent or content-length), so it’s better to use some kind of matcher likeHasHeaders
.data
is abytes
.
For the response tuples:
code
is an integer representing the HTTP status code to return.headers
is a dictionary mappingbytes
tobytes
orstr
. Note that the value is not a list.body
is abytes
.
- Variables:
sequence (list) – A sequence of (request tuple, response tuple) two-tuples, as described above.
async_failure_reporter – An optional callable that takes a
str
message indicating a failure. It’s asynchronous because it cannot just raise an exception—if it does,Resource.render
will just convert that into a 500 response, and there will be no other failure reporting mechanism.
When the async_failure_reporter parameter is not passed, async failures will be reported via a
twisted.logger.Logger
instance, which Trial’s test case classes (twisted.trial.unittest.TestCase
andSynchronousTestCase
) will translate into a test failure.Note
Some versions of
twisted.trial.unittest.SynchronousTestCase
report logged errors on the wrong test: see Twisted #9267.When not subclassing Trial’s classes you must pass async_failure_reporter and implement equivalent behavior or errors will pass silently. For example:
async_failures = [] sequence_stubs = RequestSequence([...], async_failures.append) stub_treq = StubTreq(StringStubbingResource(sequence_stubs)) with sequence_stubs.consume(self.fail): # self = unittest.TestCase stub_treq.get('http://fakeurl.com') self.assertEqual([], async_failures)
- consume(sync_failure_reporter)[source]¶
Usage:
sequence_stubs = RequestSequence([...]) stub_treq = StubTreq(StringStubbingResource(sequence_stubs)) # self = twisted.trial.unittest.SynchronousTestCase with sequence_stubs.consume(self.fail): stub_treq.get('http://fakeurl.com') stub_treq.get('http://another-fake-url.com')
If there are still remaining expected requests to be made in the sequence, fails the provided test case.
- Parameters:
sync_failure_reporter – A callable that takes a single message reporting failures. This can just raise an exception - it does not need to be asynchronous, since the exception would not get raised within a Resource.
- Returns:
a context manager that can be used to ensure all expected requests have been made.
StringStubbingResource Objects¶
- class treq.testing.StringStubbingResource(get_response_for)[source]¶
A resource that takes a callable with 5 parameters
(method, url, params, headers, data)
and returns(code, headers, body)
.The resource uses the callable to return a real response as a result of a request.
The parameters for the callable are:
method
, the HTTP method as bytes.url
, the full URL of the request as text.params
, a dictionary of query parameters mapping query keys lists of values (sorted alphabetically).headers
, a dictionary of headers mapping header keys to a list of header values (sorted alphabetically).data
, the request body as bytes.
The callable must return a
tuple
of (code, headers, body) where the code is the HTTP status code, the headers is a dictionary of bytes (unlike the headers parameter, which is a dictionary of lists), and body is a string that will be returned as the response body.If there is a stubbing error, the return value is undefined (if an exception is raised,
Resource
will just eat it and return 500 in its place). The callable, or whomever creates the callable, should have a way to handle error reporting.
HasHeaders Objects¶
- class treq.testing.HasHeaders(headers)[source]¶
Since Twisted adds headers to a request, such as the host and the content length, it’s necessary to test whether request headers CONTAIN the expected headers (the ones that are not automatically added by Twisted).
This wraps a set of headers, and can be used in an equality test against a superset if the provided headers. The headers keys are lowercased, and keys and values are compared in their bytes-encoded forms.
Headers should be provided as a mapping from strings or bytes to a list of strings or bytes.
MultiPartProducer Objects¶
treq.multipart.MultiPartProducer
is used internally when making requests which involve files.
- class treq.multipart.MultiPartProducer(fields: ~typing.Mapping[str, str | bytes | ~typing.Tuple[str, str, <InterfaceClass twisted.web.iweb.IBodyProducer>]] | ~typing.Iterable[~typing.Tuple[str, str | bytes | ~typing.Tuple[str, str, <InterfaceClass twisted.web.iweb.IBodyProducer>]]], boundary: str | bytes | None = None, cooperator: ~twisted.internet.task.Cooperator = <module 'twisted.internet.task' from '/home/docs/checkouts/readthedocs.org/user_builds/treq/envs/latest/lib/python3.11/site-packages/twisted/internet/task.py'>)[source]¶
MultiPartProducer
takes parameters for a HTTP request and produces bytes in multipart/form-data format defined in RFC 2388 and RFC 2046.The encoded request is produced incrementally and the bytes are written to a consumer.
Fields should have form:
[(parameter name, value), ...]
Accepted values:
Unicode strings (in this case parameter will be encoded with utf-8)
Tuples with (file name, content-type,
IBodyProducer
objects)
Since
MultiPartProducer
can accept objects likeIBodyProducer
which cannot be read from in an event-driven manner it uses uses aCooperator
instance to schedule reads from the underlying producers. Reading is also paused and resumed based on notifications from theIConsumer
provider being written to.- Variables:
_fields – Sorted parameters, where all strings are enforced to be unicode and file objects stacked on bottom (to produce a human readable form-data request)
_cooperate – A method like Cooperator.cooperate which is used to schedule all reads.
boundary – The generated boundary used in form-data encoding
- pauseProducing() None [source]¶
Temporarily suspend copying bytes from the input file to the consumer by pausing the CooperativeTask which drives that activity.
- resumeProducing() None [source]¶
Undo the effects of a previous pauseProducing and resume copying bytes to the consumer by resuming the CooperativeTask which drives the write activity.
- startProducing(consumer: <InterfaceClass twisted.internet.interfaces.IConsumer>) Deferred[None] [source]¶
Start a cooperative task which will read bytes from the input file and write them to consumer. Return a Deferred which fires after all bytes have been written.
- Parameters:
consumer – Any IConsumer provider