API reference¶

Sending requests¶

fetch_all `async` ¶

fetch_all(
    urls: Iterable[Union[str, Request]],
    method: str = ...,
    *,
    headers: Optional[Mapping[str, str]] = ...,
    params: Optional[Mapping[str, Any]] = ...,
    json: Any = ...,
    data: Any = ...,
    concurrency: Union[int, str] = ...,
    max_concurrency: int = ...,
    timeout: Optional[float] = ...,
    retries: int = ...,
    backoff: float = ...,
    max_backoff: float = ...,
    jitter: bool = ...,
    retry_statuses: Sequence[int] = ...,
    retry_rounds: int = ...,
    retry_round_delay: float = ...,
    verify_ssl: bool = ...,
    ssl: Optional[SSLContext] = ...,
    rate_limit: Union[RateLimit, str, None] = ...,
    concurrency_per_host: int = ...,
    auth: Any = ...,
    cache: Any = ...,
    proxy: Union[str, Sequence[str], None] = ...,
    paginate: Any = ...,
    callback: Optional[Callback] = ...,
    progress: ProgressTarget = ...,
    total: Optional[int] = ...,
    log_level: LogLevel = ...,
    log_file: LogTarget = ...,
    log_format: str = ...,
    session: Optional[ClientSession] = ...,
) -> Results

fetch_all(
    urls: Iterable[str],
    method: Callable[..., Any],
    *,
    headers: Optional[Mapping[str, str]] = ...,
    verify_ssl: bool = ...,
    request_type: str = ...,
    semaphore_limit: int = ...,
) -> None

fetch_all(
    urls: Iterable[Union[str, Request]],
    method: Union[str, Callable[..., Any]] = "GET",
    *,
    headers: Optional[Mapping[str, str]] = None,
    params: Optional[Mapping[str, Any]] = None,
    json: Any = None,
    data: Any = None,
    concurrency: Union[int, str] = 100,
    max_concurrency: int = 500,
    timeout: Optional[float] = 30.0,
    retries: int = 0,
    backoff: float = 0.5,
    max_backoff: float = 30.0,
    jitter: bool = True,
    retry_statuses: Sequence[int] = DEFAULT_RETRY_STATUSES,
    retry_rounds: int = 0,
    retry_round_delay: float = 5.0,
    verify_ssl: bool = True,
    ssl: Optional[SSLContext] = None,
    rate_limit: Union[RateLimit, str, None] = None,
    concurrency_per_host: int = 0,
    auth: Any = None,
    cache: Any = None,
    proxy: Union[str, Sequence[str], None] = None,
    paginate: Any = None,
    callback: Optional[Callback] = None,
    progress: ProgressTarget = False,
    total: Optional[int] = None,
    log_level: LogLevel = None,
    log_file: LogTarget = None,
    log_format: str = "text",
    session: Optional[ClientSession] = None,
    **legacy: Any,
) -> Optional[Results]

Send all requests concurrently and return their results in the order given.

Parameters:

Name	Type	Description	Default
`urls`	`Iterable[Union[str, Request]]`	URLs, or `Request` objects for per-request method, headers, body or params.	required
`method`	`Union[str, Callable[..., Any]]`	HTTP method for plain URLs (default `"GET"`).	`'GET'`
`headers`	`Optional[Mapping[str, str]]`	Headers sent with every request; a `Request`'s own headers are merged on top.	`None`
`params`	`Optional[Mapping[str, Any]]`	Query parameters for every request, unless a `Request` sets its own.	`None`
`json`	`Any`	JSON body for every request, unless a `Request` sets its own.	`None`
`data`	`Any`	Form or raw body for every request, unless a `Request` sets its own.	`None`
`concurrency`	`Union[int, str]`	Maximum number of requests in flight at once, or `"auto"` to find the best number from the responses: it starts at 8, doubles while responses stay healthy, and backs off when servers answer 429/5xx, time out, drop connections or slow down to more than twice their normal response time. When the number is higher than the open file limit allows (one file per connection), the limit is raised if the system permits, else the number is lowered with a warning.	`100`
`max_concurrency`	`int`	With `concurrency="auto"`, the highest number it may reach (default 500).	`500`
`timeout`	`Optional[float]`	Seconds allowed per attempt, including reading the body. `None` disables it.	`30.0`
`retries`	`int`	How many times to retry a request right away after a connection error, a timeout or a status in `retry_statuses`. Invalid URLs are not retried.	`0`
`backoff`	`float`	Delay before the first retry in seconds, doubled for each further retry (0.5, 1, 2, 4, ... with the default).	`0.5`
`max_backoff`	`float`	Upper limit for that delay in seconds (default 30), so many retries never wait minutes. A `Retry-After` from the server is followed instead, up to 60 seconds.	`30.0`
`jitter`	`bool`	Wait a random time between half and all of the delay, so requests that failed together do not retry at the same moment. `False` waits exactly. A `Retry-After` header (in seconds, up to 60) takes precedence.	`True`
`retry_rounds`	`int`	After all requests have finished, send the ones that still failed for a retryable reason again, up to this many more rounds.	`0`
`retry_round_delay`	`float`	Seconds to wait before each retry round.	`5.0`
`verify_ssl`	`bool`	Verify TLS certificates (default). Only disable this for hosts you control.	`True`
`ssl`	`Optional[SSLContext]`	A custom `ssl.SSLContext`, for example to trust a private certificate authority.	`None`
`rate_limit`	`Union[RateLimit, str, None]`	Maximum request rate per host (`host:port`), retries included: requests per second as a number, or a string such as `"100/min"`, `"30/5min"` or `"1000/h"`, or `(count, seconds)`. `"auto"` follows the server instead: it slows down on 429 responses and `Retry-After`, spreads requests by `X-RateLimit-Remaining` / `X-RateLimit-Reset`, speeds back up when the server stops pushing back, and retries 429 responses (up to 10 times, not counted in `retries`).	`None`
`concurrency_per_host`	`int`	Maximum requests in flight to each host; 0 means no per-host limit.	`0`
`auth`	`Any`	`reqstorm.BearerAuth`: sends a token and refreshes it once on a 401.	`None`
`cache`	`Any`	`reqstorm.Cache`: reuses earlier GET responses, revalidating with ETag.	`None`
`proxy`	`Union[str, Sequence[str], None]`	A proxy URL, or a list of them used in turn, attempt by attempt: `"http://user:pass@host:port"`, or `socks5://`, `socks5h://` (the proxy resolves host names), `socks4://` and `socks4a://` with `pip install 'reqstorm[socks]'`. `Request(proxy=...)` sets one per request.	`None`
`paginate`	`Any`	`reqstorm.NextLink`, `LinkHeader`, `Cursor` or `PageNumber`: each starting URL is followed through its pages. Results carry `page` and `seed_index`. Cannot be combined with `retry_rounds`.	`None`
`callback`	`Optional[Callback]`	Called with each final `Result` as soon as it is known. May be a coroutine function.	`None`
`progress`	`ProgressTarget`	`True` prints a progress line to stderr every few seconds, even while no request finishes: done/total, ok, failed, requests in flight, retries, the rate over the last minute, paused hosts and the time left. A text stream prints it there instead; a function receives a `reqstorm.ProgressInfo`.	`False`
`total`	`Optional[int]`	Number of requests, for the percentage and the time left when `urls` is a generator (a list is counted automatically). Ignored with `paginate`, where the number of pages is not known in advance.	`None`
`log_level`	`LogLevel`	`"DEBUG"`, `"INFO"`, `"WARNING"` or `"ERROR"` to log this run on its own, whatever the application's logging configuration: to stderr, or to `log_file`. Leave it out to log through the standard `"reqstorm"` logger, which follows the application's configuration. See the logging guide.	`None`
`log_file`	`LogTarget`	With `log_level`: a file name, a name containing `{time}`, or a directory for time-stamped files. An existing file is never overwritten: `run.log` becomes `run-2.log`. A text stream also works.	`None`
`log_format`	`str`	`"text"` (default) or `"json"`, one object per line.	`'text'`
`session`	`Optional[ClientSession]`	An existing `aiohttp.ClientSession` to use instead of creating one.	`None`

Returns a Results list; results.errors() and results.summary() report the failures. A failing request never stops the others: its Result.error holds the exception and Result.history lists every attempt.

fetch_all_sync ¶

fetch_all_sync(
    urls: Iterable[Union[str, Request]],
    method: str = "GET",
    **options: Any,
) -> Results

Blocking version of fetch_all: send the requests and return their results in order.

Takes the same options as fetch_all except session, which belongs to an event loop.

results = reqstorm.fetch_all_sync(urls, concurrency=20, retries=2)

stream `async` ¶

stream(
    urls: Iterable[Union[str, Request]],
    method: str = "GET",
    *,
    headers: Optional[Mapping[str, str]] = None,
    params: Optional[Mapping[str, Any]] = None,
    json: Any = None,
    data: Any = None,
    concurrency: Union[int, str] = 100,
    max_concurrency: int = 500,
    timeout: Optional[float] = 30.0,
    retries: int = 0,
    backoff: float = 0.5,
    max_backoff: float = 30.0,
    jitter: bool = True,
    retry_statuses: Sequence[int] = DEFAULT_RETRY_STATUSES,
    verify_ssl: bool = True,
    ssl: Optional[SSLContext] = None,
    rate_limit: Union[RateLimit, str, None] = None,
    concurrency_per_host: int = 0,
    auth: Any = None,
    cache: Any = None,
    proxy: Union[str, Sequence[str], None] = None,
    paginate: Any = None,
    progress: ProgressTarget = False,
    total: Optional[int] = None,
    log_level: LogLevel = None,
    log_file: LogTarget = None,
    log_format: str = "text",
    session: Optional[ClientSession] = None,
    _run: Optional[Run] = None,
) -> AsyncGenerator[Result, None]

Send the requests concurrently and yield each Result as soon as it completes.

urls may be any iterable, including a generator; it is consumed lazily, so millions of requests can be streamed without building them all in memory. See fetch_all for the parameters. Retry rounds are not available here; use fetch_all or fetch_to_file for those.

stream_sync ¶

stream_sync(
    urls: Iterable[Union[str, Request]],
    method: str = "GET",
    **options: Any,
) -> Iterator[Result]

Blocking version of stream: yield each Result as soon as it completes.

The requests run on an event loop in a background thread. Leaving the loop early (break) stops the remaining requests.

for result in reqstorm.stream_sync(urls, concurrency=50):
    print(result.url, result.status)

Writing results¶

fetch_to_file `async` ¶

fetch_to_file(
    urls: Iterable[Union[str, Request]],
    path: Union[str, PathLike],
    *,
    format: Optional[str] = None,
    body: str = "text",
    include_headers: bool = False,
    resume: bool = False,
    ordered: bool = False,
    progress: ProgressTarget = False,
    retry_rounds: int = 0,
    retry_round_delay: float = 5.0,
    table: str = "reqstorm_results",
    batch_size: int = 100,
    flush_interval: float = 1.0,
    schema: Any = None,
    explode: Optional[str] = None,
    rejects_table: Optional[str] = None,
    include_source: bool = True,
    **options: Any,
) -> Summary

Send the requests and write one record per request to path as soon as it is final.

Results are not collected in memory, so this suits very large batches.

Parameters:

Name	Type	Description	Default
`path`	`Union[str, PathLike]`	`.csv` is written as CSV; `.db`, `.sqlite` and `.sqlite3` as an SQLite database (see `fetch_to_db`); anything else as JSON Lines.	required
`format`	`Optional[str]`	`"jsonl"`, `"csv"` or `"sqlite"` to override the choice made from the name.	`None`
`body`	`str`	`"text"` (decoded with the response charset), `"base64"`, `"bytes"` (SQLite only) or `"none"` to leave it out.	`'text'`
`include_headers`	`bool`	Also write the response headers.	`False`
`resume`	`bool`	Skip the requests already recorded as successful in an existing file or table; the rest (including earlier failures) are sent and appended. Without it a JSONL or CSV file is overwritten; database rows are never deleted.	`False`
`ordered`	`bool`	Write records in input order. By default they are written as they complete. Ordered output holds back records that finish early until the ones before them are done, so it uses more memory when a few requests are slow.	`False`
`progress`	`ProgressTarget`	As for `fetch_all`: a progress line, a stream, or a function.	`False`
`retry_rounds`	`int`	As for `fetch_all`; each request still gets one record.	`0`
`retry_round_delay`	`float`	Seconds to wait before each retry round.	`5.0`
`table`	`str`	Table name for SQLite output.	`'reqstorm_results'`
`batch_size`	`int`	Write as soon as this many records are waiting.	`100`
`flush_interval`	`float`	And at least every this many seconds while records are waiting (default 1), even when no request finishes, so other programs reading the output are never more than about a second behind. `0` writes every record at once. SQLite files are opened in WAL mode so they can be read meanwhile.	`1.0`
`schema`	`Any`	Write typed fields parsed from each JSON response instead of the raw response: a dict of column name to `reqstorm.Field`, a `reqstorm.Schema`, or a Pydantic v2 model (`pip install reqstorm[pydantic]`). See `fetch_to_db` and the "Structured data" guide.	`None`
`explode`	`Optional[str]`	With a schema, path to an array in each response; every element becomes a row.	`None`
`rejects_table`	`Optional[str]`	With a schema and SQLite output, also write rejected records to this table.	`None`
`include_source`	`bool`	With a schema, add `source_index`, `source_method` and `source_url` to each row (needed for `resume`).	`True`
`options`	`Any`	Any other option of `fetch_all`, such as `method`, `concurrency`, `timeout`, `retries` or `rate_limit`.	`{}`

Each record has index (position in urls), method, url, status, ok, error, attempts, elapsed, final_url and history (every attempt). Returns a Summary whose errors lists the failed records.

fetch_to_file_sync ¶

fetch_to_file_sync(
    urls: Iterable[Union[str, Request]],
    path: Any,
    **options: Any,
) -> Summary

Blocking version of fetch_to_file.

fetch_to_db `async` ¶

fetch_to_db(
    urls: Iterable[Union[str, Request]],
    connection: Any,
    *,
    table: str = "reqstorm_results",
    body: str = "text",
    include_headers: bool = False,
    resume: bool = False,
    ordered: bool = False,
    progress: ProgressTarget = False,
    retry_rounds: int = 0,
    retry_round_delay: float = 5.0,
    batch_size: int = 100,
    flush_interval: float = 1.0,
    schema: Any = None,
    explode: Optional[str] = None,
    rejects_table: Optional[str] = None,
    include_source: bool = True,
    **options: Any,
) -> Summary

Send the requests and insert one row per request into table.

connection is an open DB-API connection from sqlite3, psycopg / psycopg2 (PostgreSQL), or pymysql / MySQLdb / mysql.connector (MySQL). reqstorm does not close it. The table is created if it does not exist, with one column per field (request_index, method, url, status, ok, error, attempts, elapsed, final_url), JSON columns for history and headers, body, and created_at. Rows are inserted in batches on a background thread, except for sqlite3 connections, which must stay on the thread that created them.

Existing rows are never deleted. With resume=True, requests that already have a successful row are skipped. body="bytes" stores the raw body as a binary column. Rows are committed per batch and at least every flush_interval seconds. The other arguments, including logging, are as for fetch_to_file and fetch_all.

With schema, each JSON response is turned into typed rows instead: one column per field (BIGINT, DOUBLE PRECISION, TEXT, BOOLEAN, TIMESTAMPTZ or JSONB in PostgreSQL, and their equivalents), plus source_index, source_method and source_url unless include_source=False. explode turns each element of an array into a row. Records with a missing required field or a value of the wrong type (including NaN and Infinity) are not written: they are listed in Summary.errors and, with rejects_table, written to that table with their reasons. Fields marked key=True form the primary key, and a record whose key already exists updates that row. An existing table must already have the columns.

fetch_to_db_sync ¶

fetch_to_db_sync(
    urls: Iterable[Union[str, Request]],
    connection: Any,
    **options: Any,
) -> Summary

Blocking version of fetch_to_db.

The requests run on a helper thread when called from a running event loop; a sqlite3 connection must then be opened with check_same_thread=False.

Summary `dataclass` ¶

What fetch_to_file or fetch_to_db did.

rows `class-attribute` `instance-attribute` ¶

rows: int = 0

With a schema: rows written.

rejected `class-attribute` `instance-attribute` ¶

rejected: int = 0

With a schema: records rejected because a field was missing or had the wrong type.

report `class-attribute` `instance-attribute` ¶

report: Dict[str, Any] = field(default_factory=dict)

Response times, statuses, errors and per-host figures, as Results.report().

log_file `class-attribute` `instance-attribute` ¶

log_file: Optional[str] = None

The log file of this run, when log_file= was given.

Schemas¶

Field `dataclass` ¶

Where a column's value comes from in the JSON, and what type it must have.

Parameters:

Name	Type	Description	Default
`path`	`str`	Dotted path to the value, such as `"pricing.amount"` or `"tags.0"`. When exploding an array, paths are relative to each element; start a path with `"$."` to read from the root of the response instead (`"$.meta.page"`).	required
`type`	`FieldType`	`int`, `float`, `str`, `bool`, `"datetime"` or `"json"` (any JSON value, stored as JSON).	`'json'`
`required`	`bool`	Reject the record when the value is missing or null.	`False`
`coerce`	`bool`	Also accept compatible values of another type, such as `"12.5"` for a float or `1` for a bool. Without it, types must match exactly.	`False`
`key`	`bool`	Part of the table's primary key. Writing a record whose key already exists updates that row instead of adding a new one.	`False`

Schema `dataclass` ¶

Columns of a table and the fields they are read from.

Parameters:

Name	Type	Description	Default
`fields`	`Mapping[str, Field]`	Column name to `Field`.	required
`explode`	`Optional[str]`	Path to an array in the response; each element becomes a row (`"items"` or `"data.items"`; a trailing `"[*]"` is optional).	`None`

rows ¶

rows(
    document: Any,
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]

Rows and rejected records from one parsed JSON document.

A rejected record is {"record": <the JSON it came from>, "reasons": [...]}.

str ¶

__str__() -> str

The schema as Python code, ready to paste and edit.

infer_schema ¶

infer_schema(
    samples: Iterable[Any],
    *,
    explode: Optional[str] = None,
    max_depth: int = 3,
) -> Schema

Draft a schema from sample responses, for you to review and edit.

samples can be Result objects, parsed JSON documents, or JSON text. Nested objects become dotted paths (up to max_depth levels), arrays become "json" fields, and a field present and non-null in every sample is marked required. print(schema) shows it as Python code.

The draft only reflects the samples: a field that was always an integer there may be a float in other responses. Check it against the API's documentation.

extract ¶

extract(
    results: Iterable[Result],
    schema: Any,
    *,
    explode: Optional[str] = None,
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]

Typed rows from results you already have, for example from fetch_all.

Returns (rows, rejected). Each row is a dict of column to value plus source_index and source_url. Each rejected entry has source_index, source_url, reasons and the record it came from. Failed requests are rejected with the reason request failed.

SchemaError ¶

Bases: ValueError

A schema definition is invalid (not raised for invalid data, which is rejected instead).

Pagination¶

NextLink `dataclass` ¶

Bases: Paginator

The response JSON holds the next page's URL, for example {"links": {"next": "..."}}.

Parameters:

Name	Type	Description	Default
`path`	`str`	Dotted path to the URL (`"links.next"`, `"next"`, `"paging.next_url"`). A relative URL is resolved against the current page.	`'next'`
`max_pages`	`int`	Stop after this many pages per starting URL.	`1000`

LinkHeader `dataclass` ¶

Bases: Paginator

The next page is in the Link response header (RFC 8288), as GitHub and many APIs do.

Parameters:

Name	Type	Description	Default
`rel`	`str`	The relation to follow (default `"next"`).	`'next'`
`max_pages`	`int`	Stop after this many pages per starting URL.	`1000`

Cursor `dataclass` ¶

Bases: Paginator

The response JSON holds a cursor, sent back as a query parameter for the next page.

Parameters:

Name	Type	Description	Default
`path`	`str`	Dotted path to the cursor in the response (`"meta.next_cursor"`).	required
`param`	`str`	Query parameter to send it in (`"cursor"`).	`'cursor'`
`max_pages`	`int`	Stop after this many pages per starting URL.	`1000`

Pagination stops when the cursor is missing, null or an empty string.

PageNumber `dataclass` ¶

Bases: Paginator

Request ?page=1, ?page=2, ... until a page comes back empty.

Parameters:

Name	Type	Description	Default
`param`	`str`	Query parameter holding the page number (`"page"`).	`'page'`
`start`	`int`	Number of the first page, used when the starting URL has no such parameter.	`1`
`items`	`Optional[str]`	Dotted path to the array of results; pagination stops when it is empty or missing. Without it, it stops at the first non-2xx response.	`None`
`max_pages`	`int`	Stop after this many pages per starting URL.	`1000`

Paginator ¶

Base class: given a page's request and result, return the next page's request or None.

Requests from data¶

from_template ¶

from_template(
    url: str,
    rows: Iterable[Mapping[str, Any]],
    *,
    method: Optional[str] = None,
    json: BodyTemplate = None,
    params: BodyTemplate = None,
    headers: Optional[Mapping[str, str]] = None,
) -> Iterator[Request]

One Request per row, with {field} placeholders filled from the row.

Parameters:

Name	Type	Description	Default
`url`	`str`	URL template such as `"https://api.example.com/users/{id}/orders"`. Values are percent-encoded, so a value never changes the URL's structure.	required
`rows`	`Iterable[Mapping[str, Any]]`	Dicts, for example from `read_csv` or `read_sql`.	required
`method`	`Optional[str]`	HTTP method; the default of `fetch_all` when omitted.	`None`
`json`	`BodyTemplate`	A JSON body: a template dict whose `"{field}"` strings are filled from the row (a string that is only `"{field}"` keeps the value's type), or a function of the row.	`None`
`params`	`BodyTemplate`	Query parameters, in the same forms as `json`.	`None`
`headers`	`Optional[Mapping[str, str]]`	Headers sent with every request.	`None`

The result is a generator, so millions of rows are read lazily.

rows = reqstorm.read_csv("users.csv")
requests = reqstorm.from_template("https://api.example.com/users/{id}", rows)
reqstorm.fetch_to_file_sync(requests, "users.jsonl")

read_csv ¶

read_csv(
    path: str | PathLike[str], **options: Any
) -> Iterator[Dict[str, str]]

Rows of a CSV file as dicts keyed by the header row, read lazily.

Extra keyword arguments go to csv.DictReader (for example delimiter=";").

read_sql ¶

read_sql(
    connection: Any,
    query: str,
    parameters: Any = None,
    batch_size: int = 1000,
) -> Iterator[Dict[str, Any]]

Rows of a SQL query as dicts keyed by column name, fetched in batches.

Works with any DB-API connection: sqlite3, psycopg, psycopg2, PyMySQL, ...

Tokens and caching¶

BearerAuth ¶

Send Authorization: Bearer <token>, and get a new token when the server says it expired.

auth = reqstorm.BearerAuth(refresh=get_token)
reqstorm.fetch_all_sync(urls, auth=auth)

Parameters:

Name	Type	Description	Default
`token`	`Optional[str]`	The current token. Leave it out to fetch one with `refresh` before the first request.	`None`
`refresh`	`Optional[TokenSource]`	A function or coroutine function that returns a new token. It is called when a response is 401 Unauthorized; the request is then sent again once with the new token. Concurrent 401s share a single refresh.	`None`
`header`	`str`	Header to send the token in (default `Authorization`).	`'Authorization'`
`scheme`	`str`	Prefix before the token (default `Bearer`); `""` sends the token alone.	`'Bearer'`

generation `property` ¶

generation: int

Changes each time the token is refreshed.

refresh `async` ¶

refresh(seen_generation: int) -> None

Get a new token, unless another request already did since seen_generation.

Cache ¶

Keep successful GET responses on disk and reuse them on the next run.

Parameters:

Name	Type	Description	Default
`path`	`str \| PathLike[str]`	SQLite file to keep the responses in (created if missing).	`'reqstorm-cache.sqlite'`
`ttl`	`Optional[float]`	Seconds a cached response is used without asking the server. `None` (the default) always asks, but with `If-None-Match` / `If-Modified-Since`, so an unchanged resource costs a 304 with no body. `0` behaves the same.	`None`

Only GET and HEAD responses with status 200 are cached. A result served from the cache has from_cache=True. Close the cache when done, or use it in a with block.

with reqstorm.Cache("responses.sqlite", ttl=3600) as cache:
    reqstorm.fetch_all_sync(urls, cache=cache)

touch ¶

touch(key: str) -> None

Mark a revalidated response as fresh again.

Requests and results¶

Request `dataclass` ¶

A single request. Fields left as None take the defaults given to fetch_all.

Result `dataclass` ¶

The outcome of one request: a response, or the error that prevented one.

from_cache `class-attribute` `instance-attribute` ¶

from_cache: bool = False

True when the response came from the cache (fresh, or revalidated with a 304).

page `class-attribute` `instance-attribute` ¶

page: int = 0

With paginate: 0 for the starting URL, 1 for the next page, and so on.

seed_index `class-attribute` `instance-attribute` ¶

seed_index: Optional[int] = None

Position in the input of the starting URL this result belongs to.

ok `property` ¶

ok: bool

True when a response arrived and its status is below 400.

raise_for_error ¶

raise_for_error() -> None

Raise the request's error, or HTTPStatusError for a status of 400 or above.

to_dict ¶

to_dict(
    body: str = "none", include_headers: bool = False
) -> Dict[str, Any]

A JSON-serialisable summary, as written by fetch_to_file.

Parameters:

Name	Type	Description	Default
`body`	`str`	`"none"` (default), `"text"` or `"base64"`.	`'none'`
`include_headers`	`bool`	Include the response headers.	`False`

Results ¶

Bases: List[Result]

The list returned by fetch_all, in input order, with helpers for reporting.

errors ¶

errors() -> List[Dict[str, Any]]

Every failed request as a dict: index, method, url, status, error, attempts and history.

summary ¶

summary() -> Dict[str, Any]

Counts of the batch: total, ok, failed, failures grouped by status or error type, response times (latency: min, p50, p90, p95, p99, max, mean in seconds), statuses and per-host figures (hosts). See report() for the full set.

report ¶

report() -> Dict[str, Any]

Response times (p50, p90, p95, p99), statuses, errors and per-host figures.

Attempt `dataclass` ¶

One try at a request: the status it got or the error it raised.

HTTPStatusError ¶

Bases: Exception

Raised by Result.raise_for_error for a response status of 400 or above.

Progress¶

ProgressInfo `dataclass` ¶

A snapshot of a running batch, passed to progress= when it is a function.

The function is called every few seconds while the batch runs, even when no request finishes, and once more at the end with finished=True.

total `instance-attribute` ¶

total: Optional[int]

Number of requests, or None when the input is a generator of unknown length.

done `instance-attribute` ¶

done: int

Requests with a final result (after all their retries).

in_flight `instance-attribute` ¶

in_flight: int

Requests being sent, waiting for a response, or waiting to retry right now.

retries `instance-attribute` ¶

retries: int

Retries so far, of all requests.

rate `instance-attribute` ¶

rate: float

Finished requests per second over the last minute.

eta `instance-attribute` ¶

eta: Optional[float]

Estimated seconds left, or None when unknown.

round `class-attribute` `instance-attribute` ¶

round: int = 0

Retry round being sent (0 for the first pass).

paused `class-attribute` `instance-attribute` ¶

paused: Dict[str, float] = field(default_factory=dict)

Hosts the server asked to wait (rate_limit="auto"), with seconds left.

concurrency `class-attribute` `instance-attribute` ¶

concurrency: Optional[int] = None

With concurrency="auto": the number of requests currently allowed in flight.

Planning¶

estimate ¶

estimate(
    requests: int,
    *,
    rate_limit: Optional[RateLimit] = None,
    hosts: int = 1,
    concurrency: int = 100,
    concurrency_per_host: int = 0,
    latency: float = 0.5,
) -> Estimate

Estimate the duration of a batch, without sending anything.

>>> print(reqstorm.estimate(7000, rate_limit="100/min"))
7000 requests: about 1h 10m (limited by rate_limit)

Parameters:

Name	Type	Description	Default
`requests`	`int`	Number of requests.	required
`rate_limit`	`Optional[RateLimit]`	The per-host rate limit you plan to use (same formats as `fetch_all`).	`None`
`hosts`	`int`	How many different hosts the requests are spread evenly over.	`1`
`concurrency`	`int`	The overall concurrency you plan to use.	`100`
`concurrency_per_host`	`int`	The per-host concurrency you plan to use; 0 means none.	`0`
`latency`	`float`	Typical seconds per request (response time); 0.5 by default.	`0.5`

Retries are not included; they add time for requests that fail.

Estimate `dataclass` ¶

How long requests requests should take, and which setting limits them.

parse_rate ¶

parse_rate(rate: RateLimit) -> float

Requests per second from 5, "100/min", "30/5min", "1000/h" or (100, 60).

API reference¶

Sending requests¶

fetch_all async ¶

fetch_all_sync ¶

stream async ¶

stream_sync ¶

Writing results¶

fetch_to_file async ¶

fetch_to_file_sync ¶

fetch_to_db async ¶

fetch_to_db_sync ¶

Summary dataclass ¶

rows class-attribute instance-attribute ¶

rejected class-attribute instance-attribute ¶

report class-attribute instance-attribute ¶

log_file class-attribute instance-attribute ¶

Schemas¶

Field dataclass ¶

Schema dataclass ¶

rows ¶

__str__ ¶

infer_schema ¶

extract ¶

SchemaError ¶

Pagination¶

NextLink dataclass ¶

LinkHeader dataclass ¶

Cursor dataclass ¶

PageNumber dataclass ¶

Paginator ¶

Requests from data¶

from_template ¶

read_csv ¶

read_sql ¶

Tokens and caching¶

BearerAuth ¶

generation property ¶

refresh async ¶

Cache ¶

touch ¶

Requests and results¶

Request dataclass ¶

Result dataclass ¶

from_cache class-attribute instance-attribute ¶

page class-attribute instance-attribute ¶

seed_index class-attribute instance-attribute ¶

ok property ¶

raise_for_error ¶

to_dict ¶

Results ¶

errors ¶

summary ¶

report ¶

Attempt dataclass ¶

HTTPStatusError ¶

Progress¶

ProgressInfo dataclass ¶

total instance-attribute ¶

done instance-attribute ¶

in_flight instance-attribute ¶

retries instance-attribute ¶

rate instance-attribute ¶

eta instance-attribute ¶

round class-attribute instance-attribute ¶

paused class-attribute instance-attribute ¶

concurrency class-attribute instance-attribute ¶

Planning¶

estimate ¶

Estimate dataclass ¶

parse_rate ¶

fetch_all `async` ¶

stream `async` ¶

fetch_to_file `async` ¶

fetch_to_db `async` ¶

Summary `dataclass` ¶

rows `class-attribute` `instance-attribute` ¶

rejected `class-attribute` `instance-attribute` ¶

report `class-attribute` `instance-attribute` ¶

log_file `class-attribute` `instance-attribute` ¶

Field `dataclass` ¶

Schema `dataclass` ¶

str ¶

NextLink `dataclass` ¶

LinkHeader `dataclass` ¶

Cursor `dataclass` ¶

PageNumber `dataclass` ¶

generation `property` ¶

refresh `async` ¶

Request `dataclass` ¶

Result `dataclass` ¶

from_cache `class-attribute` `instance-attribute` ¶

page `class-attribute` `instance-attribute` ¶

seed_index `class-attribute` `instance-attribute` ¶

ok `property` ¶

Attempt `dataclass` ¶

ProgressInfo `dataclass` ¶

total `instance-attribute` ¶

done `instance-attribute` ¶

in_flight `instance-attribute` ¶

retries `instance-attribute` ¶

rate `instance-attribute` ¶

eta `instance-attribute` ¶

round `class-attribute` `instance-attribute` ¶

paused `class-attribute` `instance-attribute` ¶

concurrency `class-attribute` `instance-attribute` ¶

Estimate `dataclass` ¶