Skip to content

reqstorm

Send thousands of HTTP requests without writing the plumbing. Rate limits, retries, progress, and results straight to a file or a database, with one result per request.

Get started View on GitHub

$ python -m pip install reqstorm

import reqstorm

urls = [f"https://api.example.com/items/{i}" for i in range(7000)]

print(reqstorm.estimate(len(urls), rate_limit="100/min"))
# 7000 requests: about 1h 10m (limited by rate_limit)

results = reqstorm.fetch_all_sync(urls, rate_limit="100/min", retries=2, progress=True)
print(results.summary())
# {'total': 7000, 'ok': 6987, 'failed': 13, 'failures': {'HTTP 404': 9, 'TimeoutError': 4},
#  'latency': {'p50': 0.21, 'p95': 0.73, ...}, ...}

  • One result per request

    A timeout or a dropped connection is recorded on that request and never stops the others. Every attempt is kept in its history.

    Results and reports

  • Rate limits in any unit

    "10/s", "100/min", "1000/h" per host, or "auto" to follow the server's 429s and rate-limit headers. concurrency="auto" finds how many requests at once a server handles well. Know how long a batch takes before you start it.

    Rate limits

  • Retries that make sense

    Immediate retries with backoff and Retry-After, and retry rounds that resend what still failed at the end. 404s are not retried.

    Timeouts and retries

  • Straight to a file or a database

    JSONL, CSV, SQLite, PostgreSQL or MySQL, written as results arrive, so millions of results never fill your memory. Resume an interrupted run. With a schema, JSON fields go into typed, checked columns, with upserts on key fields.

    Files · Databases · Schemas

  • Every page of an API

    Follow next links, Link headers, cursors or page numbers, and turn every page into typed rows in the same call.

    Pagination · Requests from data

  • Tokens, proxies and caching

    Refresh an expired token on a 401 without stopping the run, rotate through proxies, and skip unchanged resources with ETags.

    Tokens, proxies and caching

  • A command for quick jobs

    reqstorm urls.txt -o results.jsonl --rate 100/min runs a batch with no Python code, and --report shows p50/p95/p99 response times per host.

    Command line

  • With or without asyncio

    Blocking functions for scripts and Jupyter notebooks, async functions for code that already runs an event loop. Same options for both.

    Getting started

  • Safe defaults

    TLS certificates verified, timeouts on by default, bounded concurrency. Fully typed, one dependency: aiohttp.

    Requests and TLS