Self-hosted Playwright scraping cluster on a VPS: Docker, Redis queue, retries and dead-letter handling

Managed scraping APIs are the fastest way to start, but you pay for every page forever. Once you scrape the same kinds of pages every day, a small cluster on your own server can do the same job for the price of the VPS. This guide builds one: Redis as the queue, Playwright workers in Docker, retries with backoff, and a dead-letter queue for jobs that can't succeed. Everything runs on a single 4 GB VPS and scales out by adding servers.

When self-hosting makes sense

• Good fit: recurring jobs on public pages (prices, stock, catalog changes), custom browser flows such as clicking through variants, and volumes of tens of thousands of pages per month or more.
• Poor fit: sites behind aggressive bot protection, one-off jobs, or teams without anyone to maintain a server. A managed API is cheaper than your time in those cases. See scraping tools compared.
• Always: read each target's robots.txt and terms, scrape only public data you're allowed to collect, and keep request rates low. The queue manager below skips URLs that robots.txt disallows, and workers enforce a per-domain rate limit.

Architecture

Cluster architecture: one VPS, one queue, many browser pages

Why these pieces:

• Redis Streams with a consumer group give each job to exactly one worker and keep it in a pending list until the worker acknowledges it. If a worker dies, nothing is lost.
• One browser per container, several pages per browser. A Chromium process costs memory before it loads anything. Sharing one browser across a few isolated contexts is far cheaper than launching a browser per page.
• A sorted set for delayed retries. Failed jobs wait there with a timestamp and return to the stream when their backoff expires, without blocking a worker slot.
• A dead-letter stream holds jobs that failed permanently or ran out of attempts, with the last error, so you can inspect and requeue them.

The job lifecycle

What happens to one job

1. Server setup

Any VPS with 4 GB of RAM and 2 vCPUs works. Two common choices are Vultr's Regular Cloud Compute (2 vCPU, 4 GB, $20/month at the time of writing; confirm the plan is still offered) and DigitalOcean's Basic Droplet (2 vCPU, 4 GB, $24/month or $0.03571/hour, 4,000 GiB transfer included). Pick a region close to your target sites.

# Ubuntu 24.04: Docker Engine + Compose plugin, a firewall, and 2 GB of swap as a safety net
curl -fsSL https://get.docker.com | sh
ufw allow OpenSSH && ufw --force enable
fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

mkdir -p /opt/scraper && cd /opt/scraper
echo "REDIS_PASSWORD=$(openssl rand -hex 24)" > .env   # hex only, so it is safe inside a URL
chmod 600 .env

Docker bypasses ufw for published ports, which is why the compose file below binds Redis to 127.0.0.1 only. Never publish Redis on a public interface.

2. The code

Five files in /opt/scraper: docker-compose.yml, Dockerfile, worker.py, enqueue.py and export.py.

docker-compose.yml

services:
  redis:
    image: redis:7.4-alpine
    command: ["redis-server", "--appendonly", "yes", "--maxmemory", "256mb",
              "--maxmemory-policy", "noeviction", "--requirepass", "${REDIS_PASSWORD}"]
    environment:
      REDIS_PASSWORD: ${REDIS_PASSWORD}
    volumes:
      - redis-data:/data
    ports:
      - "127.0.0.1:6379:6379"          # reachable from the host only
    healthcheck:
      test: ["CMD-SHELL", "redis-cli --no-auth-warning -a \"$$REDIS_PASSWORD\" ping | grep -q PONG"]
      interval: 5s
      retries: 10
    restart: unless-stopped

  worker:
    build: .
    init: true                          # reaps orphaned Chromium processes
    shm_size: "1gb"                     # Chromium needs more than Docker's 64 MB default
    environment:
      REDIS_URL: redis://:${REDIS_PASSWORD}@redis:6379/0
      CONCURRENCY: "3"                  # pages open at once per worker
      MAX_ATTEMPTS: "4"
      NAV_TIMEOUT_MS: "30000"
      RECYCLE_AFTER: "500"              # restart the browser after this many jobs
      USER_AGENT: ${USER_AGENT:-}
      PROXY_SERVER: ${PROXY_SERVER:-}
      PROXY_USERNAME: ${PROXY_USERNAME:-}
      PROXY_PASSWORD: ${PROXY_PASSWORD:-}
    depends_on:
      redis:
        condition: service_healthy
    deploy:
      replicas: 2
      resources:
        limits:
          memory: 1400M
          cpus: "1.0"
    restart: unless-stopped

volumes:
  redis-data:

noeviction matters: with any other policy, Redis would silently delete queued jobs when memory fills up. With noeviction, writes fail loudly instead.

Dockerfile

FROM mcr.microsoft.com/playwright/python:v1.48.0-jammy
WORKDIR /app
# keep the playwright package version identical to the image tag
RUN pip install --no-cache-dir playwright==1.48.0 redis==5.0.8
COPY worker.py enqueue.py export.py ./
USER pwuser
CMD ["python", "worker.py"]

worker.py

"""Playwright worker: pulls jobs from a Redis stream, retries with backoff, dead-letters failures."""
import asyncio, json, os, random, signal, socket, time
from urllib.parse import urlsplit

import redis.asyncio as redis
from playwright.async_api import async_playwright, Error as PWError

STREAM, GROUP = "scrape:jobs", "workers"
DELAYED, DEAD, RESULTS = "scrape:delayed", "scrape:dead", "scrape:results"
CONCURRENCY = int(os.getenv("CONCURRENCY", "3"))
MAX_ATTEMPTS = int(os.getenv("MAX_ATTEMPTS", "4"))
NAV_TIMEOUT_MS = int(os.getenv("NAV_TIMEOUT_MS", "30000"))
RECYCLE_AFTER = int(os.getenv("RECYCLE_AFTER", "500"))
CLAIM_IDLE_MS = 5 * 60 * 1000          # reclaim jobs held this long by a dead worker
CONSUMER = f"{socket.gethostname()}-{os.getpid()}"
USER_AGENT = os.getenv("USER_AGENT") or None
PROXY = {k: v for k, v in {"server": os.getenv("PROXY_SERVER"), "username": os.getenv("PROXY_USERNAME"),
                            "password": os.getenv("PROXY_PASSWORD")}.items() if v} or None
BLOCKED = {"image", "media", "font"}
RETRY_STATUS = {408, 425, 429, 500, 502, 503, 504}


class Retryable(Exception):
    pass


class Permanent(Exception):
    pass


async def ensure_group(r):
    try:
        await r.xgroup_create(STREAM, GROUP, id="0", mkstream=True)
    except redis.ResponseError as e:
        if "BUSYGROUP" not in str(e):
            raise


async def move_due(r):
    """Return retries whose backoff has expired to the stream. ZREM succeeds for one worker only."""
    for raw in await r.zrangebyscore(DELAYED, 0, time.time(), start=0, num=100):
        if await r.zrem(DELAYED, raw):
            await r.xadd(STREAM, {"job": raw})


async def block_heavy(route):
    if route.request.resource_type in BLOCKED:
        await route.abort()
    else:
        await route.continue_()


async def scrape(browser, job):
    ctx = await browser.new_context(user_agent=USER_AGENT)
    try:
        await ctx.route("**/*", block_heavy)
        page = await ctx.new_page()
        try:
            resp = await page.goto(job["url"], wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS)
        except PWError as e:                   # timeouts, DNS and connection errors
            raise Retryable(str(e).splitlines()[0]) from e
        status = resp.status if resp else 0
        if status == 0 or status in RETRY_STATUS:
            raise Retryable(f"HTTP {status}")
        if status >= 400:
            raise Permanent(f"HTTP {status}")
        value = None
        if job.get("selector"):
            try:
                await page.wait_for_selector(job["selector"], timeout=10_000)
            except PWError as e:
                raise Retryable(f"selector not found: {job['selector']}") from e
            value = (await page.inner_text(job["selector"])).strip()
        return {"url": job["url"], "final_url": page.url, "status": status,
                "title": await page.title(), "value": value}
    finally:
        await ctx.close()


async def handle(r, browser, msg_id, fields):
    try:
        job = json.loads(fields["job"])    # a malformed message is acknowledged and dropped below
        domain = urlsplit(job["url"]).hostname
        # politeness: at most one request per domain per interval, across all workers
        if not await r.set(f"rate:{domain}", 1, nx=True, px=int(job.get("min_interval_ms", 3000))):
            await r.zadd(DELAYED, {json.dumps(job): time.time() + 1})
            return
        job["attempts"] = int(job.get("attempts", 0)) + 1
        started = time.monotonic()
        try:
            result = await scrape(browser, job)
            result.update(id=job["id"], attempts=job["attempts"], worker=CONSUMER,
                          ms=int((time.monotonic() - started) * 1000), at=int(time.time()))
            await r.xadd(RESULTS, {"data": json.dumps(result)}, maxlen=100_000, approximate=True)
            await r.incr("stats:ok")
        except Exception as e:                 # unknown errors are retried like temporary ones
            job["last_error"] = f"{type(e).__name__}: {e}"[:500]
            if isinstance(e, Permanent) or job["attempts"] >= MAX_ATTEMPTS:
                await r.xadd(DEAD, {"job": json.dumps(job), "error": job["last_error"], "at": str(int(time.time()))})
                await r.incr("stats:dead")
            else:
                backoff = min(300, 5 * 2 ** job["attempts"]) + random.uniform(0, 3)
                await r.zadd(DELAYED, {json.dumps(job): time.time() + backoff})
                await r.incr("stats:retry")
    finally:
        # at-least-once delivery: a crash before this line means the job runs again, so keep consumers idempotent
        await r.xack(STREAM, GROUP, msg_id)
        await r.xdel(STREAM, msg_id)


async def run(r, browser, msg_id, fields, slots):
    try:
        await handle(r, browser, msg_id, fields)
    finally:
        slots.release()


async def main():
    r = redis.from_url(os.environ["REDIS_URL"], decode_responses=True)
    await ensure_group(r)
    stop = asyncio.Event()
    for sig in (signal.SIGTERM, signal.SIGINT):
        asyncio.get_running_loop().add_signal_handler(sig, stop.set)
    slots, tasks, taken = asyncio.Semaphore(CONCURRENCY), set(), 0
    async with async_playwright() as pw:
        browser = await pw.chromium.launch(proxy=PROXY)
        # recycling the browser every RECYCLE_AFTER jobs caps slow memory growth; Docker restarts us
        while not stop.is_set() and browser.is_connected() and taken < RECYCLE_AFTER:
            await move_due(r)
            await slots.acquire()              # only take a job when a page slot is free
            claimed = await r.xautoclaim(STREAM, GROUP, CONSUMER, CLAIM_IDLE_MS, "0-0", count=1)
            msgs = claimed[1]
            if not msgs:
                resp = await r.xreadgroup(GROUP, CONSUMER, {STREAM: ">"}, count=1, block=2000)
                msgs = resp[0][1] if resp else []
            if not msgs:
                slots.release()
                continue
            msg_id, fields = msgs[0]
            task = asyncio.create_task(run(r, browser, msg_id, fields, slots))
            tasks.add(task)
            task.add_done_callback(tasks.discard)
            taken += 1
        await asyncio.gather(*tasks, return_exceptions=True)
        await browser.close()
    await r.aclose()


if __name__ == "__main__":
    asyncio.run(main())

enqueue.py (the queue manager)

"""Read 'url,selector' lines from stdin and queue them. URLs disallowed by robots.txt are skipped."""
import csv, json, os, sys, uuid
import urllib.robotparser
from urllib.parse import urlsplit

import redis

r = redis.from_url(os.environ["REDIS_URL"], decode_responses=True)
AGENT = os.getenv("USER_AGENT") or "*"
INTERVAL_MS = int(os.getenv("MIN_INTERVAL_MS", "3000"))
robots = {}


def allowed(url):
    p = urlsplit(url)
    base = f"{p.scheme}://{p.netloc}"
    if base not in robots:
        rp = urllib.robotparser.RobotFileParser(base + "/robots.txt")
        try:
            rp.read()
        except OSError:
            pass  # unreadable robots.txt: can_fetch() returns False, so the site is skipped
        robots[base] = rp
    return robots[base].can_fetch(AGENT, url)


queued = skipped = 0
for row in csv.reader(sys.stdin):
    if not row or row[0].startswith("#"):
        continue
    url = row[0].strip()
    if not allowed(url):
        skipped += 1
        continue
    job = {"id": uuid.uuid4().hex, "url": url, "selector": row[1].strip() if len(row) > 1 else "",
           "min_interval_ms": INTERVAL_MS}
    r.xadd("scrape:jobs", {"job": json.dumps(job)})
    queued += 1
print(f"queued {queued}, skipped by robots.txt {skipped}", file=sys.stderr)

export.py

"""Print a stream as JSON lines: results by default, or 'scrape:dead' for the dead-letter queue."""
import json, os, sys

import redis

r = redis.from_url(os.environ["REDIS_URL"], decode_responses=True)
stream = sys.argv[1] if len(sys.argv) > 1 else "scrape:results"
start = "-"
while batch := r.xrange(stream, min=start, count=1000):
    for _, fields in batch:
        print(fields.get("data") or json.dumps(fields))
    start = "(" + batch[-1][0]          # exclusive start for the next page

3. Run it

cd /opt/scraper
docker compose up -d --build
docker compose ps                     # redis healthy, two workers running

# queue jobs: one URL per line, optional CSS selector after a comma
docker compose run --rm -T worker python enqueue.py < urls.csv

# progress
docker compose exec redis sh -c 'redis-cli --no-auth-warning -a "$REDIS_PASSWORD" MGET stats:ok stats:retry stats:dead'
docker compose exec redis sh -c 'redis-cli --no-auth-warning -a "$REDIS_PASSWORD" XLEN scrape:jobs'

# results and failures as JSON lines
docker compose run --rm -T worker python export.py > results.jsonl
docker compose run --rm -T worker python export.py scrape:dead > dead.jsonl

# scale out on a bigger server
docker compose up -d --scale worker=4

To requeue dead-letter jobs after fixing the cause, feed their URLs back through enqueue.py. To add a second server, run only the worker service there and point REDIS_URL at the first server over a private network (VPC) or a WireGuard tunnel, never over the public internet.

4. Resource math: how many browsers fit on 4 GB and 2 vCPUs

The real question isn't how many browsers but how many pages you can render at once. Each Chromium process has a fixed cost, and each open page adds to it. Work it out for memory and for CPU, then take the lower number.

Memory

Memory budget on a 4 GB VPS

• Chromium's base process: roughly 150–250 MB.
• One page in its own context, with images, fonts and media blocked, on a typical e-commerce product page: roughly 100–250 MB. Heavy single-page apps can reach 400 MB.
• Per worker: 1,400 MB limit − 250 MB browser base = 1,150 MB for pages. At 250 MB per page, that's 4 pages; at 350 MB, 3 pages.
• Memory ceiling: 2 workers × 3–4 pages = 6–8 concurrent pages.

CPU

• Rendering is CPU-bound while a page loads and runs its JavaScript. With heavy resources blocked and domcontentloaded as the wait condition, a vCPU typically keeps 2–4 pages moving without stalls.
• CPU ceiling: 2 vCPU × 2–4 = 4–8 concurrent pages.

Result and how to verify it

1. Run a test batch of 500 representative URLs and watch docker stats. Each worker should stay under about 85% of its memory limit.
2. Check uptime: the load average should stay below the number of vCPUs (2). Sustained load above that means pages are queuing for CPU and getting slower.
3. Check dmesg | grep -i oom after the run. Any OOM kill means lowering CONCURRENCY or blocking more resources.
4. If memory and CPU both have room, raise CONCURRENCY to 4 and repeat. Increase one step at a time.

Throughput

Pages per hour ≈ concurrent pages × 3,600 ÷ average seconds per page. The per-domain rate limit caps that for any single site: at one request every 3 seconds, one domain can't exceed 1,200 pages per hour no matter how many workers you run. Spread jobs across domains, or lower the interval only where the site clearly allows it.

Expected performance on this server, estimated for typical e-commerce pages with heavy resources blocked and jobs spread across many domains:

• Average time per page: about 4–6 seconds, including navigation, rendering and extraction.
• Throughput: 6 pages × 3,600 ÷ 5 s ≈ 4,300 pages per hour in theory. Allow for retries and slow pages and plan on about 3,500 pages per hour.
• Wall-clock time for 10,000 pages: about 3 hours. If all 10,000 URLs are on one domain, the 3-second rate limit makes it about 8.5 hours.

5. Cost per 10,000 pages: self-hosted vs ScrapingBee vs ScraperAPI

Compared like for like: JavaScript-rendered pages. ScrapingBee renders JavaScript by default at 5 credits per request. ScraperAPI charges 10 credits for a request with render=true and bills only successful requests. API plans are monthly, so the real cost is the smallest plan that covers your volume.

The honest result at 10,000 pages a month: a managed API is about as cheap as a server, and ScrapingBee's smallest plan is cheaper. Self-hosting wins as volume grows, because the server's price stays flat while API plans step up:

• Servers needed = pages per month ÷ (pages per hour × 730). At an estimated 3,500 pages per hour, one server covers up to about 2.5 million pages a month in theory, so proxies and rate limits become the real constraints long before the server does.
• Proxy cost = pages × average MB per page ÷ 1,024 × price per GB. Blocking images and fonts cuts page weight sharply. Expect roughly 0.3–0.8 MB per page with heavy resources blocked, so 10,000 pages use about 3–8 GB. For pricing models, see how to cut proxy bandwidth costs and proxy providers compared.
• Your time is the hidden line item. Count a few hours a month for updates, failed-job reviews and selector fixes.

6. Production hardening

• Monitoring: alert when stats:dead grows faster than usual or XLEN scrape:jobs stops falling. Both mean something changed on the target or the server.
• Updates: bump the Playwright image tag and the playwright pip version together, rebuild, and run a 100-URL smoke test before a big batch.
• Backups: Redis persists to its append-only file in the redis-data volume. Export results to durable storage (a database or object storage) after each batch instead of treating Redis as the archive.
• Security: workers run as the non-root pwuser inside containers with memory and CPU limits. Keep Redis on localhost or a private network, and keep the .env file readable by root only.
• Selectors: store a selector per site and treat "selector not found" failures in the dead-letter stream as the signal that a site changed its layout.