Data and accounts
Proxy types and providers, scraping APIs, headless browsers and what they cost per page.
15 guides12 in-depth108 min of reading
Collecting product and price data at volume is mostly a cost problem with a reliability problem inside it. Each request has a price: the proxy traffic it uses, the browser time it takes to render, and the retries when it fails. Those prices change by an order of magnitude with the target. A plain HTTP fetch of a public JSON endpoint through a datacenter IP costs almost nothing. The same page rendered in a headless browser through residential proxies, billed per gigabyte, can cost hundreds of times more. Most of that extra cost is images, fonts and scripts the scraper never reads. The guides in this topic start from that unit cost, measured per 1,000 successful pages rather than per request, because a cheap request that fails half the time is not cheap.
Residential, ISP, datacenter and mobile proxies matches each proxy type to a target. Datacenter suits unprotected public data, rotating residential suits bot-protected retail sites, and static ISP suits long-lived logins. Proxy rotation in Python handles failures: bench a proxy for a few minutes after a timeout, 403, 429 or 5xx, retry through another with backoff, and keep a sticky session for multi-step flows. Proxy health monitoring compares providers on success rate, p50 and p95 latency and cost per 1,000 successful requests, using a standard-library Python script and SQLite. Cutting residential bandwidth covers what to do once the bill arrives: fetch HTML or JSON instead of rendering, block heavy resources and cache pages that haven't changed.
The second decision is whether to run the browsers yourself. A self-hosted Playwright cluster uses a Redis stream as a job queue, retries with exponential backoff and moves jobs that keep failing to a dead-letter stream, starting at 2 browsers with 3 pages each on a 4 GB server. ZenRows, Zyte API, Firecrawl and Browserless move rendering, IP rotation or extraction to a paid API, and each guide prices that against the DIY version. When a site actively blocks automated access, how Cloudflare and DataDome score requests explains the signals and the legitimate options: an official API or feed, allowlisting, slower identified crawling, or licensed data.
Datacenter proxies are the cheapest and fastest choice for unprotected public data. Rotating residential proxies handle bot-protected retail sites. Static ISP proxies give a fixed, ISP-registered IP for long-lived account logins. Mobile proxies route through carrier networks for mobile-first social and ad platforms. Choose by use case, not price.
ZenRows renders the page in its own headless browser, routes it through rotating residential IPs and handles anti-bot challenges, then returns the finished HTML from a single GET request. With JavaScript rendering and premium proxies a page costs 25 credits, which works out to about $6.90 per 1,000 pages on the $69 Launch plan and about $2.75 per 1,000 on the $549 Scale plan. Running Playwright yourself makes the browser almost free, but on a protected store you still pay for residential proxy bandwidth and for the hours spent keeping stealth patches working: by our estimate about $190 to $460 a month for 10,000 protected pages, against $69 on ZenRows. Self-hosting wins on high-volume sites that don't block you.
Send the same set of real target URLs through every proxy you pay for, record each request's status, timing and bytes in SQLite, and compare providers on three numbers: success rate (a 200 response that contains the content you expect), p50 and p95 latency of successful requests, and cost per 1,000 successful requests. The script in this guide does that as a cron-friendly CLI with no dependencies beyond Python's standard library.
Firecrawl takes a URL and a JSON schema, renders the page, and returns the product as structured JSON in one API call, for 5 credits a page (1 for the scrape plus 4 for the JSON format). 10,000 product pages a month is 50,000 credits, which fits the Standard plan at $83 a month billed yearly. A DIY pipeline with Playwright and Beautiful Soup costs little to run, but you still need an LLM call to turn the cleaned text into JSON, proxies for protected stores and a few hours a month of upkeep: by our estimate about $180 to $490 a month for the same 10,000 pages. DIY wins when you scrape a few sites with good schema.org data, or at very high volume.
Replace chromium.launch() with chromium.connect_over_cdp('wss://production-sfo.browserless.io?token=...') and your existing Playwright code drives a Chrome running on Browserless instead of your own server. Browserless bills units of up to 30 seconds of browser time per connection, so how you connect decides the bill: opening one session per page costs at least a unit a page, while reusing each session for many pages costs only the seconds you use. For 100,000 six-second pages a month, that's about $35 with reused sessions or $200 with a session per page, against about $124 for a 4 GB VPS plus an estimated two hours of upkeep. Keep workers at or below your plan's concurrent browsers, reconnect before the session time cap, and always close the browser.
Zyte API's automatic extraction takes a product URL and returns name, price, currency, availability and SKU as JSON, without any selectors, so a store redesign doesn't break your scraper. On pay-as-you-go it costs about $0.40 to $2.35 per 1,000 product pages fetched over HTTP, and about $1.30 to $17 per 1,000 when the page has to be rendered in a browser. Hand-written Playwright selectors cost almost nothing per page but need developer time every time a store changes its HTML. At an estimated half hour of maintenance per store per month, Zyte is cheaper until a single store passes roughly 16,500 to 35,000 HTTP pages a month on a moderate (tier 3) site.
Run Redis and several Playwright worker containers with Docker Compose on one VPS. Workers pull jobs from a Redis stream, render pages in a shared Chromium with a few pages open at once, retry temporary failures with exponential backoff, and move jobs that keep failing to a dead-letter stream. On a 4 GB, 2 vCPU server, start with 2 browsers and 3 pages each, then tune from measured memory and CPU.
Bot management systems score every request on IP reputation, TLS and browser signals, JavaScript challenges and behavior. A block means the site owner doesn't want automated access on those terms. The reliable fixes are legitimate ones: use an official API or data feed, ask for allowlisting, identify your crawler, slow down and cache, or buy the data from a provider that licenses it.
Pick Apify for ready-made scrapers (Actors) for Amazon, Google Maps and social platforms, plus scheduling and storage. Pick ScrapingBee if you write code and want one API call that handles proxies and JavaScript rendering. Pick Browse AI for point-and-click monitoring of a few competitor pages, synced to Google Sheets, without code.
Webshare is the cheapest place to start, with 10 free datacenter proxies and low-cost static plans. IPRoyal suits irregular workloads because its residential traffic doesn't expire. Decodo is the balanced mid-market pick with simple dashboards and scraping APIs. Bright Data costs the most but has the deepest network and unblocking tools.
Residential proxies bill per gigabyte, and a full browser page load can transfer several megabytes of images, scripts and fonts you never use. Cut costs by fetching raw HTML or JSON instead of rendering, blocking heavy resources in headless browsers, caching unchanged pages, scraping less often, and moving unprotected requests to cheap datacenter proxies.
Keep a pool of proxy URLs, pick a healthy one per request, and on a timeout, 403, 429 or 5xx bench that proxy for a few minutes and retry through another with exponential backoff. Use the provider's sticky-session username for multi-step flows, and rotate per request only for independent pages.
A private Squid or Dante SOCKS5 proxy on a $5 DigitalOcean or Vultr VPS gives you a fast, dedicated IP with unlimited bandwidth, ideal for general browsing, staging servers, or API webhooks. However, cloud VPS IPs carry a Datacenter ASN, so Amazon Seller Central, Meta Ads, and strict retail checkouts will still block them.
Default headless Chromium instances set `navigator.webdriver = true` and leak distinct CDP (Chrome DevTools Protocol) serialization patterns that Cloudflare and DataDome detect immediately. Connect Playwright/Puppeteer via WebSocket (`wss://`) to a remote Scraping Browser cluster (Bright Data / Oxylabs) to bypass bot checks automatically.
Standard HTTP requests fail against Cloudflare Turnstile and Akamai because of TLS JA3/JA4 fingerprinting. To scrape reliably at scale, use either a managed Web Unlocker API (Bright Data / Oxylabs) or rotate residential proxies per request with a TLS-spoofed client like curl_cffi.
Disclosure: these are affiliate links. If you sign up through them, I may earn a commission at no extra cost to you. How this site makes money.
Webshare ProxiesBudget Datacenter & Residential IPs
Vultr Cloud ComputeGlobal Cloud VPS
DigitalOcean DropletsDeveloper-Friendly Linux VPS
ScraperAPIScraping API with Proxy Rotation
IPRoyal ResidentialNon-Expiring Residential Traffic
Decodo ResidentialRotating Residential IP Pool
ScrapingBeeHeadless Browser Scraping API
Bright Data Web Unlocker
Managed Unblocking for Hard Targets