Why scrapers get blocked by Cloudflare and DataDome, and how to collect web data responsibly instead

If your scraper suddenly returns challenge pages, 403 errors or endless CAPTCHAs, the target site has turned on bot management. Cloudflare and DataDome protect a large share of e-commerce and travel sites. They're designed to stop credential stuffing, inventory hoarding and price scraping that the site owner hasn't agreed to. This guide explains how they decide, and the approaches that work long-term without fighting the site owner.

How bot management decides

• IP reputation: traffic from cloud and datacenter ranges, or from addresses with a history of abuse, starts with a worse score.
• Client signals: HTTP libraries and headless browsers present different TLS and header patterns from real browsers.
• Challenges: Cloudflare's managed challenges and Turnstile run a short check in the browser. DataDome uses its own device checks and CAPTCHA.
• Behavior: request rate, page order and timing. Hundreds of product pages per minute from one client stands out.
• The owner's rules: sites decide what to block, challenge or allow, including allowlists for partners and verified bots.

Legitimate ways to get the data

Polite crawling: the rules that keep you welcome

1. Read robots.txt and the terms of service for every target, and keep a dated copy.
2. Identify your crawler. Use a descriptive user agent with a contact URL or email, for example PriceMonitor/1.0 (+https://example.com/bot).
3. Rate-limit hard. One request every few seconds per site is plenty for price monitoring. Respect Retry-After headers and back off on 429 and 503 responses.
4. Cache and use conditional requests. Send If-Modified-Since or ETag headers, and don't refetch pages that haven't changed.
5. Collect only what you need. Prices and availability, not personal data. Personal data brings GDPR and CCPA obligations.
6. Stop when asked. A block, a cease-and-desist or a changed robots.txt is an answer. Switch to an API, a partnership or licensed data.

import time, requests

HEADERS = {"User-Agent": "PriceMonitor/1.0 (+https://example.com/bot; [email protected])"}
etags = {}

def polite_get(url, min_interval=5):
    h = dict(HEADERS)
    if url in etags:
        h["If-None-Match"] = etags[url]
    r = requests.get(url, headers=h, timeout=30)
    if r.status_code in (429, 503):
        time.sleep(int(r.headers.get("Retry-After", 60)))  # back off as the site asks
        return None
    if r.status_code == 304:
        return None  # unchanged since last fetch
    if "ETag" in r.headers:
        etags[url] = r.headers["ETag"]
    time.sleep(min_interval)
    return r.text if r.ok else None

For site owners: let the right bots in

If you run the protected site and your own monitoring, partners or uptime checks are getting blocked:

• Create a WAF custom rule that skips bot checks for your partners' fixed IP ranges or for requests carrying a secret header.
• Allow known verified bots (search engines, uptime monitors) through Cloudflare's verified bot category instead of disabling protection.
• Offer partners an API or feed so they don't need to crawl your storefront at all.
• Review bot analytics monthly to see what's blocked, and why.

When blocks hit your business, not just your scraper

If your team gets challenged while logging into supplier portals or marketplaces you're entitled to use, it's usually a network-reputation problem: shared VPNs, cloud IPs or unusual locations. Use a consistent business connection or a dedicated static IP, and ask the platform's support to review it. See the proxy and browser leak checklist and scraping tools compared.