Extract product data from any store: Zyte API auto-extraction vs hand-written selectors

Every product scraper built on selectors has the same life cycle. You write h1.product-title and .price .money, it works, and weeks later the store ships a redesign and your price column fills with blanks. With one store, that's an annoyance. With twenty, keeping selectors alive becomes a part-time job. This guide compares two ways to get product data from stores you don't control: Zyte API's automatic extraction, where an AI model finds the fields for you, and hand-written XPath or CSS selectors in Playwright. It covers how auto-extraction works, a script you can run today, and what each approach costs per 1,000 pages once you count your time.

1. How auto-extraction works

You send Zyte a URL and the type of data you want. Zyte fetches the page through its own unblocking and proxy layer, then runs a machine-learning model that reads the page the way a person would: it finds the main product, its name, price, currency, stock status, SKU, brand and images, whatever the HTML looks like. You get back JSON in a fixed schema, the same for every store.

Zyte API automatic extraction, step by step

A request looks like this:

curl https://api.zyte.com/v1/extract \
  --user "$ZYTE_API_KEY": \
  --header 'Content-Type: application/json' \
  --data '{"url": "https://example-store.com/products/blue-mug",
           "product": true,
           "productOptions": {"extractFrom": "httpResponseBody"}}'

And the part of the response you care about:

{
  "url": "https://example-store.com/products/blue-mug",
  "statusCode": 200,
  "product": {
    "name": "Blue Ceramic Mug",
    "price": "18.00",
    "currency": "USD",
    "availability": "InStock",
    "sku": "MUG-BLU-12",
    "metadata": {"probability": 0.98}
  }
}

The full product schema has more fields, including brand, images, breadcrumbs, variants and GTINs. Fields the model can't find are simply missing, so treat every field as optional in your code.

HTTP or browser: the setting that decides your bill

productOptions.extractFrom tells Zyte where to extract from:

• httpResponseBody: a plain HTTP request. Zyte's docs call it "usually faster and cheaper". It works on most stores, because most stores render product data on the server for SEO.
• browserHtml: Zyte renders the page in a browser first. Use it for stores that build the product page in JavaScript. Zyte's docs say it's the default for AI extraction when you don't set extractFrom.

Browser-rendered requests cost roughly eight to thirteen times more than HTTP requests (section 3). If you leave extractFrom out, you may pay browser prices for pages that didn't need a browser. Set it explicitly, start with httpResponseBody and only switch stores where fields come back empty.

2. A script: product URLs to CSV

Python 3.9 or newer, standard library only. Put one product URL per line in urls.txt.

#!/usr/bin/env python3
"""Product pages to CSV with Zyte API automatic extraction. Python 3.9+, standard library only.

  export ZYTE_API_KEY=...
  python3 zyte_products.py urls.txt > products.csv
"""
import base64
import csv
import json
import os
import sys
import time
import urllib.error
import urllib.request

API = os.environ.get('ZYTE_API_URL', 'https://api.zyte.com/v1/extract')
AUTH = 'Basic ' + base64.b64encode((os.environ['ZYTE_API_KEY'] + ':').encode()).decode()  # key as username, empty password
FIELDS = ['url', 'name', 'price', 'currency', 'availability', 'sku', 'probability']


def extract(url, retries=4):
    # httpResponseBody is the cheaper HTTP-request tier; without it Zyte renders the page in a browser
    body = json.dumps({'url': url, 'product': True, 'productOptions': {'extractFrom': 'httpResponseBody'}}).encode()
    for attempt in range(retries + 1):
        req = urllib.request.Request(API, body, {'Authorization': AUTH, 'Content-Type': 'application/json'})
        try:
            with urllib.request.urlopen(req, timeout=120) as r:
                return json.load(r).get('product') or {}
        except urllib.error.HTTPError as e:
            if e.code not in (429, 503, 520, 521) or attempt == retries:  # rate limit or temporary ban: retry; anything else: give up
                print('%s: HTTP %d %s' % (url, e.code, e.read()[:200]), file=sys.stderr)
                return None
            time.sleep(2 ** attempt)


def main():
    w = csv.DictWriter(sys.stdout, FIELDS)
    w.writeheader()
    low = 0
    for url in (line.strip() for line in open(sys.argv[1]) if line.strip()):
        p = extract(url)
        if p is None:
            continue
        prob = (p.get('metadata') or {}).get('probability', 0)
        if prob < 0.5:  # probably not a product page (category, 404, login wall)
            low += 1
        w.writerow({'url': url, 'name': p.get('name'), 'price': p.get('price'), 'currency': p.get('currency'),
                    'availability': p.get('availability'), 'sku': p.get('sku'), 'probability': prob})
    print('done, %d rows below probability 0.5' % low, file=sys.stderr)


if __name__ == '__main__':
    main()

• Retries: 429 (you're sending faster than your account's rate limit) and 503, 520 and 521 (temporary errors or a ban Zyte couldn't get past this time) are retried with exponential backoff. Other errors are logged and skipped. Zyte doesn't bill for failed or rate-limited responses.
• Probability: rows below 0.5 are counted so you can review them. A low score usually means the URL isn't a single product page, for example a category page, a removed product or a login wall.
• Scaling up: the loop sends one request at a time, which is plenty for a few thousand pages a day. For more, run several workers in parallel up to your account's rate limit, or use Zyte's scrapy-zyte-api plugin if you already use Scrapy.

3. What 1,000 pages cost on Zyte

Zyte bills two things for each successful product page: the request itself, priced by how hard the website is to access, and the automatic extraction on top. Zyte puts every website into one of five tiers per request type, automatically, based on how much effort it takes to get a response. You can't choose the tier, but the API playground and cost calculator show it for a URL before you commit.

Pay-as-you-go prices from Zyte's pricing page and API docs, October 2026. Monthly commitments of $100, $200, $350 and $500 cut the rates by 25%, 40%, 48% and 52%. New accounts get $5 of free credit for 30 days, enough for several thousand HTTP product pages on easy sites.

Putting the two together, 1,000 product pages cost:

4. The other side: hand-written selectors in Playwright

Here's what the same job looks like with selectors, for one store:

from playwright.sync_api import sync_playwright

SELECTORS = {  # one block like this per store, and it breaks when the store's theme changes
    'name': 'h1.product__title',
    'price': '.price-item--regular',
    'sku': '.product__sku',
    'in_stock': 'button[name="add"]:not([disabled])',
}

with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto('https://example-store.com/products/blue-mug')
    row = {k: page.locator(sel).first.inner_text()
           for k, sel in SELECTORS.items() if k != 'in_stock'}
    row['in_stock'] = page.locator(SELECTORS['in_stock']).count() > 0
    print(row)

Per page, this is almost free. The cost is everywhere else:

• Writing selectors. Our estimate is one to two hours per store to find stable selectors, handle variants and sale prices, and normalise the price text into a number and currency.
• Silent breakage. When a store changes its theme, a selector scraper often doesn't crash. It returns empty or wrong values, so you need checks that notice, and someone has to fix them.
• Getting the page at all. Larger stores block datacenter IPs. You'll need proxies, retries and ban detection, which is where most of the work in our self-hosted Playwright cluster guide and proxy monitoring guide goes.
• Infrastructure. A 2 to 4 GB VPS for headless Chromium, about $12 to $24 a month.

5. Cost comparison: where each approach wins

The two approaches scale on different things. Zyte's cost grows with pages. A selector scraper's cost grows with stores, because every store has its own selectors to maintain. That gives a simple per-store rule:

Zyte is cheaper for a store when:
  pages per month × Zyte price per page  <  maintenance hours per month × your hourly rate

Using an estimated half hour of maintenance per store per month (spotting breakage, fixing selectors, re-running) at $50 an hour, that's $25 a month per store. At $0.71 to $1.51 per 1,000 HTTP product pages (tier 3), Zyte stays cheaper until a single store passes about 16,500 to 35,000 pages a month. For browser-rendered pages at about $4.29 to $5.09 per 1,000 (tier 3), the crossover is about 5,000 to 6,000 pages a month.

Example: monitoring 10 competitor stores

10 stores, 1,000 product pages each per month (for example 250 products checked weekly), 10,000 pages in total. Selector maintenance is our estimate of half an hour per store per month.

Setup time isn't in the table: one to two hours per store for selectors (another 10 to 20 hours here), against about an hour total for the Zyte script. Proxies aren't in it either, and the selector approach often needs them.

Monthly cost: 10 stores, 10,000 product pages

When selectors still win

• One or two stores at high volume. 500,000 pages a month from a single site costs $355 to $755 on Zyte at tier 3 HTTP pay-as-you-go, while the selectors for that site take the same half hour as for a small one.
• The data is already structured. Many stores embed a JSON-LD Product block for Google, and every Shopify store serves /products.json. Reading those needs no selectors and no AI. Our Shopify products.json guide shows how.
• You need a field the schema doesn't have. Store-specific fields like a custom delivery estimate need either selectors or Zyte's paid custom-attribute extraction, which is billed per token.

6. Which one to pick

7. Tools

Top pick

Alternatives: tools we have no affiliation with

We don't earn anything from these links. They're here because for some readers they're the better choice.

• Apify: a marketplace of ready-made scrapers ("Actors") for specific sites such as Amazon or Shopify stores. A good fit when someone already maintains a scraper for exactly the sites you need. Compared in our e-commerce scrapers guide.
• Bright Data: a large proxy network plus ready-made e-commerce datasets and scraper APIs for big marketplaces. Better suited to large teams and enterprise volumes.
• Playwright: the free browser automation library used for the selector approach above.

8. Pitfalls

• Not setting extractFrom. Without it you may pay browser prices for every page. Start with httpResponseBody.
• Trusting every row. Check metadata.probability and compare a sample against the live pages when you add a new store.
• Feeding it category pages. The product type expects one product per URL. Collect product URLs first, for example from the store's sitemap, then extract.
• Comparing per-page prices only. The real cost of selectors is maintenance hours, and the real cost of Zyte is volume. Count both before you choose.
• Ignoring a site's terms. Scrape public product pages only, at a polite rate, and check the site's terms of service and robots.txt.