Turn competitor product pages into clean JSON for an LLM: Firecrawl vs DIY Playwright

A competitor's product page is mostly noise for a language model: navigation, cookie banners, tracking scripts, recommendation carousels and kilobytes of markup around a price, a name and a spec table. Feed raw HTML to an LLM and you pay for tokens that carry no information, and the model may pick the wrong price. What you want is one clean JSON record per product, in a fixed schema, that you can compare, store or hand to a model to write descriptions. This guide compares two ways to get there: Firecrawl's API, which renders the page and returns JSON in your schema, and a DIY pipeline built on Playwright and Beautiful Soup.

1. The pipeline

In the DIY version, the middle box becomes three steps you run yourself: render with Playwright, reduce the HTML with Beautiful Soup, then call an LLM to fill the schema.

2. Firecrawl: one call per page

Firecrawl's /v2/scrape endpoint takes a URL and a list of formats. The json format takes a JSON schema and an optional prompt, and returns the extracted object under data.json. Standard library only:

#!/usr/bin/env python3
"""Turn product pages into JSON records with Firecrawl's JSON format. Python 3.9+, standard library only.

  export FIRECRAWL_API_KEY=fc-...
  python3 fc_extract.py urls.txt products.jsonl
"""
import json
import os
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor

API = os.environ.get('FIRECRAWL_API_URL', 'https://api.firecrawl.dev/v2/scrape')
KEY = os.environ['FIRECRAWL_API_KEY']
WORKERS = 5  # keep at or below your plan's concurrent requests

PROMPT = 'Extract the main product on this page. Use the current selling price, not a crossed-out one.'
SCHEMA = {
    'type': 'object',
    'properties': {
        'name': {'type': 'string'},
        'brand': {'type': 'string'},
        'sku': {'type': 'string'},
        'price': {'type': 'number'},
        'currency': {'type': 'string', 'description': 'ISO 4217 code, e.g. USD'},
        'in_stock': {'type': 'boolean'},
        'specs': {'type': 'object', 'additionalProperties': {'type': 'string'},
                  'description': 'Technical specifications as name: value pairs'},
        'description': {'type': 'string', 'description': 'Product description, at most 3 sentences'},
    },
    'required': ['name', 'price', 'currency'],
}


def extract(url, retries=3):
    body = json.dumps({
        'url': url,
        'formats': [{'type': 'json', 'schema': SCHEMA, 'prompt': PROMPT}],
        'maxAge': 0,  # the default serves cached pages up to 2 days old: wrong for prices
    }).encode()
    headers = {'Authorization': 'Bearer ' + KEY, 'Content-Type': 'application/json'}
    req = urllib.request.Request(API, body, headers)
    for attempt in range(retries + 1):
        try:
            with urllib.request.urlopen(req, timeout=120) as r:
                data = json.load(r)['data']
                status = data.get('metadata', {}).get('statusCode')
                return {'url': url, 'status': status, **data.get('json', {})}
        except urllib.error.HTTPError as e:
            # 429: rate or concurrency limit; 5xx: temporary.
            # 402 (out of credits) and other 4xx errors won't fix themselves.
            if e.code not in (429, 500, 502, 503) or attempt == retries:
                print('%s: HTTP %d %s' % (url, e.code, e.read()[:200]), file=sys.stderr)
                return None
            time.sleep(2 ** attempt)


def main():
    urls = [line.strip() for line in open(sys.argv[1]) if line.strip()]
    with ThreadPoolExecutor(WORKERS) as pool, open(sys.argv[2], 'a', encoding='utf-8') as out:
        ok = 0
        for rec in pool.map(extract, urls):
            if rec:
                out.write(json.dumps(rec, ensure_ascii=False) + '\n')
                ok += 1
    print('%d of %d pages extracted' % (ok, len(urls)), file=sys.stderr)


if __name__ == '__main__':
    main()

Each line of the output file is one product, ready for a database or a prompt:

{"url": "https://shop.example/p/1", "status": 200, "name": "Mug", "price": 24.0, "currency": "USD", "in_stock": true, "specs": {"Capacity": "350 ml"}}

• maxAge: 0. By default Firecrawl may return a cached copy of the page up to two days old. That's fine for specs and descriptions. For prices and stock, always fetch fresh.
• Proxies. The default auto mode retries hard sites with enhanced proxies, and the docs say an enhanced request costs the same 1 credit as a basic one. Set location with a country code to see a market's local prices. Our geo price check guide explains why location changes prices.
• onlyMainContent. On by default, it strips navigation and footers before extraction. If a field that sits in a sidebar, such as a spec table, comes back empty, set it to false.
• Concurrency. Set WORKERS to your plan's concurrent request limit: 2 on the free plan, 5 on Hobby, 25 on Standard. The script retries 429 and temporary errors and stops on 402 (out of credits).

3. DIY: Playwright plus Beautiful Soup

The DIY version renders the page in your own headless browser, keeps the schema.org Product JSON-LD if there is any, and reduces the rest of the page to plain text, which is what you send to the LLM:

#!/usr/bin/env python3
"""Render product pages with Playwright and reduce them to JSON-LD plus clean text for an LLM.

  pip install playwright beautifulsoup4 && playwright install chromium
  python3 diy_extract.py urls.txt pages.jsonl
"""
import json
import re
import sys

from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

MAX_CHARS = 12000  # roughly 3,000 tokens of English text


def reduce(html):
    soup = BeautifulSoup(html, 'html.parser')
    products = []
    for tag in soup.find_all('script', type='application/ld+json'):
        try:
            data = json.loads(tag.string or '')
        except ValueError:
            continue
        for item in data if isinstance(data, list) else data.get('@graph', [data]):
            if 'Product' in str(item.get('@type')):
                products.append(item)
    for tag in soup(['script', 'style', 'noscript', 'svg', 'iframe', 'nav', 'header', 'footer', 'form']):
        tag.decompose()
    text = re.sub(r'\n\s*\n+', '\n', soup.get_text('\n', strip=True))
    return products, text[:MAX_CHARS]


def main():
    urls = [line.strip() for line in open(sys.argv[1]) if line.strip()]
    with sync_playwright() as p, open(sys.argv[2], 'a', encoding='utf-8') as out:
        browser = p.chromium.launch()
        page = browser.new_page()
        # Skip images, fonts and video: they don't hold product data
        page.route(re.compile(r'\.(png|jpe?g|webp|gif|woff2?|mp4)(\?|$)'), lambda r: r.abort())
        for url in urls:
            try:
                page.goto(url, wait_until='networkidle', timeout=45000)
            except Exception as e:
                print('%s: %s' % (url, e), file=sys.stderr)
                continue
            products, text = reduce(page.content())
            rec = {'url': url, 'jsonld': products, 'text': text}
            out.write(json.dumps(rec, ensure_ascii=False) + '\n')
        browser.close()


if __name__ == '__main__':
    main()

In our test, a product page whose price was injected by JavaScript after 300 ms came out as six lines: the name, "$24.00 In stock" and the spec table. Navigation, header, footer and scripts were gone. On a real store, expect a few thousand tokens per page after reduction.

That's where the DIY work starts, not ends:

• You still need the extraction step. The text has to go to an LLM with your schema to become JSON, and that's an API bill or a GPU of your own. Our Ollama and vLLM guide covers running an open model on your own server.
• Use JSON-LD first. When a page's Product JSON-LD already has the price and currency, you can skip the LLM for those fields. Many stores publish it for search engines, but it's often incomplete: the test page above had the name and SKU in JSON-LD and the price only in the visible text.
• Browsers, proxies and blocks are yours. Bot-protected stores need residential proxies and stealth patches. See ZenRows vs self-hosted Playwright for what that costs, and the Playwright cluster guide for running browsers at scale.
• Selectors are the cheap alternative to an LLM, but they break when the site changes. The Zyte auto-extraction guide compares the two.

4. What Firecrawl costs

Firecrawl charges 1 credit to scrape a page and 4 more for the JSON format, so a structured product record is 5 credits. Markdown output without JSON is 1 credit. Failed scrapes are free, but pages that return an error status such as 403 or 404 still cost 1 credit.

Prices from Firecrawl's pricing page, October 2026. Extra credits on paid plans come in $5 packs: 1,000 credits on Hobby, 2,000 on Standard, 2,500 on Growth and 5,000 on Scale. Firecrawl also offers an optional prompt-injection check on LLM formats for 4 more credits, worth turning on when the pages come from sites you don't control.

5. Cost comparison: 10,000 product pages a month

The DIY total leaves out the one-off build, which we'd estimate at 10 to 20 hours. On Hobby with monthly billing, the same 10,000 pages cost $19 plus 45 packs of extra credits, $244 in all, so a steady workload belongs on Standard.

When DIY wins

• A handful of stores with complete JSON-LD. If price, currency, stock and SKU are all in the Product JSON-LD, you need no LLM and no selectors, and a small server covers it. Our Shopify price monitoring guide is an example.
• Very high volume. One million structured pages a month is 5 million credits, five times the Scale plan. At that size, your own browsers and a self-hosted model can cost less, if you have the people to run them.
• Logged-in or interactive pages, or data you can't send to a third party.

6. Which one to pick

7. Tools

Top pick

Tool	Role	Why it fits
Firecrawl	Web pages to LLM-ready JSON and markdown	One call renders the page, handles proxies and returns JSON in your schema, at 5 credits per structured page. Enhanced proxies cost no extra, and the free plan's 1,000 credits are enough to test 200 of your own product pages.

Alternatives: open-source tools

We don't earn anything from these links.

• Playwright for Python: the headless browser in the DIY pipeline.
• Beautiful Soup: HTML parsing and cleanup.
• Scrapy: a full crawling framework with scheduling, retries and pipelines, for large DIY crawls of sites that don't need a browser.
• Crawl4AI: an open-source crawler that outputs LLM-friendly markdown, if you want a DIY tool closer to what Firecrawl does.

8. Pitfalls

• Cached prices. Leaving Firecrawl's default maxAge on a price monitor gives you prices up to two days old. Set it to 0.
• Trusting the JSON without checks. Validate every record before it reaches your database: price is a positive number, currency is a three-letter code, required fields exist. LLM extraction can misread a crossed-out price or a bundle price.
• Paying for dead URLs. 403 and 404 pages still cost a credit. Remove them from your lists.
• Prompt injection. Pages from competitors are untrusted input. Text hidden in a page can try to steer the model that reads it. Use Firecrawl's injection check, or keep the extraction model's output strictly limited to your schema.
• Ignoring a site's terms. Scrape public pages at a polite rate and check the site's terms of service.