Monitor competitor prices on Shopify stores with products.json and Python

If a competitor runs on Shopify, you probably don't need a scraper to track their prices. Most Shopify stores publish their whole catalog as JSON at /products.json, the same public data their product pages show to every visitor. This guide gives you a single Python file that reads that endpoint politely, keeps a snapshot of every variant, and tells you what changed since the last run: price drops, price rises, items going out of stock or coming back, new products and removed ones. It also covers when this simple approach stops being enough and a tool like Apify or a proxy network earns its cost.

1. What /products.json gives you

Open https://any-shopify-store.com/products.json in a browser and you'll see a JSON object with a products list. Each product has a variants list, and each variant carries the fields that matter for price monitoring:

• Pagination: the endpoint returns up to 250 products per request with ?limit=250. Add &page=2, &page=3 and so on until a page comes back with fewer than 250 products.
• Collections: /collections/<handle>/products.json works the same way for one collection. Use it to watch only the category you compete in, which means fewer requests.
• Not every store allows it. Some stores disable the endpoint, put it behind a bot wall or block it in robots.txt. The script treats all three as "skip this store", not as obstacles to work around.

2. How the monitor works

3. The script: shopwatch.py

One file, Python's standard library only, so it runs on any server with Python 3.9 or later. Change the contact address in UA before you run it: a real contact in the User-Agent lets a store owner reach you instead of blocking you.

#!/usr/bin/env python3
"""shopwatch: alert on price and stock changes in public Shopify catalogs (/products.json).

Python 3.9+, standard library only. Each run compares the catalog with the last snapshot.
  python3 shopwatch.py competitor-a.com competitor-b.com
  python3 shopwatch.py --stores stores.txt --min-drop 5 > changes.txt
"""
import argparse
import json
import os
import random
import re
import sys
import time
import urllib.error
import urllib.request
import urllib.robotparser
from urllib.parse import urlparse

UA = 'shopwatch/1.0 (price monitoring; contact: [email protected])'  # put a real contact address here
PAGE_SIZE = 250  # the most products.json returns per page
RETRY_CODES = {429, 500, 502, 503, 504}


def get(opener, url, retries=5, base=2.0):
    """GET with exponential backoff on 429/5xx and network errors. Honors Retry-After."""
    for attempt in range(retries + 1):
        wait = base * 2 ** attempt
        try:
            with opener.open(urllib.request.Request(url, headers={'User-Agent': UA}), timeout=30) as r:
                return r.read()
        except urllib.error.HTTPError as e:
            if e.code not in RETRY_CODES or attempt == retries:
                raise
            retry_after = e.headers.get('Retry-After', '')
            wait = float(retry_after) if retry_after.isdigit() else wait
            reason = 'HTTP %d' % e.code
        except (urllib.error.URLError, TimeoutError) as e:
            if attempt == retries:
                raise
            reason = type(e).__name__
        wait = min(wait, 300) + random.uniform(0, 1)  # jitter so many stores don't retry in lockstep
        print('  %s on %s, retrying in %.0fs' % (reason, url, wait), file=sys.stderr)
        time.sleep(wait)


def robots_delay(opener, base):
    """Return the Crawl-delay robots.txt asks for, or None if /products.json is disallowed."""
    rp = urllib.robotparser.RobotFileParser()
    try:
        rp.parse(get(opener, base + '/robots.txt').decode('utf-8', 'replace').splitlines())
    except urllib.error.HTTPError as e:
        return None if e.code in (401, 403) else 0  # no robots.txt means no restrictions
    if not rp.can_fetch(UA, base + '/products.json'):
        return None
    return float(rp.crawl_delay(UA) or 0)


def fetch_catalog(opener, base, delay):
    """All variants of a store, keyed by variant ID. Pages through until a short page."""
    items, page = {}, 1
    while True:
        products = json.loads(get(opener, '%s/products.json?limit=%d&page=%d' % (base, PAGE_SIZE, page)))['products']
        for p in products:
            for v in p['variants']:
                items[str(v['id'])] = {
                    'name': p['title'] + ('' if v['title'] == 'Default Title' else ' / ' + v['title']),
                    'url': '%s/products/%s' % (base, p['handle']),
                    'price': v['price'], 'compare_at': v.get('compare_at_price'), 'available': v.get('available')}
        if len(products) < PAGE_SIZE:
            return items
        page += 1
        time.sleep(delay)


def diff(old, new, min_drop):
    """Changes between two snapshots as (event, item, detail) tuples."""
    events = []
    for vid, n in new.items():
        o = old.get(vid)
        if o is None:
            events.append(('new', n, n['price']))
            continue
        op, np = float(o['price']), float(n['price'])
        if np < op and (op - np) / op * 100 >= min_drop:
            events.append(('price_drop', n, '%s -> %s (-%.1f%%)' % (o['price'], n['price'], (op - np) / op * 100)))
        elif np > op:
            events.append(('price_up', n, '%s -> %s (+%.1f%%)' % (o['price'], n['price'], (np - op) / op * 100)))
        if o['available'] != n['available']:
            events.append(('back_in_stock' if n['available'] else 'out_of_stock', n, n['price']))
    events += [('removed', old[vid], old[vid]['price']) for vid in old.keys() - new.keys()]
    return events


def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument('stores', nargs='*', help='store domains, e.g. shop.example.com')
    ap.add_argument('--stores', dest='stores_file', help='file with one store domain per line')
    ap.add_argument('--dir', default='snapshots', help='where snapshots and changes.jsonl are kept')
    ap.add_argument('--delay', type=float, default=2.0, help='seconds between requests to the same store')
    ap.add_argument('--min-drop', type=float, default=0, help='only report price drops of at least this %%')
    ap.add_argument('--proxy', help='http://user:pass@host:port, e.g. to see a specific country\'s prices')
    a = ap.parse_args()

    stores = list(a.stores)
    if a.stores_file:
        with open(a.stores_file) as f:
            stores += [s.strip() for s in f if s.strip() and not s.startswith('#')]
    if not stores:
        ap.error('give at least one store')
    handlers = [urllib.request.ProxyHandler({'http': a.proxy, 'https': a.proxy})] if a.proxy else []
    opener = urllib.request.build_opener(*handlers)
    os.makedirs(a.dir, exist_ok=True)
    failed = 0

    for store in stores:
        base = (store if '://' in store else 'https://' + store).rstrip('/')
        path = os.path.join(a.dir, re.sub(r'[^\w.-]', '_', urlparse(base).netloc) + '.json')
        try:
            crawl_delay = robots_delay(opener, base)
            if crawl_delay is None:
                print('%s: robots.txt disallows /products.json, skipped' % store, file=sys.stderr)
                continue
            new = fetch_catalog(opener, base, max(a.delay, crawl_delay))
        except Exception as e:  # 404 = products.json disabled or not a Shopify store
            print('%s: failed (%s: %s)' % (store, type(e).__name__, e), file=sys.stderr)
            failed += 1
            continue

        if os.path.exists(path):
            with open(path) as f:
                events = diff(json.load(f), new, a.min_drop)
        else:
            events = []
            print('%s: baseline saved, %d variants' % (store, len(new)), file=sys.stderr)
        with open(path + '.tmp', 'w') as f:
            json.dump(new, f)
        os.replace(path + '.tmp', path)  # atomic: a crash mid-write never corrupts the last snapshot

        stamp = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())
        with open(os.path.join(a.dir, 'changes.jsonl'), 'a') as log:
            for event, item, detail in events:
                print('%-13s %-28s %-40s %s  %s' % (event, store[:28], item['name'][:40], detail, item['url']))
                log.write(json.dumps({'ts': stamp, 'store': store, 'event': event, 'detail': detail, **item}) + '\n')
        print('%s: %d variants, %d changes' % (store, len(new), len(events)), file=sys.stderr)
        time.sleep(a.delay)

    sys.exit(1 if failed else 0)


if __name__ == '__main__':
    main()

A few design choices worth knowing:

• Prices are compared per variant ID, so a renamed product or a reordered catalog doesn't show up as a change.
• The first run only saves a baseline. Alerts start from the second run.
• Snapshots are written atomically through a temporary file, so a crash or a full disk can't leave you with a half-written snapshot and a flood of false "new product" alerts on the next run.
• A failed store doesn't stop the run. It's reported on stderr, the next store runs, and the exit code is 1 so cron can tell you.

4. Run it

First run and output

mkdir -p /opt/shopwatch && cd /opt/shopwatch
# copy shopwatch.py here, then list the stores you watch:
printf "competitor-a.com\ncompetitor-b.com\n" > stores.txt

python3 shopwatch.py --stores stores.txt              # saves the baseline
python3 shopwatch.py --stores stores.txt --min-drop 5 # later runs report changes

Changes go to stdout, one per line, and progress goes to stderr. Here's the format, from the local test store:

price_drop    competitor-a.com     Merino Runner / 10           65.00 -> 49.00 (-24.6%)  https://competitor-a.com/products/merino-runner
price_up      competitor-a.com     Trail Sock 3-Pack            18.00 -> 19.50 (+8.3%)  https://competitor-a.com/products/trail-sock
out_of_stock  competitor-a.com     Rain Shell / M               120.00  https://competitor-a.com/products/rain-shell
new           competitor-b.com     Winter Runner / 9            95.00  https://competitor-b.com/products/winter-runner
removed       competitor-b.com     Canvas Slip-On / 8           55.00  https://competitor-b.com/products/canvas-slip-on

Every change is also appended to snapshots/changes.jsonl with a timestamp, which gives you a price history you can load into a spreadsheet or pandas later.

Options

Schedule it and alert on drops

Prices on most stores change a few times a week, not every minute. Every 6 hours catches same-day sales and keeps your load on each store to a handful of requests a day. Add this with crontab -e:

0 */6 * * * cd /opt/shopwatch && python3 shopwatch.py --stores stores.txt --min-drop 5 > changes.txt 2>>run.log; grep -E '^(price_drop|back_in_stock)' changes.txt > alert.txt && curl -fsS -X POST --data-binary @alert.txt "$ALERT_WEBHOOK"

Set ALERT_WEBHOOK at the top of the crontab to a Slack, Discord or ntfy webhook. The grep keeps the alert to price drops and restocks. Drop it to get every change. To see the recent history of one product:

grep '"merino-runner"' snapshots/changes.jsonl | tail -n 20

5. Ethics and rate limiting

This script is for monitoring public pricing information: data that each store publishes to every visitor without a login. Used that way, competitor price monitoring is a normal part of retail. Keep it that way:

• Public data only. Don't log in, don't touch cart or checkout endpoints to probe stock levels, and never collect customer or personal data. The script only reads /robots.txt and /products.json.
• Respect robots.txt. The script skips any store that disallows /products.json and uses a longer Crawl-delay when robots.txt asks for one. Shopify's default robots.txt doesn't block the endpoint, so a block means the owner chose it.
• Crawl politely. One store at a time, one request every 2 seconds or more, a few runs a day. A 1,000-product store takes 4 requests per run. That's less load than one visitor browsing a collection page.
• Treat 429 as a "slow down", not a challenge. The script waits for the time in Retry-After or backs off exponentially, and gives up after 5 tries. Don't rotate proxies to get around a store's rate limit. That turns monitoring into evasion.
• Identify yourself. Put a real contact address in the User-Agent. If an owner asks you to stop, stop and remove their store from the list.
• Use prices, don't republish content. Product descriptions and images are the store's copyrighted content. Store prices and availability for your own analysis, and don't copy their catalog onto your site.
• Read the terms. Shopify's and each store's terms of service apply to how you access the site. This guide isn't legal advice. If monitoring is central to your business, have counsel review your setup.

6. products.json script vs Apify vs proxies

The script covers most small and mid-sized monitoring jobs on its own. Here's when it's the right tool and when you'll need something heavier:

Estimated monthly cost by scale

These are estimates from public pricing pages as of October 2026. Apify actors are priced by their authors, either per result or per compute unit, so check the actor's pricing page before you schedule it.

7. Tools for running it

Top picks

Tool	Role	Why it fits
Vultr	Server for scheduled runs	Hourly-billed cloud servers in many regions. The smallest plan is plenty for this script. Pick a region close to your competitors' main market.
DigitalOcean	Server for scheduled runs	Simple Droplets with a clear dashboard and good docs, a comfortable choice if this is your first Linux server.
Apify	Managed scraping at scale	Ready-made Shopify scrapers in the Apify Store, with scheduling, retries and datasets. The right step up when stores block products.json or the list grows into the thousands.
Webshare	Country-specific prices	Low-cost datacenter and residential proxies with country selection. The free plan's 10 proxies are enough to check what prices a market sees. Pass one with --proxy.

For more on choosing a server, see the self-hosted scraping cluster guide. For comparing proxy pools on your targets, see proxy health monitoring. For managed scrapers, see Apify vs ScrapingBee vs Browse AI.

Alternatives: free tools and tools we have no affiliation with

We don't earn anything from these. They're here because for some readers they're the better choice.

• GitHub Actions: run the script on a schedule trigger and commit the snapshots back to the repository, with no server at all. Free for public repositories, with a monthly minutes allowance for private ones. Scheduled runs can start late at busy times, which doesn't matter for price checks.
• changedetection.io: open-source change monitoring you can self-host. It can watch a JSON URL with a JSONPath filter and send notifications, which is enough for a handful of key products without writing any code.
• Oracle Cloud Free Tier or a Raspberry Pi: a free always-on machine for cron. Fine for a few dozen stores.
• Distill.io and Visualping: browser-based page monitors with free tiers for a small number of pages. Good when you watch 5 products, not 5,000.
• Prisync and Price2Spy: commercial competitor price tracking with dashboards, product matching across stores and reports. Worth it for a team that needs those features and doesn't want to maintain a script.

8. Pitfalls

• Sales create noise. A sitewide 20% sale turns into hundreds of alerts at once. Group alerts by store, or alert when many variants drop together, before you react product by product.
• Prices are strings. Compare them as numbers, as the script does, or "100.00" sorts below "99.00".
• Only published products appear. Drafts and products hidden from the online store aren't in the feed, so "removed" can mean unpublished, not discontinued.
• Responses can be cached for a few minutes. A change made in the store admin may take a short time to show up. That's another reason not to poll every minute.
• Very large catalogs. On stores with tens of thousands of products, deep page numbers may be slow or stop returning results. Split the job by collection with /collections/<handle>/products.json.
• Store moves off Shopify. The endpoint starts returning 404 and the script reports the store as failed. Check the exit code in cron so it doesn't fail silently for weeks.