Syncing 50,000 products: BigCommerce vs Shopify API rate limits, variants and B2B pricing

At 500 products, any API works and the choice of platform barely matters to the integration. At 50,000 products with several variants each, the sync job becomes a capacity problem: how many requests the platform lets you make, how much data each one returns, and what happens when an ERP, a PIM and a repricer all share the same budget. BigCommerce and Shopify answer this with different models. This guide compares them where large catalogs feel it, with two tested exporters and our view on which platform fits which catalog.

1. The short answer

	BigCommerce	Shopify
Rate limit model	Requests per 30-second window, per store, shared by all apps	Query cost in points per second, per app and store
Budget by plan	Core and Growth: 150 per 30 s (20,000 an hour). Scale: 450 per 30 s. Performance: by plan, some unlimited	Standard: 100 points/s. Advanced: 200. Plus: 1,000. Enterprise: 2,000
Full catalog export	Parallel page requests: 250 products per page, variants included	Bulk operation: one query, not rate limited, JSONL result
Pagination limits	Page numbers; responses capped at 8 MB	250 per page, cursor-based; pagination capped at 25,000 objects
Batch writes	10 products or 50 variants per request	All variants of one product per mutation; bulk mutations from a JSONL file up to 100 MB
Variants per product	Up to 600 SKUs; up to 250 values per option	Up to 2,048 variants; 3 options
B2B pricing	Price lists by customer group, bulk upsert of 1,000 records; customer groups from Growth, Price Lists on Performance	B2B catalogs and price lists, built for Plus; price inputs up to 250 per call

2. Two rate limit models

BigCommerce counts requests. Every REST call costs one unit of a quota that refills every 30 seconds. The quota belongs to the store, not to your app: the docs state that all apps accessing the store share it. Every response tells you where you stand in X-Rate-Limit-Requests-Left, X-Rate-Limit-Time-Reset-Ms, X-Rate-Limit-Requests-Quota and X-Rate-Limit-Time-Window-Ms. Over the quota you get HTTP 429, and you wait the reset time. A 429 without those headers means the platform itself is under load, and you should back off harder.

Shopify counts query cost. Each GraphQL query has a requested cost computed before it runs: objects cost 1, mutations 10, and a connection costs in proportion to its first argument. A single query may not exceed 1,000 points on any plan. Points refill at the plan's rate, and the response's extensions.cost.throttleStatus reports what's left. The consequence for catalogs: nesting variants inside products multiplies the cost. By our reading of the cost rules, products(first: 250) with variants(first: 100) requests roughly 25,000 points, far over the cap, so paged queries with variants are forced down to a handful of products per page.

3. Exporting the Shopify catalog: use a bulk operation

Paging through 50,000 products on Shopify runs into two documented walls: the per-query cost cap above, and a pagination limit of 25,000 objects. Shopify's own docs point large datasets to bulk operations. You submit one query, Shopify runs it in the background (up to 10 days for a query), and you download a JSONL file. Since API version 2026-01, an app can run up to five bulk queries per shop at once and poll each by ID. Bulk queries allow at most five connections, nested at most two levels deep, which products with variants fits.

#!/usr/bin/env python3
"""Export every Shopify product with its variants through a GraphQL bulk operation,
then rebuild product -> variants from the JSONL result. Python 3.9+, standard library only.

  export SHOP=your-store.myshopify.com SHOPIFY_TOKEN=shpat_...   # read_products scope
  python3 shopify_bulk_export.py products.jsonl
  python3 shopify_bulk_export.py changed.jsonl --since 2026-10-18T00:00:00Z

Output: one product per line, with a "variants" list.
"""
import argparse
import json
import os
import sys
import time
import urllib.request

API = os.environ.get('SHOPIFY_API') or 'https://%s/admin/api/2026-07/graphql.json' % os.environ.get('SHOP')

QUERY = '''{ products%s { edges { node {
  id handle title status updatedAt
  variants { edges { node { id sku price compareAtPrice inventoryQuantity
    selectedOptions { name value } } } } } } } }'''


def gql(query, variables=None):
    body = json.dumps({'query': query, 'variables': variables or {}}).encode()
    headers = {'X-Shopify-Access-Token': os.environ['SHOPIFY_TOKEN'], 'Content-Type': 'application/json'}
    for attempt in range(6):
        with urllib.request.urlopen(urllib.request.Request(API, body, headers), timeout=60) as r:
            res = json.load(r)
        throttled = any(e.get('extensions', {}).get('code') == 'THROTTLED' for e in res.get('errors', []))
        if not throttled:
            if res.get('errors'):
                sys.exit('GraphQL error: %s' % res['errors'])
            return res['data']
        time.sleep(2 ** attempt)  # bulk operations aren't rate limited, but these small calls are
    sys.exit('still throttled after 6 attempts')


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('out')
    ap.add_argument('--since', help="only products updated after this ISO time, e.g. 2026-10-18T00:00:00Z")
    a = ap.parse_args()
    query = QUERY % ('(query: "updated_at:>\'%s\'")' % a.since if a.since else '')

    started = gql('''mutation($q: String!) { bulkOperationRunQuery(query: $q) {
        bulkOperation { id status } userErrors { field message } } }''', {'q': query})
    run = started['bulkOperationRunQuery']
    if run['userErrors']:
        sys.exit('could not start: %s' % run['userErrors'])
    op_id, t0 = run['bulkOperation']['id'], time.time()
    print('started', op_id)

    while True:  # bulkOperation(id:) needs API 2026-01+; a bulk_operations/finish webhook saves polling
        time.sleep(5)
        op = gql('query($id: ID!) { bulkOperation(id: $id) { status errorCode objectCount url } }',
                 {'id': op_id})['bulkOperation']
        print('%-9s %8s objects  %4.0fs' % (op['status'], op['objectCount'], time.time() - t0))
        if op['status'] not in ('CREATED', 'RUNNING'):
            break
    if op['status'] != 'COMPLETED':
        sys.exit('bulk operation %s: %s' % (op['status'], op['errorCode']))
    if not op['url']:  # completed with no matching products
        open(a.out, 'w').close()
        return print('0 products')

    # Variants arrive as their own lines with __parentId, after their product.
    # Collect them in memory: about 1-2 KB per variant, so 500k variants fit in under 1 GB.
    products, order = {}, []
    with urllib.request.urlopen(op['url'], timeout=300) as r:
        for line in r:
            row = json.loads(line)
            parent = row.pop('__parentId', None)
            if parent:
                products[parent]['variants'].append(row)
            else:
                row['variants'] = []
                products[row['id']] = row
                order.append(row['id'])
    with open(a.out, 'w', encoding='utf-8') as f:
        for pid in order:
            f.write(json.dumps(products[pid]) + '\n')
    variants = sum(len(p['variants']) for p in products.values())
    print('%d products, %d variants -> %s' % (len(order), variants, a.out))


if __name__ == '__main__':
    main()

• Rebuild from __parentId. Variants are separate lines that point to their product. Shopify guarantees children come after their parent, not directly after it, so the script keeps products in a dictionary until the file ends. Our mock interleaved them on purpose.
• Retry THROTTLED. The bulk job isn't rate limited, but the calls that start and poll it are. The mock throttled the first call; the script waited and retried.
• Use --since for deltas. The same job with an updated_at filter exports only changed products, which is how a nightly sync should run between full exports.
• Prefer the webhook in production. Subscribing to bulk_operations/finish replaces polling, but Shopify notes that webhook delivery isn't guaranteed, so keep a slow poll as a fallback.

started gid://shopify/BulkOperation/1
RUNNING      20000 objects     5s
RUNNING      40000 objects    10s
COMPLETED   256093 objects    15s
50000 products, 206093 variants -> products.jsonl

That's the mock's output. A real job's duration depends on Shopify's queue and the catalog, and Shopify doesn't publish a throughput figure; plan for minutes, not seconds, and don't block a web request on it.

4. Exporting the BigCommerce catalog: parallel pages on a shared quota

BigCommerce's catalog endpoint returns up to 250 products per page, and include=variants embeds each product's variants, so 50,000 products is about 200 requests. On Core or Growth that's at least 40 seconds of quota; on Scale at least 14. The work is in sharing the quota between parallel workers without tripping 429s, and leaving room for the store's other apps.

#!/usr/bin/env python3
"""Export a BigCommerce catalog (products with variants, optionally a B2B price list)
with parallel page requests that respect the store's shared rate limit.
Python 3.9+, standard library only.

  export BC_STORE_HASH=abc123 BC_TOKEN=...      # API account with Products read-only scope
  python3 bigcommerce_export.py products.jsonl --workers 4
  python3 bigcommerce_export.py prices.jsonl --price-list 3   # price list records (Price Lists scope)
"""
import argparse
import concurrent.futures
import itertools
import json
import os
import sys
import threading
import time
import urllib.error
import urllib.request

API = os.environ.get('BC_API') or 'https://api.bigcommerce.com/stores/%s/v3' % os.environ.get('BC_STORE_HASH')


class Limiter:
    """Shares one view of the quota headers between all threads. All apps on a store share
    its quota, so leave headroom for the others instead of draining it to zero."""

    def __init__(self, headroom):
        self.headroom, self.lock, self.pause_until = headroom, threading.Lock(), 0.0

    def wait(self):
        while True:
            with self.lock:
                delay = self.pause_until - time.time()
            if delay <= 0:
                return
            time.sleep(delay)

    def update(self, headers):
        left, reset = headers.get('X-Rate-Limit-Requests-Left'), headers.get('X-Rate-Limit-Time-Reset-Ms')
        if left is not None and reset is not None and int(left) <= self.headroom:
            with self.lock:
                self.pause_until = max(self.pause_until, time.time() + int(reset) / 1000)


def get(limiter, path, params):
    url = '%s%s?%s' % (API, path, '&'.join('%s=%s' % kv for kv in params.items()))
    req = urllib.request.Request(url, headers={'X-Auth-Token': os.environ['BC_TOKEN'],
                                               'Accept': 'application/json'})
    for attempt in range(8):
        limiter.wait()
        try:
            with urllib.request.urlopen(req, timeout=120) as r:
                limiter.update(r.headers)
                return json.load(r)
        except urllib.error.HTTPError as e:
            if e.code == 429:
                reset = e.headers.get('X-Rate-Limit-Time-Reset-Ms')
                # No headers on a 429 means the store is under platform load: back off harder
                delay = int(reset) / 1000 if reset else 2 ** attempt
                with limiter.lock:
                    limiter.pause_until = max(limiter.pause_until, time.time() + delay)
            elif e.code >= 500:
                time.sleep(2 ** attempt)
            else:
                sys.exit('%s -> HTTP %d: %s' % (url, e.code, e.read()[:300]))
    sys.exit('%s: gave up after 8 attempts' % url)


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('out')
    ap.add_argument('--workers', type=int, default=4)
    ap.add_argument('--limit', type=int, default=250,
                    help='items per page (max 250); lower it if products have hundreds of variants')
    ap.add_argument('--price-list', type=int, help='export this price list instead of products')
    a = ap.parse_args()

    if a.price_list:
        path, params = '/pricelists/%d/records' % a.price_list, {'limit': a.limit}
    else:
        # sort=id keeps pages stable while the export runs: new products get higher IDs
        path, params = '/catalog/products', {'limit': a.limit, 'sort': 'id', 'include': 'variants',
                                             'include_fields': 'name,sku,price,sale_price,inventory_level,'
                                                               'is_visible,date_modified'}
    limiter, t0 = Limiter(headroom=a.workers * 2), time.time()
    first = get(limiter, path, dict(params, page=1))
    pages = first['meta']['pagination']['total_pages']
    print('%d items on %d pages' % (first['meta']['pagination']['total'], pages))

    seen, rows = set(), 0
    with open(a.out, 'w', encoding='utf-8') as f, \
            concurrent.futures.ThreadPoolExecutor(a.workers) as pool:
        jobs = [pool.submit(get, limiter, path, dict(params, page=p)) for p in range(2, pages + 1)]
        for page in itertools.chain([first], (j.result() for j in jobs)):  # write in page order
            for item in page['data']:
                key = (item.get('id'), item.get('variant_id'), item.get('price_list_id'))
                if key not in seen:  # a deletion mid-export can shift one item onto the next page
                    seen.add(key)
                    f.write(json.dumps(item) + '\n')
                    rows += 1
    print('%d rows -> %s in %.0fs' % (rows, a.out, time.time() - t0))


if __name__ == '__main__':
    main()

• One limiter for all threads. When the remaining quota drops to the headroom, every worker pauses until the window resets. Per-thread backoff would let four workers stampede the quota together.
• Headroom for other apps. Your ERP connector, reviews app and repricer draw from the same quota. Draining it to zero makes their requests fail, not yours.
• Stable pages. Sorting by ID means products created during the export land on the last page instead of shifting every page. A deletion can still shift one product across a page boundary, so the script de-duplicates and a delta run catches the rest.
• Watch the 8 MB response cap. 250 products with dozens of variants each can exceed it. Lower --limit for variant-heavy catalogs.
• Price lists use the same pager. --price-list exports a B2B price list's records instead of products.

5. Writing changes back: where the plans differ most

Reads are a nightly job; writes are where a repricer or ERP sync lives all day. Take a price update for 200,000 variants across 50,000 products:

• BigCommerce: the batch variants endpoint takes 50 variants per request, so 4,000 requests. At 150 per 30 seconds that's about 13 minutes; on Scale about 4.5.
• Shopify, one mutation per product: productVariantsBulkUpdate updates all variants of one product at a time, so 50,000 mutations of at least 10 points each, 500,000 points or more.
• Shopify, bulk mutation: bulkOperationRunMutation runs the same mutation over a JSONL file of up to 100 MB in the background, isn't subject to the rate limit, and must finish within 24 hours. For a full repricing, this is the way.

The point isn't that Shopify is slow: with bulk mutations it isn't. It's that on BigCommerce's lower plans, plain request-by-request updates are fast enough for intraday changes, while on Shopify below Plus you design around bulk jobs from the start.

6. Variants and B2B pricing

For apparel with sizes, colors and fits, Shopify's 2,048 variants are generous but its three options can be the tighter limit; BigCommerce's 600 SKUs per product can bind first for configurable industrial parts. For B2B, the deciding factor is usually the plan, not the API: tiered prices for a few customer groups on BigCommerce sit in a mid-tier plan, while per-company catalogs and price lists on Shopify mean Plus.

One BigCommerce detail that catches parallel sync jobs: only one price list bulk upsert can run per store at a time, and a second one in parallel fails with 429. Queue price list writes in a single worker even if everything else runs in parallel.

7. The sync architecture we'd use on either platform

Don't poll date_modified on BigCommerce for stock: its docs note that inventory changes made through the Inventory API don't update it. Use the inventory webhook for stock, and the delta for everything else.

8. Our opinion: trade-offs

• Large B2B catalogs on a mid-market budget: BigCommerce. Request-based limits that are easy to plan, batch writes that are fast enough without bulk jobs, and customer-group pricing without moving to an enterprise plan. The weak spots: the quota is shared with every app on the store, and price lists proper need Performance or B2B Edition.
• DTC catalogs with deep variants and an app-heavy stack: Shopify. Bulk operations are the best full-export tool of the two, and 2,048 variants cover most apparel. Plan for bulk jobs from day one, and budget for Plus if you need per-company B2B pricing.
• Don't pick a platform on API speed alone. Both can export 50,000 products in minutes when used the right way. The difference is in how much engineering the "right way" takes on your plan.
• Measure the shared budget, not just yours. On BigCommerce, log X-Rate-Limit-Requests-Left for a day before adding a sync: if other apps already use most of the quota, your job's speed is theirs to decide.

Related: Amazon SP-API price and inventory sync and monitoring competitor Shopify prices.

9. Tools

Top picks

Tool	Role	Why it fits
BigCommerce	SaaS commerce platform for large and B2B catalogs	250 products per page with variants embedded, batch variant writes, per-window quotas you can read from every response, and customer-group pricing from the Growth plan. Plans from $29 a month billed yearly (Core); Scale at $299 triples the API quota.

Alternatives: free and open-source tools

We don't earn anything from these links.

• Shopify bulk operations documentation: the reference for bulk queries, limits and the JSONL format.
• BigCommerce rate limits documentation: quotas by plan and the rate limit headers.
• Python standard library: both exporters use only urllib, json and concurrent.futures; no SDK required.

10. Pitfalls

• Paging Shopify with nested variants. The query cost forces tiny pages, and the 25,000-object pagination limit stops you halfway through 50,000 products. Use a bulk query.
• Per-thread backoff on BigCommerce. Each worker backing off on its own still hits the quota together. Share one view of the headers.
• Parallel price list upserts. BigCommerce allows one at a time per store; the second fails with 429.
• Trusting one webhook. Both platforms can miss or delay deliveries. Always have a delta export and a reconciliation behind them.
• Forgetting API versions. Shopify releases a version every quarter and supports each for at least 12 months. Pin a version in the URL and schedule upgrades; bulkOperation(id:) needs 2026-01 or later.
• Testing on a tiny store. Rate limit behavior only shows at scale. Generate a test catalog of realistic size in a sandbox before launch.