allanninal.dev / bigcommerce

Field notes / Catalog / Products

Diagnostic Catalog / Products

V3 pagination breaks when options or modifiers are included

You send limit=250 and add include=options,modifiers so you get variants and modifiers hydrated in the same call. BigCommerce quietly ignores your limit and hands back 10 records a page instead of 250, but meta.pagination.total_pages is still calculated as if your 250 had been honored. Any script that stops walking pages once it hits total_pages stops early, and a chunk of your catalog never makes it into the sync. Here is why the metadata lies and a small script that detects it and works around it.

Python and Node.js BigCommerce V3 Catalog API Read-only diagnostic
An online store on a laptop
Photo by charlesdeluvio on Unsplash
The short answer

BigCommerce's /v3/catalog/products endpoint documents that when include=options, include=modifiers, or include=variants is requested, the server silently caps the page size at 10 items per page no matter what limit you asked for, because hydrating those nested sub-resources per product is expensive to join and serialize. The response's meta.pagination.total is still correct, but total_pages is computed from the same count query used for the plain list, so it understates how many 10-record pages you actually need to walk. A script that stops at meta.pagination.total_pages silently truncates the product set. Run a baseline pull without include, a suspect pull with include=options,modifiers, and compare the product id sets. If any are missing, stop trusting total_pages and page until data comes back empty instead. Full code, tests, and the comparison logic are below.

The problem in plain words

Ask /v3/catalog/products for a plain list, no include, and limit=250 gets you up to 250 products per page, and meta.pagination.total_pages matches reality. Add include=options,modifiers so each product arrives with its options and modifiers already hydrated, and the same request comes back with only 10 records in data, even though you asked for 250. BigCommerce documents this cap directly, because joining and serializing options, modifiers, or variants for every product in a page is expensive, so the server enforces a hard ceiling of 10 records per page whenever those sub-resources are requested.

The part that actually breaks scripts is not the 10-item cap itself, it is that meta.pagination.total_pages does not know about it. That figure is computed from the same underlying count query used for the un-hydrated list, so it reports the number of pages you would need at your requested limit, not at the 10-per-page the server is actually enforcing. A client that walks pages until page > meta.pagination.total_pages runs out of pages long before it runs out of products, and the tail end of the catalog just never shows up. Nothing in the response says this happened. The loop finishes cleanly, the script exits zero, and the missing products are simply gone from whatever you built with the result.

limit=250 include=options,modifiers Server caps page data.length = 10 ✕ total_pages unaware total_pages based on limit=250 not the 10 enforced Loop stops early products missed
The server enforces a real cap of 10 records per page whenever options or modifiers are included, but total_pages is still calculated as if the requested limit had been honored, so a loop that trusts it exits before the catalog is fully read.

Why it happens

This is a documented server behavior, not an accident in any one client library. A few specifics worth knowing:

See the citations at the end for the exact support thread, the SDK issue, and the API reference language.

The key insight

meta.pagination.total_pages is not a promise about how many requests you need to make. It is a number calculated from your requested limit, and the server is free to ignore that same limit when your include list is expensive to hydrate. So the safe pattern is never "loop while page <= total_pages." It is "loop until the response's own data array comes back empty," which is the one signal BigCommerce actually guarantees marks the end of the result set, independent of whatever total_pages claims.

The fix, as a flow

We do not patch anything in the store's catalog, because there is nothing wrong with the products themselves. We run a detector that compares a baseline pull against an include pull, and we change how the client decides when to stop paging.

Baseline pull no include, limit=250 Suspect pull include=options,modifiers Collect id sets baselineIds, includeIds Missing ids found? yes no, trustworthy keep total_pages Flag store use empty data as stop condition
The detector never writes anything. It compares two read-only pulls, and the only decision it makes is which stop condition the client should trust going forward.

Build it step by step

1

Get a store hash and an API access token

Create an API account in your BigCommerce control panel under Settings, API, or reuse an existing app's credentials. This is a read-only diagnostic, so Products (read-only) scope is enough. You need the store hash from your control panel URL and the access token, sent on every call as the X-Auth-Token header. Keep both in environment variables, never in the file.

setup (shell)
pip install requests

export BIGCOMMERCE_STORE_HASH="abc123"
export BIGCOMMERCE_ACCESS_TOKEN="..."
export DRY_RUN="true"   # this script never writes, but the flag stays for consistency
setup (shell)
// Node 18+ has fetch built in, no dependencies needed

export BIGCOMMERCE_STORE_HASH="abc123"
export BIGCOMMERCE_ACCESS_TOKEN="..."
export DRY_RUN="true"   // this script never writes, but the flag stays for consistency
2

Talk to the V3 Catalog API

Every call goes to https://api.bigcommerce.com/stores/{store_hash}/v3/catalog/products with the token in the X-Auth-Token header. A small helper handles GET and raises on a non-2xx response. We use it for both the baseline pull and the suspect pull, changing only the query params.

step2.py
import os, requests

STORE_HASH = os.environ["BIGCOMMERCE_STORE_HASH"]
ACCESS_TOKEN = os.environ["BIGCOMMERCE_ACCESS_TOKEN"]
API_BASE = f"https://api.bigcommerce.com/stores/{STORE_HASH}/v3"

HEADERS = {
    "X-Auth-Token": ACCESS_TOKEN,
    "Content-Type": "application/json",
    "Accept": "application/json",
}

def bc_get(path, params=None):
    r = requests.get(f"{API_BASE}{path}", headers=HEADERS, params=params or {}, timeout=30)
    r.raise_for_status()
    return r.json()
step2.js
const STORE_HASH = process.env.BIGCOMMERCE_STORE_HASH;
const ACCESS_TOKEN = process.env.BIGCOMMERCE_ACCESS_TOKEN;
const API_BASE = `https://api.bigcommerce.com/stores/${STORE_HASH}/v3`;

const HEADERS = {
  "X-Auth-Token": ACCESS_TOKEN,
  "Content-Type": "application/json",
  Accept: "application/json",
};

async function bcGet(path, params = {}) {
  const url = new URL(`${API_BASE}${path}`);
  for (const [key, value] of Object.entries(params)) url.searchParams.set(key, value);
  const res = await fetch(url, { headers: HEADERS });
  if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
  return res.json();
}
3

Walk the baseline pull and the suspect pull separately

The baseline pull has no include, so meta.pagination.total_pages is trustworthy there. Walk it with page += 1 until page > meta.pagination.total_pages, and collect every data[].id plus the reported meta.pagination.total. The suspect pull adds include=options,modifiers and must walk using its own returned metadata for comparison, page by page, collecting the same fields plus the actual data.length the server gave back on each page.

step3.py
def pull_baseline(limit=250):
    page = 1
    ids = []
    total = None
    while True:
        resp = bc_get("/catalog/products", {"limit": limit, "page": page})
        pagination = resp["meta"]["pagination"]
        total = pagination["total"]
        ids.extend(str(p["id"]) for p in resp["data"])
        if page >= pagination["total_pages"]:
            break
        page += 1
    return ids, total

def pull_with_include(include="options,modifiers", limit=250):
    page = 1
    pages = []
    while True:
        resp = bc_get("/catalog/products", {"limit": limit, "page": page, "include": include})
        pages.append(resp)
        if not resp["data"]:
            break
        page += 1
    return pages
step3.js
async function pullBaseline(limit = 250) {
  let page = 1;
  const ids = [];
  let total = null;
  while (true) {
    const resp = await bcGet("/catalog/products", { limit, page });
    const pagination = resp.meta.pagination;
    total = pagination.total;
    ids.push(...resp.data.map((p) => String(p.id)));
    if (page >= pagination.total_pages) break;
    page += 1;
  }
  return { ids, total };
}

async function pullWithInclude(include = "options,modifiers", limit = 250) {
  let page = 1;
  const pages = [];
  while (true) {
    const resp = await bcGet("/catalog/products", { limit, page, include });
    pages.push(resp);
    if (!resp.data.length) break;
    page += 1;
  }
  return pages;
}
4

Reconcile the two pulls with one pure function

Keep the comparison in its own function that takes the baseline id list and the array of pages returned by the include pull, and returns the missing ids plus whether total_pages can be trusted for this store. It flattens every page's data[].id into a set, subtracts it from the baseline, and checks whether the include pull's own total_pages covers the number of 10-record pages actually needed.

reconcile.py
import math

def reconcile_paginated_product_ids(baseline_ids, include_pull_pages):
    include_ids = set()
    for page in include_pull_pages:
        for item in page["data"]:
            include_ids.add(str(item["id"]))

    missing_ids = [pid for pid in baseline_ids if pid not in include_ids]

    per_page = include_pull_pages[0]["meta"]["pagination"]["per_page"]
    implied_full_pages = math.ceil(len(baseline_ids) / per_page)
    reported_total_pages = include_pull_pages[0]["meta"]["pagination"]["total_pages"]

    pagination_trustworthy = (
        reported_total_pages >= implied_full_pages and len(missing_ids) == 0
    )

    return {
        "missingIds": missing_ids,
        "paginationTrustworthy": pagination_trustworthy,
        "recommendedStopCondition": "total_pages" if pagination_trustworthy else "empty_data_array",
    }
reconcile.js
export function reconcilePaginatedProductIds(baselineIds, includePullPages) {
  const includeIds = new Set();
  for (const page of includePullPages) {
    for (const item of page.data) includeIds.add(String(item.id));
  }

  const missingIds = baselineIds.filter((id) => !includeIds.has(id));

  const perPage = includePullPages[0].meta.pagination.per_page;
  const impliedFullPages = Math.ceil(baselineIds.length / perPage);
  const reportedTotalPages = includePullPages[0].meta.pagination.total_pages;

  const paginationTrustworthy =
    reportedTotalPages >= impliedFullPages && missingIds.length === 0;

  return {
    missingIds,
    paginationTrustworthy,
    recommendedStopCondition: paginationTrustworthy ? "total_pages" : "empty_data_array",
  };
}
5

Report, never repair

There is no PATCH or PUT call anywhere in this script, because nothing about the merchant's catalog data is wrong. When missingIds is non-empty, log the store hash, the count, and a sample of the missing product ids and skus, so the integration owner can see exactly what a total_pages-based loop would have dropped. That report is the whole deliverable for this defect.

report.py
def log_report(store_hash, result, sample_size=10):
    if not result["missingIds"]:
        log.info("store=%s pagination is trustworthy, total_pages is safe to use.", store_hash)
        return
    log.warning(
        "store=%s total_pages UNDERSTATES the real page count. missing=%d sample=%s "
        "recommended_stop_condition=%s",
        store_hash, len(result["missingIds"]),
        result["missingIds"][:sample_size], result["recommendedStopCondition"],
    )
report.js
function logReport(storeHash, result, sampleSize = 10) {
  if (!result.missingIds.length) {
    console.log(`store=${storeHash} pagination is trustworthy, total_pages is safe to use.`);
    return;
  }
  console.warn(
    `store=${storeHash} total_pages UNDERSTATES the real page count. ` +
    `missing=${result.missingIds.length} sample=${result.missingIds.slice(0, sampleSize)} ` +
    `recommended_stop_condition=${result.recommendedStopCondition}`
  );
}
6

Wire it together with a dry run guard

The full script never writes to the catalog either way, so DRY_RUN only controls whether it also prints the recommended client-side workaround alongside the report. Run it once against any store where a product sync using include=options or include=modifiers looks like it is coming up short, and again any time you add a new sync path that hydrates those sub-resources.

Run it safe

This script only reads. There is no BigCommerce write endpoint invoked anywhere in it, and there should not be, since this is a response-metadata defect in BigCommerce's API, not a data problem on the merchant's catalog. The only output is a report and a recommended stop condition for your own client code.

The full code

Here is the complete script in one file for each language. It reads settings from the environment, runs the baseline pull, runs the include pull, reconciles the two id sets with the pure function, and logs a report. It never mutates catalog data, so it is safe to run against a production store hash at any time.

View this code on GitHub Full runnable folder with tests in the bigcommerce-fixes repo.

check_include_pagination.py
"""Detect BigCommerce v3 catalog/products pagination truncation with include.

BigCommerce's v3 catalog/products endpoint documents that when include=options,
include=modifiers, or include=variants is requested, the server silently caps the
page size at 10 records per page regardless of the limit query param sent, because
hydrating those nested sub-resources per product is expensive to join and
serialize. meta.pagination.total is still computed correctly, but total_pages is
calculated from the same count query used for the plain, un-hydrated list, so it
understates how many 10-record pages are actually needed. A client that walks
pages until page > meta.pagination.total_pages stops early and silently drops
products from the tail of the catalog.

This script never writes anything. It pulls a baseline list (no include) and a
suspect list (include=options,modifiers), reconciles the product id sets with a
pure function, and logs which product ids the include pull would have missed if
total_pages had been trusted as the stop condition. Safe to run again and again
against a live store.

Guide: https://www.allanninal.dev/bigcommerce/v3-pagination-breaks-with-includes/
"""
import os
import math
import logging

import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("check_include_pagination")

STORE_HASH = os.environ["BIGCOMMERCE_STORE_HASH"]
ACCESS_TOKEN = os.environ["BIGCOMMERCE_ACCESS_TOKEN"]
API_BASE = f"https://api.bigcommerce.com/stores/{STORE_HASH}/v3"
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"

INCLUDE_PARAM = os.environ.get("INCLUDE_PARAM", "options,modifiers")
LIMIT = int(os.environ.get("LIMIT", "250"))
SAMPLE_SIZE = int(os.environ.get("SAMPLE_SIZE", "10"))

HEADERS = {
    "X-Auth-Token": ACCESS_TOKEN,
    "Content-Type": "application/json",
    "Accept": "application/json",
}


def bc_get(path, params=None):
    r = requests.get(f"{API_BASE}{path}", headers=HEADERS, params=params or {}, timeout=30)
    r.raise_for_status()
    return r.json()


def pull_baseline(limit=LIMIT):
    """Walk the un-hydrated list, where total_pages is trustworthy."""
    page = 1
    ids = []
    total = None
    while True:
        resp = bc_get("/catalog/products", {"limit": limit, "page": page})
        pagination = resp["meta"]["pagination"]
        total = pagination["total"]
        ids.extend(str(p["id"]) for p in resp["data"])
        if page >= pagination["total_pages"]:
            break
        page += 1
    return ids, total


def pull_with_include(include=INCLUDE_PARAM, limit=LIMIT):
    """Walk the hydrated list using the authoritative empty-data-array signal."""
    page = 1
    pages = []
    while True:
        resp = bc_get("/catalog/products", {"limit": limit, "page": page, "include": include})
        pages.append(resp)
        if not resp["data"]:
            break
        page += 1
    return pages


def reconcile_paginated_product_ids(baseline_ids, include_pull_pages):
    """Pure decision. No network, no side effects.

    Flattens include_pull_pages[].data[].id into a set, computes missingIds as the
    baseline ids not present in that set, computes the number of 10-record pages
    actually needed against the include pull's own per_page, and compares that
    against the include pull's own reported total_pages. paginationTrustworthy is
    true only when total_pages covers the pages actually needed AND no ids are
    missing. recommendedStopCondition is "total_pages" when trustworthy, otherwise
    "empty_data_array", which is what a caller should switch to.
    """
    include_ids = set()
    for page in include_pull_pages:
        for item in page["data"]:
            include_ids.add(str(item["id"]))

    missing_ids = [pid for pid in baseline_ids if pid not in include_ids]

    per_page = include_pull_pages[0]["meta"]["pagination"]["per_page"]
    implied_full_pages = math.ceil(len(baseline_ids) / per_page) if per_page else 0
    reported_total_pages = include_pull_pages[0]["meta"]["pagination"]["total_pages"]

    pagination_trustworthy = (
        reported_total_pages >= implied_full_pages and len(missing_ids) == 0
    )

    return {
        "missingIds": missing_ids,
        "paginationTrustworthy": pagination_trustworthy,
        "recommendedStopCondition": "total_pages" if pagination_trustworthy else "empty_data_array",
    }


def run():
    baseline_ids, baseline_total = pull_baseline()
    include_pages = pull_with_include()

    result = reconcile_paginated_product_ids(baseline_ids, include_pages)

    if not result["missingIds"]:
        log.info(
            "store=%s baseline_total=%d pagination is trustworthy, total_pages is safe to use.",
            STORE_HASH, baseline_total,
        )
        return

    log.warning(
        "store=%s baseline_total=%d total_pages UNDERSTATES the real page count. "
        "missing=%d sample_ids=%s recommended_stop_condition=%s",
        STORE_HASH, baseline_total, len(result["missingIds"]),
        result["missingIds"][:SAMPLE_SIZE], result["recommendedStopCondition"],
    )
    if DRY_RUN:
        log.info(
            "DRY_RUN=true: report only. Client-side workaround: when include contains "
            "options or modifiers, ignore meta.pagination.total_pages and loop page += 1 "
            "until a response returns data: [] (empty array)."
        )


if __name__ == "__main__":
    run()
check-include-pagination.js
/**
 * Detect BigCommerce v3 catalog/products pagination truncation with include.
 *
 * BigCommerce's v3 catalog/products endpoint documents that when include=options,
 * include=modifiers, or include=variants is requested, the server silently caps the
 * page size at 10 records per page regardless of the limit query param sent, because
 * hydrating those nested sub-resources per product is expensive to join and
 * serialize. meta.pagination.total is still computed correctly, but total_pages is
 * calculated from the same count query used for the plain, un-hydrated list, so it
 * understates how many 10-record pages are actually needed. A client that walks
 * pages until page > meta.pagination.total_pages stops early and silently drops
 * products from the tail of the catalog.
 *
 * This script never writes anything. It pulls a baseline list (no include) and a
 * suspect list (include=options,modifiers), reconciles the product id sets with a
 * pure function, and logs which product ids the include pull would have missed if
 * total_pages had been trusted as the stop condition. Safe to run again and again.
 *
 * Guide: https://www.allanninal.dev/bigcommerce/v3-pagination-breaks-with-includes/
 */
import { pathToFileURL } from "node:url";

const STORE_HASH = process.env.BIGCOMMERCE_STORE_HASH || "example_hash";
const ACCESS_TOKEN = process.env.BIGCOMMERCE_ACCESS_TOKEN || "bc_dummy";
const API_BASE = `https://api.bigcommerce.com/stores/${STORE_HASH}/v3`;
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";

const INCLUDE_PARAM = process.env.INCLUDE_PARAM || "options,modifiers";
const LIMIT = Number(process.env.LIMIT || 250);
const SAMPLE_SIZE = Number(process.env.SAMPLE_SIZE || 10);

const HEADERS = {
  "X-Auth-Token": ACCESS_TOKEN,
  "Content-Type": "application/json",
  Accept: "application/json",
};

/**
 * Pure decision. No network, no side effects.
 *
 * Flattens includePullPages[].data[].id into a set, computes missingIds as the
 * baseline ids not present in that set, computes the number of 10-record pages
 * actually needed against the include pull's own per_page, and compares that
 * against the include pull's own reported total_pages. paginationTrustworthy is
 * true only when total_pages covers the pages actually needed AND no ids are
 * missing. recommendedStopCondition is "total_pages" when trustworthy, otherwise
 * "empty_data_array", which is what a caller should switch to.
 */
export function reconcilePaginatedProductIds(baselineIds, includePullPages) {
  const includeIds = new Set();
  for (const page of includePullPages) {
    for (const item of page.data) includeIds.add(String(item.id));
  }

  const missingIds = baselineIds.filter((id) => !includeIds.has(id));

  const perPage = includePullPages[0].meta.pagination.per_page;
  const impliedFullPages = perPage ? Math.ceil(baselineIds.length / perPage) : 0;
  const reportedTotalPages = includePullPages[0].meta.pagination.total_pages;

  const paginationTrustworthy =
    reportedTotalPages >= impliedFullPages && missingIds.length === 0;

  return {
    missingIds,
    paginationTrustworthy,
    recommendedStopCondition: paginationTrustworthy ? "total_pages" : "empty_data_array",
  };
}

async function bcGet(path, params = {}) {
  const url = new URL(`${API_BASE}${path}`);
  for (const [key, value] of Object.entries(params)) {
    if (value !== undefined && value !== null) url.searchParams.set(key, value);
  }
  const res = await fetch(url, { headers: HEADERS });
  if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
  return res.json();
}

async function pullBaseline(limit = LIMIT) {
  let page = 1;
  const ids = [];
  let total = null;
  while (true) {
    const resp = await bcGet("/catalog/products", { limit, page });
    const pagination = resp.meta.pagination;
    total = pagination.total;
    ids.push(...resp.data.map((p) => String(p.id)));
    if (page >= pagination.total_pages) break;
    page += 1;
  }
  return { ids, total };
}

async function pullWithInclude(include = INCLUDE_PARAM, limit = LIMIT) {
  let page = 1;
  const pages = [];
  while (true) {
    const resp = await bcGet("/catalog/products", { limit, page, include });
    pages.push(resp);
    if (!resp.data.length) break;
    page += 1;
  }
  return pages;
}

export async function run() {
  const { ids: baselineIds, total: baselineTotal } = await pullBaseline();
  const includePages = await pullWithInclude();

  const result = reconcilePaginatedProductIds(baselineIds, includePages);

  if (!result.missingIds.length) {
    console.log(
      `store=${STORE_HASH} baseline_total=${baselineTotal} pagination is trustworthy, total_pages is safe to use.`
    );
    return;
  }

  console.warn(
    `store=${STORE_HASH} baseline_total=${baselineTotal} total_pages UNDERSTATES the real page count. ` +
    `missing=${result.missingIds.length} sample_ids=${JSON.stringify(result.missingIds.slice(0, SAMPLE_SIZE))} ` +
    `recommended_stop_condition=${result.recommendedStopCondition}`
  );
  if (DRY_RUN) {
    console.log(
      "DRY_RUN=true: report only. Client-side workaround: when include contains options " +
      "or modifiers, ignore meta.pagination.total_pages and loop page += 1 until a " +
      "response returns data: [] (empty array)."
    );
  }
}

if (import.meta.url === pathToFileURL(process.argv[1]).href) {
  run().catch((err) => { console.error(err); process.exit(1); });
}

Add a test

The reconciliation rule is the part most worth testing, because it decides whether your sync trusts total_pages or switches to walking until data is empty. Because reconcile_paginated_product_ids takes only plain lists and page objects and returns a plain dict, the test needs no network and no BigCommerce store. It just feeds in canned baseline ids and canned page objects.

test_pagination_reconciler.py
from check_include_pagination import reconcile_paginated_product_ids


def page(ids, total, total_pages, per_page=10):
    return {
        "data": [{"id": i} for i in ids],
        "meta": {"pagination": {"total": total, "total_pages": total_pages, "per_page": per_page}},
    }


def test_trustworthy_when_all_ids_present_and_total_pages_covers_them():
    baseline_ids = [str(i) for i in range(1, 21)]
    pages = [page(range(1, 11), 20, 2), page(range(11, 21), 20, 2)]
    result = reconcile_paginated_product_ids(baseline_ids, pages)
    assert result["missingIds"] == []
    assert result["paginationTrustworthy"] is True
    assert result["recommendedStopCondition"] == "total_pages"


def test_untrustworthy_when_include_pull_is_truncated_by_total_pages():
    baseline_ids = [str(i) for i in range(1, 31)]
    # Requested limit=250 but server capped at 10/page; total_pages wrongly says 1.
    pages = [page(range(1, 11), 30, 1, per_page=10)]
    result = reconcile_paginated_product_ids(baseline_ids, pages)
    assert result["missingIds"] == [str(i) for i in range(11, 31)]
    assert result["paginationTrustworthy"] is False
    assert result["recommendedStopCondition"] == "empty_data_array"


def test_untrustworthy_when_ids_missing_even_if_total_pages_looks_sufficient():
    baseline_ids = [str(i) for i in range(1, 11)]
    pages = [page(range(1, 9), 10, 1, per_page=10)]
    result = reconcile_paginated_product_ids(baseline_ids, pages)
    assert result["missingIds"] == ["9", "10"]
    assert result["paginationTrustworthy"] is False
    assert result["recommendedStopCondition"] == "empty_data_array"


def test_trustworthy_when_baseline_is_empty():
    result = reconcile_paginated_product_ids([], [page([], 0, 0)])
    assert result["missingIds"] == []
    assert result["paginationTrustworthy"] is True
    assert result["recommendedStopCondition"] == "total_pages"
check-include-pagination.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { reconcilePaginatedProductIds } from "./check-include-pagination.js";

function page(ids, total, totalPages, perPage = 10) {
  return {
    data: ids.map((id) => ({ id })),
    meta: { pagination: { total, total_pages: totalPages, per_page: perPage } },
  };
}

function range(start, end) {
  const out = [];
  for (let i = start; i < end; i++) out.push(i);
  return out;
}

test("trustworthy when all ids present and total_pages covers them", () => {
  const baselineIds = range(1, 21).map(String);
  const pages = [page(range(1, 11), 20, 2), page(range(11, 21), 20, 2)];
  const result = reconcilePaginatedProductIds(baselineIds, pages);
  assert.deepEqual(result.missingIds, []);
  assert.equal(result.paginationTrustworthy, true);
  assert.equal(result.recommendedStopCondition, "total_pages");
});

test("untrustworthy when include pull is truncated by total_pages", () => {
  const baselineIds = range(1, 31).map(String);
  const pages = [page(range(1, 11), 30, 1, 10)];
  const result = reconcilePaginatedProductIds(baselineIds, pages);
  assert.deepEqual(result.missingIds, range(11, 31).map(String));
  assert.equal(result.paginationTrustworthy, false);
  assert.equal(result.recommendedStopCondition, "empty_data_array");
});

test("untrustworthy when ids missing even if total_pages looks sufficient", () => {
  const baselineIds = range(1, 11).map(String);
  const pages = [page(range(1, 9), 10, 1, 10)];
  const result = reconcilePaginatedProductIds(baselineIds, pages);
  assert.deepEqual(result.missingIds, ["9", "10"]);
  assert.equal(result.paginationTrustworthy, false);
  assert.equal(result.recommendedStopCondition, "empty_data_array");
});

test("trustworthy when baseline is empty", () => {
  const result = reconcilePaginatedProductIds([], [page([], 0, 0)]);
  assert.deepEqual(result.missingIds, []);
  assert.equal(result.paginationTrustworthy, true);
  assert.equal(result.recommendedStopCondition, "total_pages");
});

Case studies

Nightly PIM sync

The catalog that lost its last few hundred products every night

A merchant with roughly 6,000 SKUs ran a nightly job into their PIM that requested include=options,modifiers so variants and modifiers would already be attached in one pass. The job walked pages using meta.pagination.total_pages and finished cleanly every night, but the PIM's product count kept drifting a few hundred short of the storefront's actual count, and nobody could see why in the logs.

Running the baseline pull against the include pull showed the include pull was capped at 10 records per page while total_pages was still calculated off the requested limit=250. The job was walking exactly the number of pages total_pages claimed and stopping four fifths of the way through the real result set every single night.

Variant export

The export tool that only ever exported the newest products

An export tool pulling include=variants for a price feed always seemed to cover recently added products but quietly dropped older ones near the end of the id range. Because the products were sorted by id ascending and the job stopped at a total_pages that undercounted the real pages, the tail of the catalog, the oldest products, never got reached.

Switching the stop condition to an empty data array immediately picked up the missing tail, and the reconciler's report showed exactly which ids and skus the old logic had been dropping every run.

What good looks like

Once any client pulling include=options, include=modifiers, or include=variants stops trusting meta.pagination.total_pages and instead loops until data comes back empty, the truncation disappears entirely, because that is the one signal BigCommerce actually enforces consistently. The reconciler above becomes a one-time check per integration, not a recurring job, since the fix lives in the client's loop condition, not in anything BigCommerce needs to change.

FAQ

Why does my product sync lose items only when I add include=options or include=modifiers?

BigCommerce's v3 catalog/products endpoint documents that when options, modifiers, or variants are included, the server silently caps the page size at 10 records regardless of the limit you sent. But meta.pagination.total_pages is still computed from the same count query used for the plain, un-hydrated list, so it understates how many pages you actually need to walk at the real 10-per-page size, and any loop that stops at total_pages truncates the product set.

Is limit=250 with include=options just being ignored?

Not ignored, overridden. BigCommerce accepts the limit query param but caps the actual per-page record count at 10 whenever options or modifiers (or variants) are in the include list, because hydrating and serializing those nested sub-resources per product is expensive to join. The response still reports meta.pagination.total correctly, but total_pages is calculated as if your requested limit were honored, not the enforced 10.

What should I use as the stop condition instead of meta.pagination.total_pages?

When your request includes options or modifiers, ignore total_pages entirely and keep incrementing page until a response comes back with an empty data array. That is the only condition BigCommerce actually guarantees marks the end of the result set, regardless of what the pagination metadata claims about page counts.

Related field notes

Orders stuck on Awaiting Payment after capture Transactions total does not match the order Browse all BigCommerce field notes

Citations

On the problem:

  1. BigCommerce Support: bug with pagination when including options/modifiers in Products v3 API. support.bigcommerce.com bug with pagination when including options/modifiers
  2. bigcommerce/bigcommerce-api-python, GitHub Issue #59: unable to get meta.pagination.total_pages from Products.all(). github.com bigcommerce-api-python issue 59
  3. BigCommerce Support: problem with paging and GET catalog/variants API (v3). bigcommerce.my.site.com problem with paging and GET catalog/variants

On the solution:

  1. BigCommerce API Reference: List Products (Catalog v3), include parameter and pagination behavior. developer.bigcommerce.com list products
  2. BigCommerce Developer Center: Products, rest-catalog/products, include sub-resources behavior. developer.bigcommerce.com rest-catalog/products
  3. BigCommerce Developer Center: Filtering / Common Query Params, pagination and limit. developer.bigcommerce.com common query params

Stuck on a tricky one?

If you have a problem in BigCommerce catalog, orders, payments, webhooks, or inventory that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.

Contact me on LinkedIn

Did this catch a truncated sync?

If this explained a mystery gap in your product sync or export, you can buy me a coffee. It is the best way to keep these field notes free and growing.

Buy me a coffee on Ko-fi

Back to all BigCommerce field notes