Repair WooCommerce Subscriptions: switches, coupons, and data
Bulk subscription export runs out of memory
You click "Export all subscriptions" on a store with tens of thousands of them, and a few minutes later the process is just gone. No file, no error a normal person can read, sometimes a log line about an allowed memory size being exhausted. The export was never broken by bad data, it was broken by trying to hold every subscription in memory at the same time. Here is why that happens and a small script that exports in safe, paged batches instead.
A bulk export runs out of memory because it asks the WooCommerce REST API for every subscription in one request, or it appends every page into one big list before writing the CSV. Either way, the whole export has to fit in RAM at once. Fetch one page at a time, write each page straight to disk as it arrives, and never hold more than one page in memory. If a page is still too heavy for your memory budget, shrink the page size and try that page again. Full code, tests, and a dry run guard are below.
The problem in plain words
A subscription export sounds simple: read every subscription, write a row for each one, save the file. The trouble starts with how "every subscription" gets fetched. Two shortcuts cause almost all of these crashes.
The first shortcut is asking the WooCommerce REST API for an enormous per_page, hoping to get everything back in one response. Woo caps per_page at 100 by default, but even a script that respects the cap can still fail the second way: it fetches page after page and appends every row into one Python list or JavaScript array, meaning the whole export lives in memory until the very last row is fetched, and only then gets written out. On a store with 50,000 subscriptions, that list can hold megabytes of JSON just for the raw rows before you have even converted anything to CSV. PHP behind the API, or the worker process running your script, has a memory limit, and once the export crosses it the process is simply killed. There is rarely a clean error, just a stopped job and a partial or missing file.
Why it happens
WooCommerce Subscriptions stores can grow into the tens of thousands of rows once a store has been running for a year or two, and export tooling is usually written and tested against a small store where the naive approach never shows a problem. A few things make it worse in production:
- WordPress hosting commonly sets the PHP memory limit between 128MB and 512MB, and a shared host will not raise it just for your export.
- A script written in Python or Node against the REST API has its own process memory limit too, separate from the site's PHP limit, and a long list of JSON objects is heavier in memory than the same data on the wire.
- CSV writing libraries that build the whole file as one string in memory before writing it double the problem, since now both the row list and the file content are held at once.
- High Performance Order Storage (HPOS) changed how subscription and order data is stored under the hood, but it does not change how much data a badly written export tries to hold at once. The REST API still pages the same way either way.
- Retrying a failed export usually means starting over from row one, so a store that crashes at row 40,000 out of 50,000 never actually finishes without a change to how the export works.
This is a well known WordPress and WooCommerce pattern. Both the WordPress core handbook and third party writeups on WooCommerce exports point at the same root cause: holding an unbounded result set in memory instead of streaming it.
An export never needs more than one page of subscriptions in memory at any moment. The moment you write a page to disk, you can throw it away. Anything that keeps growing while the export runs, like one big list of every row fetched so far, is the actual bug, not the number of subscriptions in the store.
The fix, as a flow
We do not change how subscriptions are stored or how the export is triggered. We change how the export moves data. Fetch one page, write it, forget it, fetch the next page. A small planner decides the page size for the next request from plain numbers: how many rows came back, roughly how big that page was, and the memory budget you set. If a page turns out heavier than the budget, the planner shrinks the page size and the same page is requested again at the smaller size, instead of ever holding a page that is too large.
Build it step by step
Get read access to the store
This job only reads subscriptions, so a WooCommerce REST API key pair with read access is enough. Create one under WooCommerce, Settings, Advanced, REST API. Keep the keys in environment variables, never in the file, and set a memory budget that is comfortably below whatever limit your PHP host or your script's runtime actually enforces.
pip install requests
export WOO_STORE_URL="https://yourstore.com"
export WOO_CONSUMER_KEY="ck_..."
export WOO_CONSUMER_SECRET="cs_..."
export EXPORT_PATH="subscriptions_export.csv"
export START_PAGE_SIZE="100"
export MIN_PAGE_SIZE="10"
export MEMORY_BUDGET_MB="150"
export DRY_RUN="true" # start safe, change to false to write the filenpm install # uses the built-in fetch, no extra package needed
export WOO_STORE_URL="https://yourstore.com"
export WOO_CONSUMER_KEY="ck_..."
export WOO_CONSUMER_SECRET="cs_..."
export EXPORT_PATH="subscriptions_export.csv"
export START_PAGE_SIZE="100"
export MIN_PAGE_SIZE="10"
export MEMORY_BUDGET_MB="150"
export DRY_RUN="true" // start safe, change to false to write the fileFetch one page at a time, nothing more
Call the WooCommerce REST API for subscriptions with a small per_page and an increasing page number. Order by id so every page is stable even if new subscriptions are created while the export runs. Keep the raw response bytes around just long enough to estimate how heavy the page was.
import requests
from requests.auth import HTTPBasicAuth
WOO_URL = "https://yourstore.com"
AUTH = HTTPBasicAuth("ck_...", "cs_...")
def fetch_page(page, page_size):
r = requests.get(
f"{WOO_URL}/wp-json/wc/v3/subscriptions",
params={"per_page": page_size, "page": page, "orderby": "id", "order": "asc"},
auth=AUTH, timeout=60,
)
r.raise_for_status()
return r.content, r.json()const WOO_URL = "https://yourstore.com";
const AUTH = "Basic " + Buffer.from("ck_...:cs_...").toString("base64");
async function fetchPage(page, pageSize) {
const url = `${WOO_URL}/wp-json/wc/v3/subscriptions?per_page=${pageSize}&page=${page}&orderby=id&order=asc`;
const res = await fetch(url, { headers: { Authorization: AUTH } });
if (!res.ok) throw new Error(`Woo subscriptions page ${page} returned ${res.status}`);
const text = await res.text();
return [Buffer.byteLength(text), JSON.parse(text)];
}Write each page to disk immediately
As soon as a page comes back, write its rows straight to the CSV file and flush. Do not append the rows to a list that lives past this point. Once a page has been written, nothing about it needs to stay in memory.
import csv
FIELDS = ["id", "status", "total", "billing_period", "billing_interval", "next_payment_date", "customer_id"]
def to_row(sub):
return {field: sub.get(field, "") for field in FIELDS}
def write_page(writer, rows):
for sub in rows:
writer.writerow(to_row(sub))
# rows and writer.writerow's work are done, nothing kept beyond this callconst FIELDS = ["id", "status", "total", "billing_period", "billing_interval", "next_payment_date", "customer_id"];
function toRow(sub) {
const row = {};
for (const field of FIELDS) row[field] = sub[field] ?? "";
return row;
}
function csvLine(row) {
return FIELDS.map((f) => String(row[f] ?? "")).join(",");
}
function writePage(out, rows) {
for (const sub of rows) out.write(csvLine(toRow(sub)) + "\n");
// out is a file stream; nothing about this page is kept beyond this call
}Decide the next step with one pure function
The planner never touches the network or the file. It only looks at plain numbers: the current page size, how many rows the last page had, how many bytes that page was, how many rows are written so far, and the memory budget. It returns one of three actions. Continue at the current size. Shrink the page size and retry the same page. Or stop, because there is nothing left or a row cap was reached.
MIN_PAGE_SIZE = 10
def plan_next_page(state):
page_size = state["page_size"]
rows_in_last_page = state["rows_in_last_page"]
last_page_bytes = state["last_page_bytes"]
total_rows_so_far = state["total_rows_so_far"]
max_rows = state.get("max_rows")
memory_budget_mb = state["memory_budget_mb"]
if max_rows is not None and total_rows_so_far >= max_rows:
return ("stop_done", "row cap reached")
if rows_in_last_page == 0 and state.get("has_fetched_a_page"):
return ("stop_done", "no more subscriptions")
budget_bytes = memory_budget_mb * 1024 * 1024
if last_page_bytes > budget_bytes and page_size > MIN_PAGE_SIZE:
return ("shrink", "last page was too heavy for the memory budget")
return ("continue", "keep paging at the current size")const MIN_PAGE_SIZE = 10;
export function planNextPage(state) {
const {
pageSize, rowsInLastPage, lastPageBytes, totalRowsSoFar,
maxRows = null, memoryBudgetMb, hasFetchedAPage = false,
} = state;
if (maxRows !== null && totalRowsSoFar >= maxRows) {
return ["stop_done", "row cap reached"];
}
if (rowsInLastPage === 0 && hasFetchedAPage) {
return ["stop_done", "no more subscriptions"];
}
const budgetBytes = memoryBudgetMb * 1024 * 1024;
if (lastPageBytes > budgetBytes && pageSize > MIN_PAGE_SIZE) {
return ["shrink", "last page was too heavy for the memory budget"];
}
return ["continue", "keep paging at the current size"];
}Wire it together with a dry run guard
The main loop asks the planner what to do, fetches or shrinks accordingly, and only writes the file when DRY_RUN is off. On the first run, leave it on so you can see how many pages and roughly how much data the export expects to move before anything is written. Once that looks right, switch it off.
Always start with DRY_RUN=true. It fetches and counts every page without writing the CSV, so you can see how big the export really is before you let it run for real, and set a realistic MEMORY_BUDGET_MB from what you saw.
The full code
Here is the complete export in one file for each language. It reads settings from the environment, pages through subscriptions with the pure planner deciding page size, writes each page straight to the CSV as it arrives, and respects the dry run flag so you can preview it safely first.
View this code on GitHub Full runnable folder with tests in the woocommerce-fixes repo.
"""Export every WooCommerce Subscription to CSV without loading them all into memory.
A "export all subscriptions" job that calls the REST API once with a huge
per_page, or that appends every page into one Python list before writing the
file, grows without bound as the store grows. On a store with tens of
thousands of subscriptions this is what runs out of memory and gets killed
partway through, usually with a half written, unusable CSV file.
This script fetches one page at a time, writes each row to disk as soon as
it arrives, and never keeps more than one page of subscriptions in memory.
A pure planner function decides the next step (keep paging, shrink the page
size, or stop) from plain numbers, so the paging logic can be unit tested
with no network and no real file.
Read only against WooCommerce. Safe to run again and again. Run on a
schedule or by hand whenever you need a fresh export.
"""
import csv
import os
import logging
import requests
from requests.auth import HTTPBasicAuth
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("export_subscriptions")
WOO_URL = os.environ.get("WOO_STORE_URL", "https://example.com").rstrip("/")
AUTH = HTTPBasicAuth(
os.environ.get("WOO_CONSUMER_KEY", "ck_dummy"),
os.environ.get("WOO_CONSUMER_SECRET", "cs_dummy"),
)
EXPORT_PATH = os.environ.get("EXPORT_PATH", "subscriptions_export.csv")
START_PAGE_SIZE = int(os.environ.get("START_PAGE_SIZE", "100"))
MIN_PAGE_SIZE = int(os.environ.get("MIN_PAGE_SIZE", "10"))
MEMORY_BUDGET_MB = int(os.environ.get("MEMORY_BUDGET_MB", "150"))
MAX_ROWS = int(os.environ.get("MAX_ROWS", "0")) or None # 0 means no cap
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
FIELDS = ["id", "status", "total", "billing_period", "billing_interval", "next_payment_date", "customer_id"]
def bytes_per_row_estimate(page_bytes, row_count):
"""Average bytes per subscription in a page. Zero rows means no data to size from."""
if row_count <= 0:
return 0
return page_bytes / row_count
def plan_next_page(state):
"""Pure planner. Decides the next paging action from plain numbers only.
state keys:
page_size current rows requested per page
rows_in_last_page rows actually returned by the last request (0 if none yet)
last_page_bytes approximate size in bytes of the last page's JSON
total_rows_so_far rows written to the CSV so far
max_rows cap on total rows to export, or None for no cap
memory_budget_mb the memory ceiling we plan against, in megabytes
Returns one of:
("stop_done", reason) no more pages, or the row cap was reached
("shrink", reason) a page was too heavy for the budget, halve the size and retry
("continue", reason) request the next page at the current page_size
"""
page_size = state["page_size"]
rows_in_last_page = state["rows_in_last_page"]
last_page_bytes = state["last_page_bytes"]
total_rows_so_far = state["total_rows_so_far"]
max_rows = state.get("max_rows")
memory_budget_mb = state["memory_budget_mb"]
if max_rows is not None and total_rows_so_far >= max_rows:
return ("stop_done", "row cap reached")
if rows_in_last_page == 0 and state.get("has_fetched_a_page"):
return ("stop_done", "no more subscriptions")
budget_bytes = memory_budget_mb * 1024 * 1024
# A page is only ever held in memory once, briefly, right after the request
# returns and before it is written and discarded. If that one page alone
# would already blow the budget, shrink the page size and try again rather
# than ever holding a bigger page.
if last_page_bytes > budget_bytes and page_size > MIN_PAGE_SIZE:
return ("shrink", "last page was too heavy for the memory budget")
return ("continue", "keep paging at the current size")
def next_page_size(current_page_size):
"""Halve the page size on a shrink, but never below MIN_PAGE_SIZE."""
return max(MIN_PAGE_SIZE, current_page_size // 2)
def to_row(sub):
"""Flatten one subscription REST object to the CSV row we export."""
return {field: sub.get(field, "") for field in FIELDS}
def fetch_page(page, page_size):
r = requests.get(
f"{WOO_URL}/wp-json/wc/v3/subscriptions",
params={"per_page": page_size, "page": page, "orderby": "id", "order": "asc"},
auth=AUTH,
timeout=60,
)
r.raise_for_status()
return r.content, r.json()
def run():
if DRY_RUN:
log.info("DRY_RUN is true. Counting subscriptions and planning pages, not writing %s.", EXPORT_PATH)
page = 1
page_size = START_PAGE_SIZE
total_rows_so_far = 0
state = {
"page_size": page_size,
"rows_in_last_page": 0,
"last_page_bytes": 0,
"total_rows_so_far": 0,
"max_rows": MAX_ROWS,
"memory_budget_mb": MEMORY_BUDGET_MB,
"has_fetched_a_page": False,
}
writer = None
fh = None
if not DRY_RUN:
fh = open(EXPORT_PATH, "w", newline="", encoding="utf-8")
writer = csv.DictWriter(fh, fieldnames=FIELDS)
writer.writeheader()
try:
while True:
action, reason = plan_next_page(state)
if action == "stop_done":
log.info("Stopping: %s.", reason)
break
if action == "shrink":
page_size = next_page_size(page_size)
log.warning("Shrinking page size to %d: %s.", page_size, reason)
state["page_size"] = page_size
# Retry the same page number at the smaller size, do not advance.
state["last_page_bytes"] = 0
continue
raw_bytes, rows = fetch_page(page, page_size)
row_count = len(rows)
if not DRY_RUN:
for sub in rows:
writer.writerow(to_row(sub))
fh.flush() # push each page to disk, never buffer pages in memory
total_rows_so_far += row_count
log.info(
"Page %d: %d subscription(s) at page size %d (%d bytes).",
page, row_count, page_size, len(raw_bytes),
)
state.update({
"page_size": page_size,
"rows_in_last_page": row_count,
"last_page_bytes": len(raw_bytes),
"total_rows_so_far": total_rows_so_far,
"has_fetched_a_page": True,
})
page += 1
finally:
if fh:
fh.close()
log.info("Done. %d subscription row(s) %s.", total_rows_so_far, "counted" if DRY_RUN else f"written to {EXPORT_PATH}")
if __name__ == "__main__":
run()/**
* Export every WooCommerce Subscription to CSV without loading them all into memory.
*
* A "export all subscriptions" job that calls the REST API once with a huge
* per_page, or that appends every page into one array before writing the
* file, grows without bound as the store grows. On a store with tens of
* thousands of subscriptions this is what runs out of memory and gets killed
* partway through, usually with a half written, unusable CSV file.
*
* This script fetches one page at a time, writes each row to disk as soon as
* it arrives, and never keeps more than one page of subscriptions in memory.
* A pure planner function decides the next step (keep paging, shrink the
* page size, or stop) from plain numbers, so the paging logic can be unit
* tested with no network and no real file.
*
* Read only against WooCommerce. Safe to run again and again.
* Guide: https://www.allanninal.dev/woocommerce/bulk-subscription-export-runs-out-of-memory/
*/
import { createWriteStream } from "node:fs";
import { pathToFileURL } from "node:url";
const WOO_URL = (process.env.WOO_STORE_URL || "https://example.com").replace(/\/$/, "");
const AUTH = "Basic " + Buffer.from(
`${process.env.WOO_CONSUMER_KEY || "ck_dummy"}:${process.env.WOO_CONSUMER_SECRET || "cs_dummy"}`
).toString("base64");
const EXPORT_PATH = process.env.EXPORT_PATH || "subscriptions_export.csv";
const START_PAGE_SIZE = Number(process.env.START_PAGE_SIZE || 100);
const MIN_PAGE_SIZE = Number(process.env.MIN_PAGE_SIZE || 10);
const MEMORY_BUDGET_MB = Number(process.env.MEMORY_BUDGET_MB || 150);
const MAX_ROWS = Number(process.env.MAX_ROWS || 0) || null; // 0 means no cap
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const FIELDS = ["id", "status", "total", "billing_period", "billing_interval", "next_payment_date", "customer_id"];
export function bytesPerRowEstimate(pageBytes, rowCount) {
// Average bytes per subscription in a page. Zero rows means no data to size from.
if (rowCount <= 0) return 0;
return pageBytes / rowCount;
}
/**
* Pure planner. Decides the next paging action from plain numbers only.
*
* state keys:
* pageSize current rows requested per page
* rowsInLastPage rows actually returned by the last request (0 if none yet)
* lastPageBytes approximate size in bytes of the last page's JSON
* totalRowsSoFar rows written to the CSV so far
* maxRows cap on total rows to export, or null for no cap
* memoryBudgetMb the memory ceiling we plan against, in megabytes
*
* Returns [action, reason] where action is one of:
* "stop_done" no more pages, or the row cap was reached
* "shrink" a page was too heavy for the budget, halve the size and retry
* "continue" request the next page at the current pageSize
*/
export function planNextPage(state) {
const {
pageSize,
rowsInLastPage,
lastPageBytes,
totalRowsSoFar,
maxRows = null,
memoryBudgetMb,
hasFetchedAPage = false,
} = state;
if (maxRows !== null && totalRowsSoFar >= maxRows) {
return ["stop_done", "row cap reached"];
}
if (rowsInLastPage === 0 && hasFetchedAPage) {
return ["stop_done", "no more subscriptions"];
}
const budgetBytes = memoryBudgetMb * 1024 * 1024;
// A page is only ever held in memory once, briefly, right after the request
// returns and before it is written and discarded. If that one page alone
// would already blow the budget, shrink the page size and try again rather
// than ever holding a bigger page.
if (lastPageBytes > budgetBytes && pageSize > MIN_PAGE_SIZE) {
return ["shrink", "last page was too heavy for the memory budget"];
}
return ["continue", "keep paging at the current size"];
}
export function nextPageSize(currentPageSize) {
// Halve the page size on a shrink, but never below MIN_PAGE_SIZE.
return Math.max(MIN_PAGE_SIZE, Math.floor(currentPageSize / 2));
}
export function toRow(sub) {
// Flatten one subscription REST object to the CSV row we export.
const row = {};
for (const field of FIELDS) row[field] = sub[field] ?? "";
return row;
}
function csvLine(row) {
return FIELDS.map((f) => {
const value = String(row[f] ?? "");
return /[",\n]/.test(value) ? `"${value.replace(/"/g, '""')}"` : value;
}).join(",");
}
async function fetchPage(page, pageSize) {
const url = `${WOO_URL}/wp-json/wc/v3/subscriptions?per_page=${pageSize}&page=${page}&orderby=id&order=asc`;
const res = await fetch(url, { headers: { Authorization: AUTH } });
if (!res.ok) throw new Error(`Woo subscriptions page ${page} returned ${res.status}`);
const text = await res.text();
return [Buffer.byteLength(text), JSON.parse(text)];
}
export async function run() {
if (DRY_RUN) {
console.log(`DRY_RUN is true. Counting subscriptions and planning pages, not writing ${EXPORT_PATH}.`);
}
let page = 1;
let pageSize = START_PAGE_SIZE;
let totalRowsSoFar = 0;
let state = {
pageSize,
rowsInLastPage: 0,
lastPageBytes: 0,
totalRowsSoFar: 0,
maxRows: MAX_ROWS,
memoryBudgetMb: MEMORY_BUDGET_MB,
hasFetchedAPage: false,
};
const out = DRY_RUN ? null : createWriteStream(EXPORT_PATH);
if (out) out.write(FIELDS.join(",") + "\n");
while (true) {
const [action, reason] = planNextPage(state);
if (action === "stop_done") {
console.log(`Stopping: ${reason}.`);
break;
}
if (action === "shrink") {
pageSize = nextPageSize(pageSize);
console.warn(`Shrinking page size to ${pageSize}: ${reason}.`);
state = { ...state, pageSize, lastPageBytes: 0 };
continue; // retry the same page number at the smaller size
}
const [pageBytes, rows] = await fetchPage(page, pageSize);
const rowCount = rows.length;
if (out) {
for (const sub of rows) out.write(csvLine(toRow(sub)) + "\n"); // one page at a time, never buffered
}
totalRowsSoFar += rowCount;
console.log(`Page ${page}: ${rowCount} subscription(s) at page size ${pageSize} (${pageBytes} bytes).`);
state = {
...state,
pageSize,
rowsInLastPage: rowCount,
lastPageBytes: pageBytes,
totalRowsSoFar,
hasFetchedAPage: true,
};
page++;
}
if (out) out.end();
console.log(`Done. ${totalRowsSoFar} subscription row(s) ${DRY_RUN ? "counted" : `written to ${EXPORT_PATH}`}.`);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((e) => { console.error(e); process.exit(1); });
}Add a test
The planner is the part most worth testing, because it decides whether the export stays inside the memory budget on a store you have never seen the size of. Because plan_next_page takes plain numbers and returns a plain answer, the test needs no network and no real file. It just builds a state object and checks the action.
from export_subscriptions import plan_next_page, next_page_size, bytes_per_row_estimate
def state(**over):
base = {
"page_size": 100,
"rows_in_last_page": 0,
"last_page_bytes": 0,
"total_rows_so_far": 0,
"max_rows": None,
"memory_budget_mb": 150,
"has_fetched_a_page": False,
}
base.update(over)
return base
def test_continue_on_the_first_request():
assert plan_next_page(state())[0] == "continue"
def test_continue_when_page_fits_comfortably_in_budget():
s = state(rows_in_last_page=100, last_page_bytes=200_000, has_fetched_a_page=True)
assert plan_next_page(s)[0] == "continue"
def test_stop_done_when_a_page_returns_no_rows():
s = state(rows_in_last_page=0, has_fetched_a_page=True)
assert plan_next_page(s)[0] == "stop_done"
def test_stop_done_when_row_cap_reached():
s = state(total_rows_so_far=500, max_rows=500)
assert plan_next_page(s)[0] == "stop_done"
def test_shrink_when_last_page_blew_the_memory_budget():
huge_bytes = 300 * 1024 * 1024 # 300MB, over the 150MB budget
s = state(page_size=100, rows_in_last_page=100, last_page_bytes=huge_bytes, has_fetched_a_page=True)
assert plan_next_page(s)[0] == "shrink"
def test_no_shrink_below_the_minimum_page_size():
huge_bytes = 300 * 1024 * 1024
s = state(page_size=10, rows_in_last_page=10, last_page_bytes=huge_bytes, has_fetched_a_page=True)
# Already at MIN_PAGE_SIZE (10), so we keep going rather than shrink forever.
assert plan_next_page(s)[0] == "continue"
def test_next_page_size_halves():
assert next_page_size(100) == 50
def test_next_page_size_never_below_minimum():
assert next_page_size(12) == 10 # MIN_PAGE_SIZE is 10
assert next_page_size(4) == 10
def test_bytes_per_row_estimate_normal():
assert bytes_per_row_estimate(1000, 100) == 10
def test_bytes_per_row_estimate_zero_rows_is_zero():
assert bytes_per_row_estimate(1000, 0) == 0import { test } from "node:test";
import assert from "node:assert/strict";
import { planNextPage, nextPageSize, bytesPerRowEstimate, toRow } from "./export-subscriptions.js";
const state = (over = {}) => ({
pageSize: 100,
rowsInLastPage: 0,
lastPageBytes: 0,
totalRowsSoFar: 0,
maxRows: null,
memoryBudgetMb: 150,
hasFetchedAPage: false,
...over,
});
test("continue on the first request", () => {
assert.equal(planNextPage(state())[0], "continue");
});
test("continue when page fits comfortably in budget", () => {
const s = state({ rowsInLastPage: 100, lastPageBytes: 200_000, hasFetchedAPage: true });
assert.equal(planNextPage(s)[0], "continue");
});
test("stop_done when a page returns no rows", () => {
const s = state({ rowsInLastPage: 0, hasFetchedAPage: true });
assert.equal(planNextPage(s)[0], "stop_done");
});
test("stop_done when row cap reached", () => {
const s = state({ totalRowsSoFar: 500, maxRows: 500 });
assert.equal(planNextPage(s)[0], "stop_done");
});
test("shrink when last page blew the memory budget", () => {
const hugeBytes = 300 * 1024 * 1024; // 300MB, over the 150MB budget
const s = state({ pageSize: 100, rowsInLastPage: 100, lastPageBytes: hugeBytes, hasFetchedAPage: true });
assert.equal(planNextPage(s)[0], "shrink");
});
test("no shrink below the minimum page size", () => {
const hugeBytes = 300 * 1024 * 1024;
const s = state({ pageSize: 10, rowsInLastPage: 10, lastPageBytes: hugeBytes, hasFetchedAPage: true });
// Already at MIN_PAGE_SIZE (10), so we keep going rather than shrink forever.
assert.equal(planNextPage(s)[0], "continue");
});
test("nextPageSize halves", () => {
assert.equal(nextPageSize(100), 50);
});
test("nextPageSize never below minimum", () => {
assert.equal(nextPageSize(12), 10); // MIN_PAGE_SIZE is 10
assert.equal(nextPageSize(4), 10);
});
test("bytesPerRowEstimate normal", () => {
assert.equal(bytesPerRowEstimate(1000, 100), 10);
});
test("bytesPerRowEstimate zero rows is zero", () => {
assert.equal(bytesPerRowEstimate(1000, 0), 0);
});
test("toRow keeps only known fields with fallback empty string", () => {
const row = toRow({ id: 42, status: "active", total: "19.99", extra_field: "ignored" });
assert.equal(row.id, 42);
assert.equal(row.status, "active");
assert.equal(row.customer_id, "");
assert.equal(row.extra_field, undefined);
});Case studies
The export that always died at the same row
A store with about 60,000 subscriptions had an export tool that appended every page into one list before writing the CSV. It always failed around row 42,000, right when the growing list finally crossed the host's memory limit, and the error log just said "allowed memory size exhausted" with no useful stack trace.
Switching to a page-write-discard loop with a 150MB budget fixed it on the first real run. Memory use stayed flat the entire time, and the export finished in about six minutes instead of crashing every time.
The host that would not raise the memory limit
A smaller store was stuck at a 128MB PHP memory limit their host refused to change. Even a supposedly paged export script was still building one big array client side before writing, so it hit the ceiling anyway once the store passed around 8,000 subscriptions.
The planner's shrink step meant the export adapted itself, dropping from a 100 row page size to 25 automatically the moment a page came back heavier than expected, and it finished cleanly without anyone touching the hosting configuration.
Once the export pages and writes as it goes, the number of subscriptions in the store stops being a risk factor. Memory use stays about the same whether the store has 1,000 subscriptions or 500,000. If a page ever does come back heavier than planned, the export shrinks itself and keeps going instead of getting killed.
FAQ
Why does exporting all my WooCommerce subscriptions run out of memory?
Most export scripts ask the REST API for every subscription in one call, or they build one giant list in memory by appending every page before writing the file. Either way, the whole export has to fit in memory at once, and on a store with tens of thousands of subscriptions that is more than PHP or Node is allowed to use, so the process is killed partway through.
What page size should I use for a large export?
Start around 100 rows per request and write each page to disk immediately instead of collecting pages in a list. If a page still comes back too large for your memory budget, halve the page size and try that same page again rather than pushing on with a page that is too heavy.
Is it safe to run the export again if it fails partway through?
Yes. The export only reads subscriptions and writes to a local CSV file, it never changes an order or a subscription. If it stops partway through, delete the partial file and run it again, or start from a later page once you add resume support.
Related field notes
Citations
On the problem:
- WordPress core handbook: understanding and raising the PHP memory limit for large operations. developer.wordpress.org
- WooCommerce REST API docs: the subscriptions and orders endpoints, including default and maximum per_page paging. woocommerce.github.io/woocommerce-rest-api-docs
- WooCommerce developer docs: High Performance Order Storage (HPOS) and how order and subscription data is read through the same REST API either way. developer.woocommerce.com/docs/hpos-extension-recipe-book
On the solution:
- Python docs: the csv module and writing rows incrementally to a file handle instead of building the file in memory. docs.python.org/3/library/csv.html
- Node.js docs: fs.createWriteStream, for writing output incrementally instead of building a string in memory. nodejs.org/api/fs.html
- WooCommerce REST API docs: listing subscriptions with per_page, page, orderby, and order parameters. woocommerce.github.io/woocommerce-rest-api-docs
Stuck on a tricky one?
If you have a bug in WooCommerce, WooCommerce Subscriptions, or the WooCommerce Stripe gateway that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this fix your export?
If this saved you a crashed export or a support ticket about a missing file, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi