Diagnostic Catalog and Visibility
Product images duplicated on repeated import or product duplication
Re-run the same CSV import, or click Save and Duplicate on a product, and the exact same picture shows up again in the gallery under a new name, image_1.jpg, image_2.jpg, and so on. Do it a few more times and one product ends up with a dozen entries that are all the same photo. Here is why Magento never checks whether the image is already attached before writing another gallery row, and a script that finds the true byte-identical duplicates and removes only those, safely.
Magento's catalog importer (Magento\CatalogImportExport\Model\Import\Product) and the product duplicate feature (Magento\Catalog\Model\Product\Copier::copy) both append to catalog_product_entity_media_gallery instead of checking whether an identical image is already attached to that SKU. On re-import the importer copies the file from pub/media/import into pub/media/catalog/product again, finds a file of that name already there, so it saves a disambiguated copy such as image_1.jpg and inserts a new gallery row for it. The Copier does the same thing, re-persisting every source entry against the duplicated product with no dedupe check. Run a small Python or Node.js script that reads each SKU's media_gallery_entries from GET /rest/V1/products/{sku}, hashes the bytes each entry's file resolves to, and groups entries by that hash. Anything that shares a hash with another entry on the same SKU is a true duplicate. Full code, tests, and a dry run guard are below.
The problem in plain words
Uploading an image to a product through the Admin works fine the first time. Magento saves the file under pub/media/catalog/product and writes one row into catalog_product_entity_media_gallery, with a value row in catalog_product_entity_media_gallery_value linking it to the product and its label, position, and disabled flag.
The trouble starts the second time the same picture shows up for the same SKU, whether that is a repeated CSV import through Magento\CatalogImportExport\Model\Import\Product or a click on Save and Duplicate through Magento\Catalog\Model\Product\Copier::copy. Neither of these paths asks "is this exact image already attached to this product." The importer just tries to copy the source file from pub/media/import into pub/media/catalog/product again, sees a file of that name is already sitting there, and saves the new copy under a disambiguated name instead, image_1.jpg, then image_2.jpg on the next run. Each renamed copy gets its own new gallery row. The Copier has the same gap, it iterates the source product's media_gallery_entries and re-persists every one of them against the new product with no check for a match, and in some versions a save loop during duplication re-triggers the copy again, multiplying the count further.
Why it happens
Magento\CatalogImportExport\Model\Import\Productcopies the source image frompub/media/importintopub/media/catalog/producton every run. When a file of that destination name already exists, it saves the incoming file under a disambiguated name instead of recognizing the picture is already there for that SKU.- Each disambiguated file gets its own fresh row in
catalog_product_entity_media_galleryandcatalog_product_entity_media_gallery_value, so the row count grows with every re-import, not just the file count. Magento\Catalog\Model\Product\Copier::copyiterates the source product'smedia_gallery_entriesand re-persists every entry against the new product, with no check for whether an identical image is already attached to the duplicate.- In some versions a save loop that fires during duplication re-triggers the copy logic, multiplying the count again, sometimes into the hundreds for a single Save and Duplicate click on a product that already had a handful of images.
This is a well documented gap in core Magento, reported both as CSV re-import piling up renamed copies of the same image, and as Save and Duplicate multiplying a product's images by a couple hundred in one click. See the citations at the end for the exact issue threads.
A merchant can legitimately have several visually similar photos on one product, so it is never safe to assume "same file size" or "similar name" means duplicate. The only trustworthy signal is the actual bytes of the image. Two gallery entries under the same SKU whose file content hashes to the same value are the exact same picture, full stop, regardless of what each one is named. Everything else, filename patterns, size alone, visual similarity, is a hint at best and stays a report, never an automatic delete.
The fix, as a flow
The script never touches the importer or the Copier, it works entirely after the fact through REST. For each SKU it wanted to check, it reads media_gallery_entries, downloads the bytes each entry's file resolves to under {MAGENTO_URL}/media/catalog/product{file}, hashes them, and groups entries by that hash. Any group larger than one is a confirmed duplicate. By default the script only reports. With repair explicitly enabled it removes the extra entries, always keeping the lowest id, the first one imported, as canonical.
Build it step by step
Get an admin token and pick your SKUs
Get an admin token by calling POST {MAGENTO_URL}/rest/V1/integration/admin/token with your admin username and password, or use a long lived integration token. Keep the base URL and token in environment variables, and either name a list of SKUs to check or let the script page through GET /rest/V1/products.
pip install requests
export MAGENTO_URL="https://yourstore.example.com"
export MAGENTO_ADMIN_TOKEN="eyJraWQ..."
export SKUS="MB01-BLUE,MB01-RED,MB02-BLACK"
export DRY_RUN="true" # start safe, change to false to remove confirmed duplicates// Node 18+ has fetch built in, no dependencies needed
export MAGENTO_URL="https://yourstore.example.com"
export MAGENTO_ADMIN_TOKEN="eyJraWQ..."
export SKUS="MB01-BLUE,MB01-RED,MB02-BLACK"
export DRY_RUN="true" // start safe, change to false to remove confirmed duplicatesTalk to the Magento REST API
Every call sends the admin token as a bearer header. A small helper wraps GET and PUT requests, raises on a bad status code, and returns the parsed JSON body so the rest of the script only deals with plain data.
import os, requests
MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
HEADERS = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}
def api_get(path, params=None):
r = requests.get(f"{MAGENTO_URL}/rest/V1{path}", headers=HEADERS, params=params or {}, timeout=30)
r.raise_for_status()
return r.json()
def api_put(path, payload):
r = requests.put(f"{MAGENTO_URL}/rest/V1{path}", headers=HEADERS, json=payload, timeout=30)
r.raise_for_status()
return r.json()const MAGENTO_URL = (process.env.MAGENTO_URL || "").replace(/\/$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN;
const HEADERS = { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" };
async function apiGet(path, params = {}) {
const url = new URL(`${MAGENTO_URL}/rest/V1${path}`);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: HEADERS });
if (!res.ok) throw new Error(`Magento ${res.status}`);
return res.json();
}
async function apiPut(path, payload) {
const res = await fetch(`${MAGENTO_URL}/rest/V1${path}`, {
method: "PUT",
headers: HEADERS,
body: JSON.stringify(payload),
});
if (!res.ok) throw new Error(`Magento ${res.status}`);
return res.json();
}Read the gallery entries and hash the bytes
Call GET /V1/products/{'{'}sku{'}'} and read the media_gallery_entries array off the response, each with id, file, media_type, label, position, disabled, and types. For each entry, fetch the bytes at {'{'}MAGENTO_URL{'}'}/media/catalog/product{'{'}file{'}'} and hash them with MD5 so entries can be compared by content, not by name.
import hashlib
def fetch_gallery_entries(sku):
product = api_get(f"/products/{sku}")
return product.get("media_gallery_entries", [])
def hash_media_file(file_path):
url = f"{MAGENTO_URL}/media/catalog/product{file_path}"
r = requests.get(url, timeout=30)
r.raise_for_status()
return hashlib.md5(r.content).hexdigest()
def entries_with_hash(sku):
entries = fetch_gallery_entries(sku)
for entry in entries:
entry["hash"] = hash_media_file(entry["file"])
entry["size"] = None
return entriesimport { createHash } from "node:crypto";
async function fetchGalleryEntries(sku) {
const product = await apiGet(`/products/${sku}`);
return product.media_gallery_entries || [];
}
async function hashMediaFile(filePath) {
const url = `${MAGENTO_URL}/media/catalog/product${filePath}`;
const res = await fetch(url);
if (!res.ok) throw new Error(`Magento media ${res.status}`);
const buf = Buffer.from(await res.arrayBuffer());
return createHash("md5").update(buf).digest("hex");
}
async function entriesWithHash(sku) {
const entries = await fetchGalleryEntries(sku);
for (const entry of entries) {
entry.hash = await hashMediaFile(entry.file);
entry.size = null;
}
return entries;
}Decide, with one pure function
Keep the decision in its own function so it needs no network to test. It groups the pre-hashed entries by content hash, falling back to a normalized filename stem when a hash is missing, and within each group keeps the lowest id as canonical, the first one imported. Everything else in a group of more than one is reported as a duplicate candidate.
import re
SUFFIX_RE = re.compile(r"^(.*?)(_\d+)?(\.[A-Za-z0-9]+)$")
def normalized_stem(file_name):
base = file_name.rsplit("/", 1)[-1]
m = SUFFIX_RE.match(base)
if not m:
return base
return f"{m.group(1)}{m.group(3)}"
def group_key(entry):
if entry.get("hash"):
return f"hash:{entry['hash']}"
return f"name:{normalized_stem(entry['file'])}"
def find_duplicate_gallery_entries(media_gallery_entries):
groups = {}
for entry in media_gallery_entries:
groups.setdefault(group_key(entry), []).append(entry)
results = []
for key, group in groups.items():
if len(group) < 2:
continue
ids_sorted = sorted(e["id"] for e in group)
keep_id = ids_sorted[0]
duplicate_ids = ids_sorted[1:]
reason = "identical file content" if key.startswith("hash:") else "identical normalized filename"
results.append({"keepId": keep_id, "duplicateIds": duplicate_ids, "reason": reason})
return resultsconst SUFFIX_RE = /^(.*?)(_\d+)?(\.[A-Za-z0-9]+)$/;
export function normalizedStem(fileName) {
const base = fileName.split("/").pop();
const m = SUFFIX_RE.exec(base);
if (!m) return base;
return `${m[1]}${m[3]}`;
}
function groupKey(entry) {
if (entry.hash) return `hash:${entry.hash}`;
return `name:${normalizedStem(entry.file)}`;
}
export function findDuplicateGalleryEntries(mediaGalleryEntries) {
const groups = new Map();
for (const entry of mediaGalleryEntries) {
const key = groupKey(entry);
if (!groups.has(key)) groups.set(key, []);
groups.get(key).push(entry);
}
const results = [];
for (const [key, group] of groups) {
if (group.length < 2) continue;
const idsSorted = group.map((e) => e.id).sort((a, b) => a - b);
const keepId = idsSorted[0];
const duplicateIds = idsSorted.slice(1);
const reason = key.startsWith("hash:") ? "identical file content" : "identical normalized filename";
results.push({ keepId, duplicateIds, reason });
}
return results;
}Guard the role fields before removing anything
Before dropping a duplicate id, check it is not the product's only image and is not the sole entry covering the base, small_image, or thumbnail role through its types array, unless a surviving sibling in the same duplicate group can take over that role. Skip the removal and log it if there is no safe sibling, rather than leave a product with no image in a required role.
ROLE_TYPES = {"base", "small_image", "thumbnail"}
def safe_duplicate_ids(all_entries, duplicate_group):
if len(all_entries) <= 1:
return []
by_id = {e["id"]: e for e in all_entries}
keep_id = duplicate_group["keepId"]
safe = []
for dup_id in duplicate_group["duplicateIds"]:
entry = by_id.get(dup_id)
if not entry:
continue
roles = set(entry.get("types") or []) & ROLE_TYPES
if roles:
keep_entry = by_id.get(keep_id, {})
keep_roles = set(keep_entry.get("types") or [])
if not roles.issubset(keep_roles):
continue
safe.append(dup_id)
return safeconst ROLE_TYPES = new Set(["base", "small_image", "thumbnail"]);
export function safeDuplicateIds(allEntries, duplicateGroup) {
if (allEntries.length <= 1) return [];
const byId = new Map(allEntries.map((e) => [e.id, e]));
const keepEntry = byId.get(duplicateGroup.keepId) || {};
const keepRoles = new Set(keepEntry.types || []);
const safe = [];
for (const dupId of duplicateGroup.duplicateIds) {
const entry = byId.get(dupId);
if (!entry) continue;
const roles = (entry.types || []).filter((t) => ROLE_TYPES.has(t));
if (roles.length && !roles.every((r) => keepRoles.has(r))) continue;
safe.push(dupId);
}
return safe;
}Log, then remove only when repair is enabled
For every SKU, log the SKU, entry id, and file for each duplicate found before writing anything. When DRY_RUN is true, that is all the script does. When repair is explicitly enabled, it removes the confirmed and safe duplicate ids by PUTting the product with its media_gallery_entries rewritten to omit them, since older Magento versions have no single-entry delete by id other than resending the full array.
Always start with DRY_RUN=true. A duplicate is only ever confirmed by a content hash match within the same SKU, never by filename pattern or size alone, and the script never removes a product's only image or a role image with no safe sibling to take over. Read the log before you flip repair on.
The full code
Here is the complete script in one file for each language. It reads settings from the environment, fetches and hashes gallery entries per SKU, finds true duplicates with the pure function, logs every removal candidate, and only writes when repair is explicitly enabled.
"""Find and safely remove duplicate Magento product gallery images caused by
repeated import or Save and Duplicate.
Magento's catalog importer (Magento\\CatalogImportExport\\Model\\Import\\Product)
and the product Copier (Magento\\Catalog\\Model\\Product\\Copier::copy) both
append to catalog_product_entity_media_gallery instead of checking whether an
identical image is already attached to the SKU. Re-running an import, or
duplicating a product, saves a renamed copy of the same file (image_1.jpg,
image_2.jpg, ...) and inserts a fresh gallery row for it every time.
This script reads media_gallery_entries per SKU over REST, hashes the bytes
each entry's file resolves to, and groups entries by that hash. Only entries
that share a hash within the same SKU are treated as true duplicates. It
reports by default. Repair only runs with DRY_RUN=false, only removes ids
confirmed as byte-identical duplicates, always keeps the lowest id (first
imported), and never removes a product's only image or an unmatched
base/small_image/thumbnail role.
"""
import os
import re
import hashlib
import logging
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("find_duplicate_gallery_entries")
MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
HEADERS = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}
SKUS = [s.strip() for s in os.environ.get("SKUS", "").split(",") if s.strip()]
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
ROLE_TYPES = {"base", "small_image", "thumbnail"}
SUFFIX_RE = re.compile(r"^(.*?)(_\d+)?(\.[A-Za-z0-9]+)$")
def api_get(path, params=None):
r = requests.get(f"{MAGENTO_URL}/rest/V1{path}", headers=HEADERS, params=params or {}, timeout=30)
r.raise_for_status()
return r.json()
def api_put(path, payload):
r = requests.put(f"{MAGENTO_URL}/rest/V1{path}", headers=HEADERS, json=payload, timeout=30)
r.raise_for_status()
return r.json()
def fetch_product(sku):
return api_get(f"/products/{sku}")
def hash_media_file(file_path):
url = f"{MAGENTO_URL}/media/catalog/product{file_path}"
r = requests.get(url, timeout=30)
r.raise_for_status()
return hashlib.md5(r.content).hexdigest()
def entries_with_hash(product):
entries = product.get("media_gallery_entries", [])
for entry in entries:
entry["hash"] = hash_media_file(entry["file"])
return entries
def normalized_stem(file_name):
base = file_name.rsplit("/", 1)[-1]
m = SUFFIX_RE.match(base)
if not m:
return base
return f"{m.group(1)}{m.group(3)}"
def group_key(entry):
if entry.get("hash"):
return f"hash:{entry['hash']}"
return f"name:{normalized_stem(entry['file'])}"
def find_duplicate_gallery_entries(media_gallery_entries):
"""Pure function. Groups entries by content hash (preferred) falling back
to normalized base filename when a hash is unavailable. Keeps the lowest
id per group as canonical, reports the rest. No I/O."""
groups = {}
for entry in media_gallery_entries:
groups.setdefault(group_key(entry), []).append(entry)
results = []
for key, group in groups.items():
if len(group) < 2:
continue
ids_sorted = sorted(e["id"] for e in group)
keep_id = ids_sorted[0]
duplicate_ids = ids_sorted[1:]
reason = "identical file content" if key.startswith("hash:") else "identical normalized filename"
results.append({"keepId": keep_id, "duplicateIds": duplicate_ids, "reason": reason})
return results
def safe_duplicate_ids(all_entries, duplicate_group):
if len(all_entries) <= 1:
return []
by_id = {e["id"]: e for e in all_entries}
keep_entry = by_id.get(duplicate_group["keepId"], {})
keep_roles = set(keep_entry.get("types") or [])
safe = []
for dup_id in duplicate_group["duplicateIds"]:
entry = by_id.get(dup_id)
if not entry:
continue
roles = set(entry.get("types") or []) & ROLE_TYPES
if roles and not roles.issubset(keep_roles):
continue
safe.append(dup_id)
return safe
def remove_entries(sku, product, remove_ids):
remaining = [e for e in product["media_gallery_entries"] if e["id"] not in remove_ids]
payload = {"product": {"sku": sku, "media_gallery_entries": remaining}}
return api_put(f"/products/{sku}", payload)
def run():
total_removed = 0
for sku in SKUS:
product = fetch_product(sku)
entries = entries_with_hash(product)
groups = find_duplicate_gallery_entries(entries)
if not groups:
log.info("SKU %s: no duplicate gallery entries found.", sku)
continue
remove_ids = set()
for group in groups:
safe_ids = safe_duplicate_ids(entries, group)
for entry in entries:
if entry["id"] in group["duplicateIds"]:
log.warning(
"SKU %s: entry id=%s file=%s is a duplicate of id=%s (%s)%s",
sku, entry["id"], entry["file"], group["keepId"], group["reason"],
"" if entry["id"] in safe_ids else " -- skipped, no safe sibling for its role",
)
remove_ids.update(safe_ids)
if remove_ids:
log.info("SKU %s: %s %d entr%s.", sku,
"would remove" if DRY_RUN else "removing",
len(remove_ids), "y" if len(remove_ids) == 1 else "ies")
if not DRY_RUN:
remove_entries(sku, product, remove_ids)
total_removed += len(remove_ids)
log.info("Done. %d duplicate entr%s %s across %d SKU(s).",
total_removed, "y" if total_removed == 1 else "ies",
"to remove" if DRY_RUN else "removed", len(SKUS))
if __name__ == "__main__":
run()/**
* Find and safely remove duplicate Magento product gallery images caused by
* repeated import or Save and Duplicate.
*
* Magento's catalog importer (Magento\CatalogImportExport\Model\Import\Product)
* and the product Copier (Magento\Catalog\Model\Product\Copier::copy) both
* append to catalog_product_entity_media_gallery instead of checking whether
* an identical image is already attached to the SKU. Re-running an import, or
* duplicating a product, saves a renamed copy of the same file (image_1.jpg,
* image_2.jpg, ...) and inserts a fresh gallery row for it every time.
*
* This script reads media_gallery_entries per SKU over REST, hashes the bytes
* each entry's file resolves to, and groups entries by that hash. Only
* entries that share a hash within the same SKU are treated as true
* duplicates. It reports by default. Repair only runs with DRY_RUN=false,
* only removes ids confirmed as byte-identical duplicates, always keeps the
* lowest id (first imported), and never removes a product's only image or an
* unmatched base/small_image/thumbnail role.
*
* Guide: https://www.allanninal.dev/magento/product-images-duplicated-on-import/
*/
import { pathToFileURL } from "node:url";
import { createHash } from "node:crypto";
const MAGENTO_URL = (process.env.MAGENTO_URL || "https://example.test").replace(/\/$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN || "dummy-token";
const HEADERS = { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" };
const SKUS = (process.env.SKUS || "")
.split(",")
.map((s) => s.trim())
.filter(Boolean);
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const ROLE_TYPES = new Set(["base", "small_image", "thumbnail"]);
const SUFFIX_RE = /^(.*?)(_\d+)?(\.[A-Za-z0-9]+)$/;
export function normalizedStem(fileName) {
const base = fileName.split("/").pop();
const m = SUFFIX_RE.exec(base);
if (!m) return base;
return `${m[1]}${m[3]}`;
}
function groupKey(entry) {
if (entry.hash) return `hash:${entry.hash}`;
return `name:${normalizedStem(entry.file)}`;
}
/**
* Pure function. Groups entries by content hash (preferred) falling back to
* normalized base filename when a hash is unavailable. Keeps the lowest id
* per group as canonical, reports the rest. No I/O.
*/
export function findDuplicateGalleryEntries(mediaGalleryEntries) {
const groups = new Map();
for (const entry of mediaGalleryEntries) {
const key = groupKey(entry);
if (!groups.has(key)) groups.set(key, []);
groups.get(key).push(entry);
}
const results = [];
for (const [key, group] of groups) {
if (group.length < 2) continue;
const idsSorted = group.map((e) => e.id).sort((a, b) => a - b);
const keepId = idsSorted[0];
const duplicateIds = idsSorted.slice(1);
const reason = key.startsWith("hash:") ? "identical file content" : "identical normalized filename";
results.push({ keepId, duplicateIds, reason });
}
return results;
}
export function safeDuplicateIds(allEntries, duplicateGroup) {
if (allEntries.length <= 1) return [];
const byId = new Map(allEntries.map((e) => [e.id, e]));
const keepEntry = byId.get(duplicateGroup.keepId) || {};
const keepRoles = new Set(keepEntry.types || []);
const safe = [];
for (const dupId of duplicateGroup.duplicateIds) {
const entry = byId.get(dupId);
if (!entry) continue;
const roles = (entry.types || []).filter((t) => ROLE_TYPES.has(t));
if (roles.length && !roles.every((r) => keepRoles.has(r))) continue;
safe.push(dupId);
}
return safe;
}
async function apiGet(path, params = {}) {
const url = new URL(`${MAGENTO_URL}/rest/V1${path}`);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: HEADERS });
if (!res.ok) throw new Error(`Magento ${res.status}`);
return res.json();
}
async function apiPut(path, payload) {
const res = await fetch(`${MAGENTO_URL}/rest/V1${path}`, {
method: "PUT",
headers: HEADERS,
body: JSON.stringify(payload),
});
if (!res.ok) throw new Error(`Magento ${res.status}`);
return res.json();
}
async function fetchProduct(sku) {
return apiGet(`/products/${sku}`);
}
async function hashMediaFile(filePath) {
const url = `${MAGENTO_URL}/media/catalog/product${filePath}`;
const res = await fetch(url);
if (!res.ok) throw new Error(`Magento media ${res.status}`);
const buf = Buffer.from(await res.arrayBuffer());
return createHash("md5").update(buf).digest("hex");
}
async function entriesWithHash(product) {
const entries = product.media_gallery_entries || [];
for (const entry of entries) {
entry.hash = await hashMediaFile(entry.file);
}
return entries;
}
async function removeEntries(sku, product, removeIds) {
const remaining = product.media_gallery_entries.filter((e) => !removeIds.has(e.id));
const payload = { product: { sku, media_gallery_entries: remaining } };
return apiPut(`/products/${sku}`, payload);
}
export async function run() {
let totalRemoved = 0;
for (const sku of SKUS) {
const product = await fetchProduct(sku);
const entries = await entriesWithHash(product);
const groups = findDuplicateGalleryEntries(entries);
if (!groups.length) {
console.log(`SKU ${sku}: no duplicate gallery entries found.`);
continue;
}
const removeIds = new Set();
for (const group of groups) {
const safeIds = new Set(safeDuplicateIds(entries, group));
for (const entry of entries) {
if (group.duplicateIds.includes(entry.id)) {
const skipped = safeIds.has(entry.id) ? "" : " -- skipped, no safe sibling for its role";
console.warn(
`SKU ${sku}: entry id=${entry.id} file=${entry.file} is a duplicate of id=${group.keepId} (${group.reason})${skipped}`
);
}
}
for (const id of safeIds) removeIds.add(id);
}
if (removeIds.size) {
console.log(`SKU ${sku}: ${DRY_RUN ? "would remove" : "removing"} ${removeIds.size} entr${removeIds.size === 1 ? "y" : "ies"}.`);
if (!DRY_RUN) await removeEntries(sku, product, removeIds);
}
totalRemoved += removeIds.size;
}
console.log(
`Done. ${totalRemoved} duplicate entr${totalRemoved === 1 ? "y" : "ies"} ${DRY_RUN ? "to remove" : "removed"} across ${SKUS.length} SKU(s).`
);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
findDuplicateGalleryEntries is the part worth testing, because it decides which entries are safe to report, and combined with safeDuplicateIds it decides which ones are safe to remove. Both are pure functions over plain arrays and objects, so the tests need no network and no Magento store.
from find_duplicate_gallery_entries import find_duplicate_gallery_entries, safe_duplicate_ids
def entry(id, file, hash=None, types=None):
return {"id": id, "file": file, "hash": hash, "types": types or []}
def test_no_duplicates_when_all_hashes_differ():
entries = [entry(1, "/m/b/a.jpg", "h1"), entry(2, "/m/b/b.jpg", "h2")]
assert find_duplicate_gallery_entries(entries) == []
def test_finds_duplicate_by_hash_keeps_lowest_id():
entries = [entry(3, "/m/b/a_2.jpg", "h1"), entry(1, "/m/b/a.jpg", "h1"), entry(2, "/m/b/a_1.jpg", "h1")]
result = find_duplicate_gallery_entries(entries)
assert len(result) == 1
assert result[0]["keepId"] == 1
assert result[0]["duplicateIds"] == [2, 3]
assert result[0]["reason"] == "identical file content"
def test_falls_back_to_normalized_filename_without_hash():
entries = [entry(1, "/m/b/photo.jpg"), entry(2, "/m/b/photo_1.jpg")]
result = find_duplicate_gallery_entries(entries)
assert len(result) == 1
assert result[0]["reason"] == "identical normalized filename"
def test_different_pictures_are_not_grouped():
entries = [entry(1, "/m/b/front.jpg", "h1"), entry(2, "/m/b/back.jpg", "h2"), entry(3, "/m/b/side.jpg", "h3")]
assert find_duplicate_gallery_entries(entries) == []
def test_safe_duplicate_ids_allows_removal_when_no_role():
entries = [entry(1, "/m/b/a.jpg", "h1"), entry(2, "/m/b/a_1.jpg", "h1")]
group = {"keepId": 1, "duplicateIds": [2], "reason": "identical file content"}
assert safe_duplicate_ids(entries, group) == [2]
def test_safe_duplicate_ids_blocks_when_role_not_covered_by_keeper():
entries = [
entry(1, "/m/b/a.jpg", "h1", types=[]),
entry(2, "/m/b/a_1.jpg", "h1", types=["base"]),
]
group = {"keepId": 1, "duplicateIds": [2], "reason": "identical file content"}
assert safe_duplicate_ids(entries, group) == []
def test_safe_duplicate_ids_allows_when_keeper_covers_role():
entries = [
entry(1, "/m/b/a.jpg", "h1", types=["base", "small_image"]),
entry(2, "/m/b/a_1.jpg", "h1", types=["base"]),
]
group = {"keepId": 1, "duplicateIds": [2], "reason": "identical file content"}
assert safe_duplicate_ids(entries, group) == [2]
def test_safe_duplicate_ids_never_removes_only_image():
entries = [entry(1, "/m/b/a.jpg", "h1")]
group = {"keepId": 1, "duplicateIds": [], "reason": "identical file content"}
assert safe_duplicate_ids(entries, group) == []import { test } from "node:test";
import assert from "node:assert/strict";
import { findDuplicateGalleryEntries, safeDuplicateIds, normalizedStem } from "./find-duplicate-gallery-entries.js";
const entry = (id, file, hash = null, types = []) => ({ id, file, hash, types });
test("no duplicates when all hashes differ", () => {
const entries = [entry(1, "/m/b/a.jpg", "h1"), entry(2, "/m/b/b.jpg", "h2")];
assert.deepEqual(findDuplicateGalleryEntries(entries), []);
});
test("finds duplicate by hash and keeps lowest id", () => {
const entries = [entry(3, "/m/b/a_2.jpg", "h1"), entry(1, "/m/b/a.jpg", "h1"), entry(2, "/m/b/a_1.jpg", "h1")];
const result = findDuplicateGalleryEntries(entries);
assert.equal(result.length, 1);
assert.equal(result[0].keepId, 1);
assert.deepEqual(result[0].duplicateIds, [2, 3]);
assert.equal(result[0].reason, "identical file content");
});
test("falls back to normalized filename without hash", () => {
const entries = [entry(1, "/m/b/photo.jpg"), entry(2, "/m/b/photo_1.jpg")];
const result = findDuplicateGalleryEntries(entries);
assert.equal(result.length, 1);
assert.equal(result[0].reason, "identical normalized filename");
});
test("different pictures are not grouped", () => {
const entries = [entry(1, "/m/b/front.jpg", "h1"), entry(2, "/m/b/back.jpg", "h2"), entry(3, "/m/b/side.jpg", "h3")];
assert.deepEqual(findDuplicateGalleryEntries(entries), []);
});
test("safeDuplicateIds allows removal when no role", () => {
const entries = [entry(1, "/m/b/a.jpg", "h1"), entry(2, "/m/b/a_1.jpg", "h1")];
const group = { keepId: 1, duplicateIds: [2], reason: "identical file content" };
assert.deepEqual(safeDuplicateIds(entries, group), [2]);
});
test("safeDuplicateIds blocks when role not covered by keeper", () => {
const entries = [entry(1, "/m/b/a.jpg", "h1", []), entry(2, "/m/b/a_1.jpg", "h1", ["base"])];
const group = { keepId: 1, duplicateIds: [2], reason: "identical file content" };
assert.deepEqual(safeDuplicateIds(entries, group), []);
});
test("safeDuplicateIds allows when keeper covers role", () => {
const entries = [entry(1, "/m/b/a.jpg", "h1", ["base", "small_image"]), entry(2, "/m/b/a_1.jpg", "h1", ["base"])];
const group = { keepId: 1, duplicateIds: [2], reason: "identical file content" };
assert.deepEqual(safeDuplicateIds(entries, group), [2]);
});
test("safeDuplicateIds never removes only image", () => {
const entries = [entry(1, "/m/b/a.jpg", "h1")];
const group = { keepId: 1, duplicateIds: [], reason: "identical file content" };
assert.deepEqual(safeDuplicateIds(entries, group), []);
});
test("normalizedStem strips Magento disambiguation suffix", () => {
assert.equal(normalizedStem("/m/b/photo_12.jpg"), "photo.jpg");
assert.equal(normalizedStem("/m/b/photo.jpg"), "photo.jpg");
});Case studies
A supplier feed re-imported the same product images weekly
A distributor synced its catalog with a weekly CSV import that always included the full product image set, even for SKUs that had not changed. Every run copied the same photos from pub/media/import again, and because a file of that name already existed under pub/media/catalog/product, the importer saved renamed copies and inserted new gallery rows. After a year, some SKUs carried thirty or more entries pointing at the same handful of pictures.
Running the script against the affected SKUs confirmed by hash that nearly all of it was true duplication, not new photography. With repair enabled it dropped back to one entry per real picture per SKU, and the weekly import stopped growing the gallery table any further.
A single click multiplied a product's images by two hundred
A merchandiser used Save and Duplicate to spin up color variants from a base product with a handful of gallery images. On one Magento version a save loop during duplication re-triggered the Copier's image copy logic repeatedly, and the new product ended up with several hundred gallery entries, all pointing at the same handful of images renamed with steadily higher suffixes.
The script's hash grouping collapsed the whole mess to the handful of pictures that were actually unique, confirmed byte for byte, and repair removed the rest while keeping the base and small_image roles intact. The product page went from painfully slow to load back to normal.
After a run, each SKU's gallery has exactly one entry per genuinely unique picture, confirmed by content hash rather than guesswork about filenames. Nothing that might be a legitimately similar but different photo gets touched, and no product loses its base, small_image, or thumbnail image. The importer and Copier keep working exactly as before, this script only cleans up after them on a schedule you control.
FAQ
Why do the same product images keep piling up after every import?
Magento's catalog importer copies the source file from pub/media/import into pub/media/catalog/product on every run. It only checks whether a file with that name already exists, not whether the same picture is already attached to that SKU, so it saves the file under a new name like image_1.jpg and inserts a fresh gallery row pointing at it. Run the import twice and you get two rows and two files for one picture.
Does Save and Duplicate have the same problem as CSV import?
Yes. Magento\Catalog\Model\Product\Copier::copy iterates the source product's media_gallery_entries and re-persists every entry against the new product without checking whether an identical image is already attached. In some versions a save loop during duplication also re-triggers the copy, which multiplies the count further, sometimes into the hundreds for a single duplicated product.
Is it safe to auto-delete duplicate gallery entries?
Only when the duplicate check is based on the actual file content, not the filename. Two entries with different names can be genuinely different pictures that just happen to be similar. The safe rule is to hash the bytes of each image and only treat entries under the same SKU with an identical hash as true duplicates, always keeping the lowest id, the first one imported, and never removing an entry if no sibling duplicate remains to cover its role as base, small_image, or thumbnail.
Related field notes
Citations
On the problem:
- Magento 2 GitHub issue: product images are being duplicated on import. github.com/magento/magento2/issues/21885
- Magento 2 GitHub issue: duplicating product copies product images a couple hundred times. github.com/magento/magento2/issues/9466
- Magento 2 GitHub issue: bug in product import with additional images. github.com/magento/magento2/issues/7826
On the solution:
- Adobe Commerce (Experience League): product image import. experienceleague.adobe.com product image import
- Adobe Commerce Web APIs: REST API overview. developer.adobe.com/commerce/webapi/rest
- Adobe Commerce Web APIs: getting started. developer.adobe.com/commerce/webapi/get-started
Stuck on a tricky one?
If you have a problem in Magento catalog, imports, media, or product data that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this clean up a bloated gallery?
If this saved you from a product page dragging under hundreds of duplicate images, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi