Field notes / Orders and payments

Diagnostic Orders and payments

Shopify high-risk orders that ship without review

Shopify quietly scores every order for fraud risk. But scoring is not stopping. A high-risk order sits in the same queue as everything else, and if nobody notices the little risk flag, it gets picked, packed, and shipped. Weeks later it comes back as a chargeback and you lose the goods and the money. Here is why these orders slip through and a small job that tags them for review before they ship.

Python and Node.js Admin GraphQL API Safe by default (dry run)

Scrabble tiles spelling out the names of different languages — Photo by Markus Winkler on Unsplash

The short answer

Shopify rates orders for fraud risk but does not hold them, so a high-risk order ships if nobody looks. Run a small Python or Node.js job that lists open orders, reads each order's risk, keeps the ones rated high risk or recommended to cancel that are not already tagged, and adds a review tag with tagsAdd. Staff then see every risky order in one filtered view before fulfillment. Full code, tests, and a dry run guard are below.

The problem in plain words

When an order comes in, Shopify looks at signals like the card, the address, and the buyer's history, and gives the order a risk level. That is helpful, but it is only information. Shopify does not pause a high-risk order or block its fulfillment. It just marks it.

On a busy store, that mark is easy to miss. Orders flow to the packing team, and unless someone opens each order and notices the risk, the risky ones ship alongside the good ones. The cost shows up later as a chargeback, when the real cardholder disputes a payment they never made.

Shopify marks the risk but does not stop the order. Without a review step, the risky order ships and returns as a chargeback.

Why it happens

The risk score is designed to inform a human, not to act on its own. A few reasons risky orders slip through:

The store has no habit of filtering for risk before fulfillment, so the flag is never seen.
Fulfillment is automated or handed to a third party that only sees the picking list, not the risk level.
A rush of orders buries the risky ones, and staff process by date, not by risk.
The person who used to check risk left, and nobody took over the daily review.

Shopify is clear that its fraud analysis is a guide and the final decision is yours. So the fix is not to make the score smarter, it is to make sure a person actually sees every high-risk order. See the citations at the end for the exact docs.

The key insight

A risk score buried inside each order is a score nobody reads. Turn it into something visible: a tag. Once every high-risk order carries a fraud-review tag, staff can open one saved filter and see the whole list, and no risky order ships without a set of eyes on it first.

The fix, as a flow

We do not cancel or refund anything. We add a job that reads the risk on each open order and tags the high-risk ones for review. Tagging is safe and reversible, and it leaves the real decision, ship or cancel, to a person looking at a clean list.

The job tags only high-risk orders that are open and not already tagged. It never cancels or refunds. A person makes that call.

Build it step by step

Get an Admin API access token

Create a custom app in your Shopify admin under Settings, Apps and sales channels, Develop apps. Give it the read_orders and write_orders scopes and install it to get an Admin API access token that starts with shpat_. Keep the token and the shop domain in environment variables, never in the file.

setup (shell)

pip install requests

export SHOPIFY_SHOP="yourstore.myshopify.com"
export SHOPIFY_ACCESS_TOKEN="shpat_..."
export REVIEW_TAG="fraud-review"
export DRY_RUN="true"   # start safe, change to false to tag

setup (shell)

// Node 18+ has fetch built in, no dependencies needed

export SHOPIFY_SHOP="yourstore.myshopify.com"
export SHOPIFY_ACCESS_TOKEN="shpat_..."
export REVIEW_TAG="fraud-review"
export DRY_RUN="true"   // start safe, change to false to tag

List open orders with their risk

Ask for orders that are unfulfilled and not cancelled, and read back the tags plus the order risk, which has a recommendation and a list of assessments each with a risk level. Reading the risk over the API means you never have to open each order by hand. We page through with a cursor so the job handles a backlog.

step2.py

ORDERS_QUERY = """
query($cursor: String) {
  orders(first: 25, after: $cursor,
         query: "fulfillment_status:unfulfilled AND -status:cancelled") {
    pageInfo { hasNextPage endCursor }
    nodes {
      id name cancelledAt tags
      risk { recommendation assessments { riskLevel } }
    }
  }
}"""

def open_orders():
    cursor = None
    while True:
        data = gql(ORDERS_QUERY, {"cursor": cursor})["orders"]
        for node in data["nodes"]:
            yield node
        if not data["pageInfo"]["hasNextPage"]:
            return
        cursor = data["pageInfo"]["endCursor"]

step2.js

const ORDERS_QUERY = `
query($cursor: String) {
  orders(first: 25, after: $cursor,
         query: "fulfillment_status:unfulfilled AND -status:cancelled") {
    pageInfo { hasNextPage endCursor }
    nodes {
      id name cancelledAt tags
      risk { recommendation assessments { riskLevel } }
    }
  }
}`;

async function* openOrders() {
  let cursor = null;
  while (true) {
    const data = (await gql(ORDERS_QUERY, { cursor })).orders;
    for (const node of data.nodes) yield node;
    if (!data.pageInfo.hasNextPage) return;
    cursor = data.pageInfo.endCursor;
  }
}

Decide, with one pure function

Keep the rule in its own function. An order is high risk when its recommendation is to cancel, or when any assessment has a risk level of HIGH. We then only flag it if it is open, not cancelled, and not already tagged, so re-running the job never tags the same order twice. A pure function like this is easy to read and easy to test.

decide.py

def is_high_risk(order):
    risk = order.get("risk") or {}
    if risk.get("recommendation") == "CANCEL":
        return True
    return any(a.get("riskLevel") == "HIGH" for a in (risk.get("assessments") or []))

def needs_review(order, review_tag):
    if order.get("cancelledAt"):
        return False
    if review_tag in (order.get("tags") or []):
        return False
    return is_high_risk(order)

decide.js

export function isHighRisk(order) {
  const risk = order.risk || {};
  if (risk.recommendation === "CANCEL") return true;
  return (risk.assessments || []).some((a) => a.riskLevel === "HIGH");
}

export function needsReview(order, reviewTag) {
  if (order.cancelledAt) return false;
  if ((order.tags || []).includes(reviewTag)) return false;
  return isHighRisk(order);
}

Tag the order for review

When an order needs review, call the tagsAdd mutation with the order id and your review tag. That is the whole action. It does not cancel, hold, or refund anything, so it is safe to run often. Always read back userErrors and stop on them rather than assume the tag was added.

apply.py

TAGS_ADD = """
mutation($id: ID!, $tags: [String!]!) {
  tagsAdd(id: $id, tags: $tags) {
    node { id }
    userErrors { field message }
  }
}"""

def tag_for_review(order_id, review_tag):
    result = gql(TAGS_ADD, {"id": order_id, "tags": [review_tag]})["tagsAdd"]
    if result["userErrors"]:
        raise RuntimeError(result["userErrors"])

apply.js

const TAGS_ADD = `
mutation($id: ID!, $tags: [String!]!) {
  tagsAdd(id: $id, tags: $tags) {
    node { id }
    userErrors { field message }
  }
}`;

async function tagForReview(orderId, reviewTag) {
  const result = (await gql(TAGS_ADD, { id: orderId, tags: [reviewTag] })).tagsAdd;
  if (result.userErrors.length) throw new Error(JSON.stringify(result.userErrors));
}

Wire it together with a dry run guard

The loop ties every piece together. On the first few runs, leave DRY_RUN on so the job only reports which orders it would tag. Read the output, agree with it, then switch it off. Run it every few minutes so a risky order is tagged soon after it comes in, well before it reaches the packing table.

Run it safe

Start with DRY_RUN=true. This job only tags, it never cancels or refunds, so the worst it can do is add a tag. Keep the ship or cancel decision with a person. Set up a saved order view filtered by the review tag so staff can work the list.

The full code

Here is the complete job in one file for each language. It reads settings from the environment, logs what it does, respects the dry run flag, and is safe to run again and again because it skips orders that are already tagged.

View this code on GitHub Full runnable folder with tests in the shopify-fixes repo.

flag_high_risk.py

"""Flag high-risk Shopify orders that slipped through without review.
Run on a schedule. Safe to run again and again.
"""
import os
import logging
import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("flag_high_risk")

SHOP = os.environ["SHOPIFY_SHOP"]
TOKEN = os.environ["SHOPIFY_ACCESS_TOKEN"]
API_VERSION = os.environ.get("SHOPIFY_API_VERSION", "2025-01")
ENDPOINT = f"https://{SHOP}/admin/api/{API_VERSION}/graphql.json"
REVIEW_TAG = os.environ.get("REVIEW_TAG", "fraud-review")
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"

ORDERS_QUERY = """
query($cursor: String) {
  orders(first: 25, after: $cursor,
         query: "fulfillment_status:unfulfilled AND -status:cancelled") {
    pageInfo { hasNextPage endCursor }
    nodes {
      id name cancelledAt tags
      risk { recommendation assessments { riskLevel } }
    }
  }
}"""

TAGS_ADD = """
mutation($id: ID!, $tags: [String!]!) {
  tagsAdd(id: $id, tags: $tags) {
    node { id }
    userErrors { field message }
  }
}"""


def gql(query, variables=None):
    r = requests.post(
        ENDPOINT,
        json={"query": query, "variables": variables or {}},
        headers={"X-Shopify-Access-Token": TOKEN, "Content-Type": "application/json"},
        timeout=30,
    )
    r.raise_for_status()
    body = r.json()
    if body.get("errors"):
        raise RuntimeError(body["errors"])
    return body["data"]


def is_high_risk(order):
    risk = order.get("risk") or {}
    if risk.get("recommendation") == "CANCEL":
        return True
    return any(a.get("riskLevel") == "HIGH" for a in (risk.get("assessments") or []))


def needs_review(order, review_tag):
    if order.get("cancelledAt"):
        return False
    if review_tag in (order.get("tags") or []):
        return False
    return is_high_risk(order)


def tag_for_review(order_id, review_tag):
    result = gql(TAGS_ADD, {"id": order_id, "tags": [review_tag]})["tagsAdd"]
    if result["userErrors"]:
        raise RuntimeError(result["userErrors"])


def open_orders():
    cursor = None
    while True:
        data = gql(ORDERS_QUERY, {"cursor": cursor})["orders"]
        for node in data["nodes"]:
            yield node
        if not data["pageInfo"]["hasNextPage"]:
            return
        cursor = data["pageInfo"]["endCursor"]


def run():
    flagged = 0
    for order in open_orders():
        if not needs_review(order, REVIEW_TAG):
            continue
        log.warning("Order %s is high risk and unreviewed. %s",
                    order["name"], "would tag" if DRY_RUN else "tagging")
        if not DRY_RUN:
            tag_for_review(order["id"], REVIEW_TAG)
        flagged += 1
    log.info("Done. %d high-risk order(s) %s.", flagged, "to tag" if DRY_RUN else "tagged")


if __name__ == "__main__":
    run()

flag-high-risk.js

/**
 * Flag high-risk Shopify orders that slipped through without review.
 * Run on a schedule. Safe to run again and again.
 */
import { pathToFileURL } from "node:url";

const SHOP = process.env.SHOPIFY_SHOP || "example.myshopify.com";
const TOKEN = process.env.SHOPIFY_ACCESS_TOKEN || "shpat_dummy";
const API_VERSION = process.env.SHOPIFY_API_VERSION || "2025-01";
const ENDPOINT = `https://${SHOP}/admin/api/${API_VERSION}/graphql.json`;
const REVIEW_TAG = process.env.REVIEW_TAG || "fraud-review";
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";

export function isHighRisk(order) {
  const risk = order.risk || {};
  if (risk.recommendation === "CANCEL") return true;
  return (risk.assessments || []).some((a) => a.riskLevel === "HIGH");
}

export function needsReview(order, reviewTag) {
  if (order.cancelledAt) return false;
  if ((order.tags || []).includes(reviewTag)) return false;
  return isHighRisk(order);
}

async function gql(query, variables = {}) {
  const res = await fetch(ENDPOINT, {
    method: "POST",
    headers: { "X-Shopify-Access-Token": TOKEN, "Content-Type": "application/json" },
    body: JSON.stringify({ query, variables }),
  });
  if (!res.ok) throw new Error(`Shopify ${res.status}`);
  const body = await res.json();
  if (body.errors) throw new Error(JSON.stringify(body.errors));
  return body.data;
}

const ORDERS_QUERY = `
query($cursor: String) {
  orders(first: 25, after: $cursor,
         query: "fulfillment_status:unfulfilled AND -status:cancelled") {
    pageInfo { hasNextPage endCursor }
    nodes {
      id name cancelledAt tags
      risk { recommendation assessments { riskLevel } }
    }
  }
}`;

const TAGS_ADD = `
mutation($id: ID!, $tags: [String!]!) {
  tagsAdd(id: $id, tags: $tags) {
    node { id }
    userErrors { field message }
  }
}`;

async function* openOrders() {
  let cursor = null;
  while (true) {
    const data = (await gql(ORDERS_QUERY, { cursor })).orders;
    for (const node of data.nodes) yield node;
    if (!data.pageInfo.hasNextPage) return;
    cursor = data.pageInfo.endCursor;
  }
}

async function tagForReview(orderId, reviewTag) {
  const result = (await gql(TAGS_ADD, { id: orderId, tags: [reviewTag] })).tagsAdd;
  if (result.userErrors.length) throw new Error(JSON.stringify(result.userErrors));
}

export async function run() {
  let flagged = 0;
  for await (const order of openOrders()) {
    if (!needsReview(order, REVIEW_TAG)) continue;
    console.warn(`Order ${order.name} is high risk and unreviewed. ${DRY_RUN ? "would tag" : "tagging"}`);
    if (!DRY_RUN) await tagForReview(order.id, REVIEW_TAG);
    flagged++;
  }
  console.log(`Done. ${flagged} high-risk order(s) ${DRY_RUN ? "to tag" : "tagged"}.`);
}

if (import.meta.url === pathToFileURL(process.argv[1]).href) {
  run().catch((err) => { console.error(err); process.exit(1); });
}

Add a test

The decision rule is the part most worth testing, because it decides which orders get a review tag. Because we kept needs_review pure, the test needs no network and no Shopify account. It just feeds in plain orders and checks the answer.

test_high_risk_review.py

from flag_high_risk import needs_review, is_high_risk


def order(**over):
    base = {
        "cancelledAt": None,
        "tags": [],
        "risk": {"recommendation": "INVESTIGATE", "assessments": [{"riskLevel": "HIGH"}]},
    }
    base.update(over)
    return base


def test_high_risk_when_assessment_is_high():
    assert is_high_risk(order()) is True


def test_high_risk_when_recommendation_is_cancel():
    o = order(risk={"recommendation": "CANCEL", "assessments": [{"riskLevel": "LOW"}]})
    assert is_high_risk(o) is True


def test_not_high_risk_when_all_low():
    o = order(risk={"recommendation": "ACCEPT", "assessments": [{"riskLevel": "LOW"}]})
    assert is_high_risk(o) is False


def test_needs_review_true_for_untagged_high_risk():
    assert needs_review(order(), "fraud-review") is True


def test_needs_review_false_when_already_tagged():
    assert needs_review(order(tags=["fraud-review"]), "fraud-review") is False


def test_needs_review_false_when_cancelled():
    assert needs_review(order(cancelledAt="2026-07-10T00:00:00Z"), "fraud-review") is False

high-risk.test.js

import { test } from "node:test";
import assert from "node:assert/strict";
import { needsReview, isHighRisk } from "./flag-high-risk.js";

const order = (over = {}) => ({
  cancelledAt: null,
  tags: [],
  risk: { recommendation: "INVESTIGATE", assessments: [{ riskLevel: "HIGH" }] },
  ...over,
});

test("high risk when an assessment is HIGH", () => {
  assert.equal(isHighRisk(order()), true);
});

test("high risk when recommendation is CANCEL", () => {
  assert.equal(isHighRisk(order({ risk: { recommendation: "CANCEL", assessments: [{ riskLevel: "LOW" }] } })), true);
});

test("not high risk when all low", () => {
  assert.equal(isHighRisk(order({ risk: { recommendation: "ACCEPT", assessments: [{ riskLevel: "LOW" }] } })), false);
});

test("needsReview true for untagged high risk", () => {
  assert.equal(needsReview(order(), "fraud-review"), true);
});

test("needsReview false when already tagged", () => {
  assert.equal(needsReview(order({ tags: ["fraud-review"] }), "fraud-review"), false);
});

test("needsReview false when cancelled", () => {
  assert.equal(needsReview(order({ cancelledAt: "2026-07-10T00:00:00Z" }), "fraud-review"), false);
});

Case studies

High value

The electronics order that never should have shipped

A store selling small electronics shipped a large high-risk order same day because the packing team worked strictly by order date. Three weeks later it came back as a chargeback, and the goods were long gone.

The job now tags every high-risk order the moment it arrives. Staff clear the review tag before anything ships, and the store has not lost a high-value order to that kind of fraud since.

Third party fulfillment

The warehouse that only saw picking lists

A brand used an outside warehouse that received orders through an integration and never saw the Shopify risk flag. Risky orders shipped automatically because nothing told the warehouse to hold them.

The team ran the job in dry run first, saw which recent orders were high risk, then let it tag them. They set the integration to skip the review tag, so tagged orders wait for a human while everything else flows through.

What good looks like

After this runs on a schedule, every high-risk order is visible in one saved view before it ships. Fraud losses drop because the risky orders stop slipping through with the crowd, and your team spends a minute on the few that matter instead of opening every order. The decision stays human, the finding is automatic.

FAQ

Why did Shopify let a high-risk order ship?

Shopify scores an order for fraud risk but does not stop it on its own. If no one reviews the high-risk orders, they get fulfilled like any other, which is where chargebacks come from. A job that tags high-risk orders for review makes sure a human sees them before they ship.

How do I find high-risk orders with the API?

Read the order risk on each order. An order is high risk when its recommendation is to cancel or when any of its risk assessments has a risk level of HIGH. Reading this over the Admin API lets you act on every high-risk order without opening each one by hand.

Is it safe to act on risky orders with a script?

Yes, when the script only tags the order for review and never cancels or refunds on its own. Tagging is reversible and leaves the decision to a person. Start in dry run mode to see the list before it tags anything.

Related field notes

Citations

On the problem:

Shopify Help Center: fraud analysis and order risk levels. help.shopify.com/en/manual/fraud/fraud-protect
Shopify Help Center: reviewing an order for fraud before you fulfill it. help.shopify.com/en/manual/fraud/managing-orders
Shopify Community: high-risk orders were fulfilled without review. community.shopify.com payments and shipping

On the solution:

Shopify Admin GraphQL: the order risk fields, recommendation and assessments. shopify.dev/docs/api/admin-graphql/latest/objects/OrderRiskSummary
Shopify Admin GraphQL: the tagsAdd mutation. shopify.dev/docs/api/admin-graphql/latest/mutations/tagsAdd
Shopify Admin GraphQL: the orders query and its search syntax. shopify.dev/docs/api/admin-graphql/latest/queries/orders

Stuck on a tricky one?

If you have a problem in Shopify orders, payments, subscriptions, inventory, or fulfillment that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.

Contact me on LinkedIn

Did this stop a chargeback?

If this caught a risky order before it shipped, you can buy me a coffee. It is the best way to keep these field notes free and growing.

Buy me a coffee on Ko-fi

Back to all Shopify field notes