Before you connect any AI to your live store, evaluate its harness — the layer of tools, permissions, logging, and scheduling between the model and your business — by asking ten questions: what tools exist, what pauses for approval, what can never run unattended, where the ledger is, who owns the schedule, how events trigger work, where memory lives, what happens when models improve, how access is granted, and what the exit looks like. The e-commerce harness is where AI-for-commerce products actually differ; the model underneath is increasingly the same. Here are the ten questions, and what good answers sound like.
1. Can I read the complete tool list?
The AI’s capabilities should be an enumerable catalog — typed tools with defined inputs, one per capability — not “the AI has access to your store.” Good answer: a browsable list (“ask it what tools do you have?“), each tool labeled read or write. Bad answer: anything involving the phrase “full access.”
2. What exactly pauses for approval — and where do I approve?
Reads should flow freely; writes should stop and show you the exact change plus the agent’s reasoning, approvable in one tap. Good: per-action cards on web and mobile, because store problems don’t wait for your desk. Bad: batch “review the log later” — that’s an audit, not a gate. The gate is the entire trust model, argued in full here.
3. What can never run unattended?
Ask this verbatim — it’s the fastest vendor filter. Good: a specific list, hard-coded above the settings: mass email to your customer list is the canonical member. Bad: “whatever you configure.” A harness whose safety is entirely user-configurable has no safety, only defaults.
4. Where is the ledger?
Every run, every read, every proposed and executed change, timestamped and permanent. Good: a results ledger you can scroll and search — delegation stays reversible because nothing is silent. Bad: run summaries that expire, or logs only support can see. (What a real audit trail contains.)
5. Who owns the schedule?
If you have to remember to ask, you’re still the operator. Good: agents on cron-like schedules (the weekly SEO audit, the daily ads sweep) that arrive as finished work. Bad: a chat box, however smart. The difference between a tool and an employee is who owns the loop.
6. Can store events trigger work?
Schedules cover the routine; events cover the urgent. Good: live triggers — new order, refund, abandoned checkout — that start the right agent within minutes, with filters so you’re not flooded. Bad: polling “every morning” for things that cost money by the hour.
7. Where does memory live?
An agent with no memory re-learns your store every run — and repeats last month’s mistake. Good: the harness carries history: past runs, past decisions, which agent owns which job, what was tried and rejected. Bad: every conversation starts from zero.
8. What happens when models improve?
The model market moves quarterly; your store shouldn’t re-platform with it. Good: provider-agnostic — swap Anthropic, OpenAI, or Gemini keys and everything else (tools, approvals, ledger, schedules) stays put, so every model upgrade is free capability. Bad: a harness welded to one provider, which quietly becomes a bet that one lab wins forever.
9. How is access granted — and revoked?
Good: platform-native scoped access — a Shopify app install, an OAuth grant — revocable from your platform’s admin in one click, with the AI never holding your passwords. Bad: shared logins, pasted admin credentials, or API keys with god-scope. The exit matters as much as the entry: revoke should be instant and complete.
10. What does week two look like?
The demo question. Any harness demos well; ask what’s running unattended-but-gated after fourteen days. Good: a concrete answer — “your ads-waste sweep and stockout watch on schedule, order-desk actions arriving as approvals, one workflow you’ve promoted to hands-off.” Bad: “it depends what you ask it.” That’s a chatbot’s answer, and a chatbot is not an autopilot.
Scoring it
No weighting formula needed — the questions are pass/fail and the failures cluster. A vendor who fails 1–3 (tools, gates, limits) is unsafe; one who fails 4–7 (ledger, schedule, triggers, memory) is a demo; one who fails 8–9 (models, access) is a lock-in; one who fails 10 hasn’t shipped the product they’re describing. Ten passes is the bar, not a bonus — every question above is table stakes for software that touches revenue.
The bottom line
You’re not buying intelligence — that’s rented from the model market by everyone equally. You’re buying the harness: the tools, gates, ledger, loop, and exit that turn intelligence into safe store operations. Ask the ten questions, demand specifics, and treat “trust us” as a no.
Run the checklist against a live one — tools, approval cards, and ledger on a real store: watch datavessel run →
Leave a Reply