AI-powered vision-enabled agentic solution for ERP order intake automation

A US fine-jewelry manufacturer producing made-to-order pieces for a couple of hundred independent retail jewelers runs every order through Microsoft Dynamics 365 Business Central. A retail jeweler describes what their customer wants; the shop casts, sets and finishes it.

That flexibility is why the front door is a mailbox — one with six figures of history — rather than an order form.

A jeweler types what they want at 11pm and attaches a phone photo of a hand sketch, sometimes a CAD file, sometimes a competitor’s product page — often all three in one thread.

Someone on the order desk reads each thread and types it into the ERP. When a detail is unstated, staff email back and wait.

Two costs sit underneath that one. Every confirmed order also fans out into three linked sub-orders, hand-created, one for each shop role — and the written instructions for each role are in a different vocabulary, so the same brief has to be told four different ways.

And the rules governing all of it — material standards, per-customer preferences, when to ask versus when to assume — live in two or three people’s heads. A cost problem and a succession problem at once.

• DataThe customer master had no email column at all, and two customer records were found sharing one address — while 100% customer identification was the hard bar.
• SignalRoughly three conversations in ten in the mailbox are machine or internal traffic — a separate, mailbox-level measurement. Inside the pilot threads themselves, 62% of the messages were internal noise a human must discard first, and only the remaining 38% is customer signal.
• SecurityThe shop’s own replies quote the finished ticket, so a leaky filter lets the agent copy the answer instead of extracting it — and one that reads message bodies lets a sender suppress their own message.
• OperationalA small review team under permanent backlog pressure; anything added to their day must beat what it replaces on the first attempt.

Each of these was written into a dependency and risk register before the production phase began, with a likelihood, an impact and a mitigation per row — and one of them was ranked above all the others. The customer master had no email column, so mapping senders to accounts depended on a list only the client could produce. Our own note on it was blunt: this is the single highest-risk dependency, and the only gap the proof of concept could not close from its own data. The agreed fallback was written down at the same time — domain matching covers most accounts, unmatched senders route to a manual identification step in the review console rather than a guess, and the fully automated identification target stays unclaimed until the mapping exists. That is why the result below is reported as 100% on the pilot corpus and not as a claim about every account.

We built the proof of concept in ten business days with every integration mocked, each one behind the interface its live counterpart would later have to satisfy. In the production service that discipline became a rule with teeth: a new capability has to land in both the mock and the live adapter, because a mock that lags the real thing produces a test suite that passes while production fails.

The evaluation set came free. The ERP already held finished job cards paired with the messy email that produced them — ground truth nobody had to annotate.

The bar was not ours to set. Before the build began, the client fixed four pass conditions for phase one: customer identification had to work for known accounts and for the domain-fallback cases; extraction and clarification quality had to match what the order desk expected on the pilot set; the output had to be accepted as ERP-entry-ready by their own team; and any field not traceable to the email or an image counted as a failure — zero tolerance, not a percentage. Day ten was a demo and a written go/no-go recommendation, with iterating the same phase as an explicit, non-embarrassing option. All four conditions were met and the client proceeded to the production build.

The production phase carried its own acceptance table — a pass condition per deliverable, agreed in advance: no internal message may ever reach the agent as a source; low-confidence images are flagged, never guessed; no order reaches assembly with a critical field missing; escalation to a human after exactly two unresolved rounds; every correction captured before-and-after and exportable, with the data owned by the client.

What we chose not to do matters as much: no fine-tuning, no bigger model. When accuracy plateaued, the gains came from a deterministic verification pass and from measuring the model’s confidence rather than trusting it.

1. 01Ingest — No model. Internal senders are filtered on the sender field only, never on body text; customer resolution is deterministic code against the ERP’s contact records, and an unresolved customer becomes a flag, not a guess. Images go to Gemini Flash.
2. 02Extraction — One Claude Sonnet call fills roughly 25 job-card fields with per-field confidence, from the filtered thread, the customer record, the image reading and the shop’s written procedure.
3. 03Verification — No model, and this is where the accuracy lives: canonicalization, house rules, vocabulary enforcement, per-service-type required-lane checks, a format linter, then confidence calibration.
4. 04Clarification — Gaps become at most three targeted questions in the shop’s voice, drafted by a smaller model and skipped when there are none. After two unresolved rounds the job escalates to a person.
5. 05Assembly — The approved order becomes one job card carrying fourseparately-voiced instruction lanes — CAD, jeweler, stone setter, polisher — and three linked sub-order records, one for each shop role that has to be scheduled and paid. Skipped whenever gaps remain open.

Every field lands in a review console showing its value, its confidence and its source.

• 01Abstain

  Below threshold, it leaves the field blank and asks.

  Zero hallucinated fields was the client’s own acceptance criterion, enforced structurally rather than by prompt wording.

• 02Calibrate

  The reviewer sees expected accuracy, not the model’s opinion of itself.

  Self-reported confidence is systematically overconfident; the thresholds run on a curve fitted to labelled data. Fixing this made the dashboard look worse, which was correct.

• 03Attribute

  An image reading is marked, never merged.

  Tagged and shown beside the field for line-by-line confirmation, never authoring the record. Confirming one is recorded as a different event from correcting it, so the learning loop cannot mistake agreement for teaching.

• 04Contain

  Read production, write sandbox.

  The service reads live ERP data but routes writes to a sandbox company by default; pointing them elsewhere is a deliberate configuration change.

• 05Gate

  Nothing reaches a real job card without a person.

  Approval is gated on permissions, required-field validation and policy checks, and every correction is stored before and after.

What is unfinished is written down rather than called done: webhook signature verification is still a stub, and access control on some routes has known gaps.

• 83% field-extraction accuracy across 11 real orders, scored field-by-field against each order’s finalized ERP ticket. The deterministic verification pass raises the offline re-score to 88.6%.
• 97.7% exact match on the eight critical fields (86 of 88) — customer, category, service type, metal, ring size, centre-stone responsibility. Those decide whether a card is usable.
• 100% customer identification on the pilot corpus, without a single guess. Measured on those orders, not claimed for every trade account — the customer master still has no email column.

The rules that lived in two or three heads are now a versioned document a non-engineer can edit. Every field carries provenance, so a disagreement about an order is a lookup, not an argument.

• About 79 seconds and eleven cents of model spend per order, over three to seven model calls.