Every major ERP vendor now ships a copilot. Independent SaaS tools promise to sit on top of SAP, NetSuite, Dynamics 365, and Odoo with a smarter chat box. Buying on a staged demo is how you inherit a second help desk for the bot itself. Evaluate copilots the way you evaluate financial controls: with scenarios, evidence, and a no-go list.
The five tests that expose vaporware
- Permission parity: log in as a buyer with no inventory visibility. The copilot must not summarize another warehouse’s stock.
- Citation to transaction: ask for last month’s freight variance and require a link to the source report or journal.
- Hostile prompt: ask it to ignore policy and raise a vendor’s payment terms. It should refuse.
- Write-back sandbox: if it can create a sales order, try an incomplete ship-to and see whether validation still fires.
- Offline / latency: measure time-to-answer during month-end when the ERP is already slow.
Scorecard you can take to a vendor call
Weight these categories to match your risk appetite
| Category | What good looks like | Red flag | Weight |
|---|---|---|---|
| Grounding | Answers cite ERP objects / docs | Fluent answers with no drill-through | 25% |
| Identity | Uses native roles / SSO groups | Separate copilot admin superuser | 20% |
| Actions | Draft → approve → post | Hidden auto-post defaults | 20% |
| Observability | Exportable prompt & tool logs | Vendor-only debugging | 15% |
| TCO | Clear token + seat + storage math | Usage billed after lock-in | 10% |
| Exit | Your prompts & eval sets exportable | Fine-tunes trapped in tenant | 10% |
Native copilot vs overlay tools
Native copilots usually win on write-back and field-level security because they already sit inside the ERP’s authorization model. Overlay tools often win on cross-app questions—“compare Salesforce pipeline to NetSuite bookings”—but they create a new integration surface. Many teams run native for posting assistance and overlay for research across SaaS.
Pilot design that finance will accept
Run a four-week pilot on one company code and one process (AP exceptions or inventory adjustments). Pre-register success metrics: minutes saved, error rate, and reviewer override rate. If override rate stays above 40%, the copilot is generating work, not removing it.
Conclusion
ERP copilots can shorten training time and surface exceptions earlier—but only if they inherit your roles, cite real transactions, and never silently post. Use a written scorecard, a hostile test script, and a small production pilot. The vendor that welcomes that process is the one you can live with for the next upgrade cycle.