AI ROI: how to measure the return on AI features

Two kinds of AI ROI

Separate these early, because they’re measured differently.

• Internal efficiency. AI that makes your own team faster or cheaper: support agent assist, document processing, coding assistants, sales research. The return is time and cost saved per task, and the counterfactual is how the team works today.
• Product value. AI features your customers use. The return shows up in conversion, expansion, retention, pricing power, or support load per customer. The counterfactual is comparable customers or periods without the feature.

Internal efficiency is easier to measure and usually the better place to prove the approach. Product value is where the bigger upside is, and where attribution is hardest.

The core unit: cost per successful task

Pick the unit of work – a resolved ticket, a processed invoice, a merged pull request, an answered question – and compute the fully loaded cost of a successful one, with and without AI.

• Model cost per task: tokens per task across every call, including retries, guardrail checks and agent loops, at current provider prices. See OpenAI API cost for how to estimate it.
• Human time per task: review, editing, approvals and handling escalations. This doesn’t go to zero; it moves.
• Failure cost: what a wrong output costs when it slips through – a refund, a rework cycle, an annoyed customer – multiplied by the error rate you measured.
• Success rate: a task the AI attempted and a person redid isn’t a saving. Divide by successful tasks, not attempted ones.

Compare that number with the current cost per task. The difference times monthly volume is your gross monthly benefit. If the AI version is only cheaper when you ignore review time, it isn’t cheaper.

Measuring support AI: resolution, not deflection

Support is the most common AI business case and the easiest to get wrong. Deflection rate – conversations that never reached a human – goes up when you make escalation hard, and the cost shows up later as repeat contacts and churn.

Measure instead: verified resolution (the customer confirmed, or didn’t come back about the same issue within a set window), repeat contact rate, CSAT by channel, handle time for escalated tickets, and cost per resolved conversation including vendor fees. I go through the operational side in AI for customer support.

Measuring product AI: retention and revenue

• Adoption with depth. Weekly active users of the feature, and repeat usage per user. A spike of curious first uses followed by a drop-off is the common pattern; watch the second and fourth weeks, not launch week.
• Task completion. Did users accept the output – keep the draft, apply the suggestion, run the generated query? Acceptance and edit rates are the closest thing to a quality metric users give you for free.
• Retention. Compare retention of accounts that adopted the feature against similar accounts that didn’t. Adopters are often your most engaged customers anyway, so match on prior engagement and plan, or the comparison flatters the feature.
• Revenue. If the feature is priced (add-on, higher tier, credits), track attach rate and gross margin after model costs. If it isn’t, look for movement in conversion or expansion – ideally with a holdout group or staged rollout so you have a real comparison.
• Support load. Tickets per active account for adopters versus non-adopters.

The hidden costs

These rarely appear in the original business case and can easily exceed the API bill in the first year:

• Engineering time to build, and then to maintain prompts, tools, retrieval pipelines and integrations as the product changes.
• Evaluation: building and labeling test sets, expert time to review outputs, running evals on every change.
• Content and data work: cleaning the help center, fixing permissions, deduplicating documents. AI exposes messy data immediately.
• Human review and escalation time, which persists after launch.
• Model churn: providers retire models; each migration means re-running evals and often re-tuning prompts.
• Security and compliance: data processing reviews, vendor assessments, audit logging, incident response for a new class of failure.
• Vendor pricing changes: per-seat or per-resolution pricing that looked fine at pilot volume can look different at full volume.
• Opportunity cost: what the team would have shipped instead.

A measurement plan you can run

1. Baseline first. Before building, record current volume, time per task, cost per task, error rate and the relevant customer metrics. Without a baseline there’s no ROI, only anecdotes.
2. Define success per task in a way you can check – a resolved ticket, an accepted extraction, a merged PR.
3. Instrument from day one: log tokens, model, latency, outcome, human edits and escalations per task, attributed to feature and customer.
4. Roll out with a comparison: a holdout group, a staged rollout by team or region, or at minimum a before-and-after on matched periods.
5. Review monthly with fully loaded costs: model spend, vendor fees, engineering and review hours.
6. Set a kill criterion up front. If cost per successful task isn’t below baseline by a date you agree, stop or change course. Stopping a weak AI project early is a good outcome.

A worked template

Use this shape for the business case, filled with your own measured numbers:

• Monthly volume of the task.
• Current cost per task: minutes × loaded hourly cost, plus error-handling cost.
• AI cost per successful task: model cost + review minutes × hourly cost + failure rate × failure cost, divided by success rate.
• Gross monthly benefit: (current cost − AI cost) × volume, plus any measured revenue or retention effect.
• Monthly running costs: maintenance engineering, evaluation, vendor fees, monitoring.
• One-off costs: build, data cleanup, security review.
• Payback period: one-off costs ÷ (gross monthly benefit − monthly running costs).

If the payback period is longer than the likely life of the current model generation, be cautious – you’ll be paying migration costs before you’ve recovered the build.

Where I fit

If you already have a use case with clear volume and a team that can build it, you may not need help – the plan above is the hard part, and it’s mostly discipline. If you’re deciding where AI would pay back at all, an AI readiness assessment looks at your processes, data and systems to find the candidates worth testing. If you’ve picked one and want it built and measured properly, that’s AI consulting done hands-on: I ship the first version in your codebase with the instrumentation in place.

Related

• An AI readiness assessment that ends in working code
• AI consulting services for software companies
• OpenAI API cost: estimate it, then cut it
• AI for customer support that actually resolves tickets
• AI agents for business: what’s worth building
• LLM evaluation for product features