AI for customer support that actually resolves tickets

The three jobs AI can do in support

• Agent assist. For each incoming ticket the model retrieves relevant help articles, past resolved tickets and the customer’s account state, then drafts a reply. A human edits and sends. Low risk, fast to ship, and the edit rate tells you exactly how good the drafts are.
• Triage and routing. Classify intent, product area, urgency and language; detect billing disputes or legal threats; route to the right queue. This is classic text classification and LLMs are very good at it with a few dozen labeled examples per category.
• Self-service resolution. A customer-facing assistant answers questions and, with tools, takes actions: checks an order, resends an invoice, changes a plan. Highest value, highest risk – the output goes straight to a customer and actions touch real accounts.

Most teams get a better return from doing the first two well than from rushing the third. They also generate the labeled data and the trust you need before letting AI talk to customers unsupervised.

Build vs buy: picking the best AI agents for customer support

If you’re on a mainstream helpdesk, start with its native AI features or a dedicated support-AI vendor. They already handle the inbox integration, handoff, analytics and multi-channel plumbing, and that plumbing is most of the work. Evaluate them on your own tickets, not their demo.

Build – or build part of it – when:

• Resolution depends on your product’s data and actions: account state, usage, configuration, entitlements. Vendors can call your API, but someone still has to design those tools and their permissions.
• Your support content is technical, and generic retrieval over help-center articles gives shallow answers.
• Per-resolution pricing gets expensive at your volume. Run the numbers against your actual ticket counts before signing.
• You need data to stay inside your infrastructure or region for compliance reasons.

A common split: keep the helpdesk and its UI, build your own retrieval and tool layer, and plug it in. That’s the shape I’d aim for in AI knowledge base work.

RAG over help docs: where answer quality comes from

Customer-facing answers should be grounded in retrieved content, not the model’s general knowledge. That’s retrieval-augmented generation, and the quality is decided almost entirely by the retrieval side:

• Fix the content first. Contradictory or stale articles produce contradictory answers. An audit of the top 50 articles by ticket volume usually pays for itself.
• Include resolved tickets, cleaned of personal data. They capture how problems are actually solved, which help articles often don’t.
• Use hybrid search – keyword plus vector – so error codes, plan names and feature names match exactly.
• Filter by audience and plan. Don’t answer a starter-plan customer with an enterprise-only feature.
• Cite sources in every answer and link them. It builds customer trust and makes bad answers easy to trace back.

I cover chunking, reranking and retrieval evaluation in RAG architecture.

Designing escalation

Escalation design separates AI support people like from AI support people screenshot and complain about. The rules I start with:

1. Always offer a human path, visibly, from the first message. Hiding it raises frustration and costs you more in churn than it saves.
2. Escalate on signals, not just requests: repeated rephrasing, negative sentiment, billing disputes, cancellation intent, security or legal topics, and any case where retrieval found nothing relevant.
3. Escalate when confidence is low. If the answer isn’t supported by retrieved content, say so and hand off rather than improvise.
4. Hand over the full context: transcript, retrieved articles, account details and what the AI already tried. The customer should never repeat themselves.
5. Cap the loop. After two or three unresolved turns, hand off automatically.
6. Gate actions with consequences. Refunds, plan downgrades and account deletion need explicit confirmation from the customer and usually a policy check or human approval.

Measure resolution, not deflection

Deflection counts conversations that didn’t reach a human. It rewards making escalation hard. Track these instead:

• Verified resolution rate – conversations where the customer confirmed the fix, or didn’t contact you again about the same issue within 7 days.
• Repeat contact rate for AI-handled vs human-handled conversations.
• CSAT split by channel, with a free-text sample reviewed weekly.
• Escalation quality – how often the human agent had to re-ask for information the AI already had.
• Draft acceptance and edit distance for agent assist.
• Cost per resolved conversation, including model usage, vendor fees and the human time spent on escalations.
• Downstream retention for customers who used the assistant vs comparable customers who didn’t.

Wire these up before launch so you have a baseline. Then compare. For the business-case side, see measuring AI ROI.

A rollout plan that limits risk

1. Weeks 1–2: export a few hundred representative tickets, label intent and correct resolution, and build the evaluation set. Ship triage and routing.
2. Weeks 3–4: ship agent assist on one queue. Track edit rate and handle time.
3. Weeks 5–8: fix the content gaps the drafts expose. Add tools for the top read-only lookups (order status, plan, recent invoices).
4. Then: turn on customer-facing answers for a narrow set of intents where draft acceptance was consistently high, behind a clear human handoff.
5. Ongoing: sample conversations weekly, add failures to the evaluation set, and re-run it before every prompt, model or content change. See LLM evaluation.

Common failure modes

• Confidently answering from outdated docs after a pricing or feature change.
• Promising things policy doesn’t allow – refunds, discounts, timelines. Constrain what the model can commit to and check it in an output filter.
• Prompt injection through customer messages or attachments, especially once the assistant has tools. Scope tools to the authenticated customer only.
• Leaking another customer’s data because retrieval wasn’t filtered by account.
• A bot that loops politely instead of escalating.

Most of these are guardrail problems rather than model problems. LLM guardrails covers the practical controls.

Related

• An AI knowledge base your team can actually trust
• RAG development services, measured on retrieval quality
• RAG architecture for production systems
• AI ROI: how to measure the return on AI features
• LLM guardrails for real product features
• AI integration services for products already in production