How to evaluate an AI customer support tool: a buyer's checklist

Every AI support tool demos well. The gap that matters shows up weeks later. When the AI invents a policy, sounds nothing like your team, or the invoice balloons because your quiet month was actually a busy one. This is a vendor-neutral checklist for evaluating an AI customer support tool before you commit, with the concrete questions to put to each vendor.

Grounding: where do the answers come from?

The single most important question is what the AI does when it doesn't know. A model asked to answer with no source material will produce a confident, plausible, wrong reply, a hallucination. The safe behaviour is the opposite: draw answers from your content, and when there's no support for an answer, flag the case for a human instead of guessing.

Ask each vendor:

• Are replies grounded in my own knowledge base, or the model's general training?
• What happens when the knowledge base has no answer, does it guess, or flag?
• Can I see why it drafted a given answer, and correct the source?

A tool that would rather say nothing than say something wrong is worth more than a tool that always has an answer.

Tone: does it sound like you?

Customers can tell when a reply came from a generic bot, and trust drops when it does. The tool should mirror how your team actually writes (greeting, length, phrasing) not impose a flat corporate voice.

• Can it learn our tone from examples of our real replies?
• Can we adjust tone per channel, or is it one setting for everything?
• Does the drafted reply read like a teammate wrote it, or like a template?

Human-in-the-loop vs full automation

"Automation" can mean two very different things: AI that drafts a reply for a person to approve, or AI that answers and sends on its own. Both have a place, but conflating them is how a small mistake reaches a thousand inboxes before anyone reads it.

• Can a human review every draft before it sends, at least to start?
• If auto-send exists, is it per-case and confidence-gated, or all-or-nothing?
• Is there always an obvious path for a customer to reach a person?

Channels: does it cover where your customers actually are?

A tool that only handles email leaves a gap the day a customer messages you on WhatsApp. Map the tool's channel coverage against where your customers actually reach you today, and where they will next year.

• Which channels are supported: email, chat, WhatsApp, Telegram?
• Does email include Gmail and Google Workspace, or only IMAP?
• Is it genuinely one shared workflow across channels, or separate silos?

For the wider picture on covering more than one channel well, see our guide to multichannel customer support.

Pricing: will the bill be predictable?

Pricing model matters as much as price. The common models behave very differently:

• Per seat: you pay per agent. Predictable, but it penalizes adding reviewers and doesn't track the value you get.
• Per message: you pay for every message processed. A single back-and-forth conversation can rack up several charges, and a busy month is a nasty surprise.
• Per resolved case: you pay when a conversation is actually handled. This tracks value most closely and is the easiest to forecast: one email thread or one chat is one case, regardless of how many messages it took.

Ask:

• What exactly counts as a billable unit, and when does the meter start?
• If a customer sends five follow-ups, is that one charge or five?
• Are cases my team handled manually still billed?
• Is there a free tier to test with, without a credit card?

There's no universally "correct" model, but predictability is worth paying attention to. Our own pricing explains one per-resolved-case approach in detail.

Data handling and GDPR

Connecting an AI tool to your inbox means handing it your customers' personal data. That's a real responsibility, not a checkbox.

• Where is the data hosted. A named EU region, or a vague "cloud"?
• What's the default retention period, and can I shorten it?
• What happens to the data when I disconnect a channel?
• Is my customers' data used to train models? (For most B2B use you want a clear no.)
• Is there a signed Data Processing Agreement?

We go deep on this in GDPR-compliant AI support. Worth reading before you sign anything.

Onboarding: how fast to first value?

A tool that takes a quarter to configure has failed before it started. You should be able to connect a channel, point it at some knowledge, and see a useful draft the same day.

• How long from sign-up to the first useful drafted reply?
• Do I need engineering help, or can a support lead set it up alone?
• Can I start small (one channel, a handful of articles) and grow?

Exit: how easily can you leave?

The question few buyers ask until it's too late: if this doesn't work out, how hard is it to get out? A tool you can't leave is a tool you don't really control.

• Can I export my data and my knowledge base in a usable format?
• If I delete my account or disconnect a channel, is the data actually deleted?
• Am I locked into a long contract, or can I stop when I need to?

How SupportWunder fits

Measured against this checklist, SupportWunder is deliberately assistance-first: it reads each incoming message and drafts a reply in your tone, grounded in your knowledge base, and when there's no grounding, it flags the case rather than inventing an answer. A human reviews before sending; auto-send is optional, per-case, and confidence-gated.

It covers email (including Gmail and Google Workspace), WhatsApp, Telegram, and an embeddable website chat in one workflow. Billing is per resolved case, not per seat or per message (one email or chat conversation within 24 hours) with a free plan of 25 resolved cases a month and no credit card, and manually handled cases free. On data: EU hosting, a 90-day default retention, and disconnecting a channel deletes its data.

None of that removes your job of evaluating it against your own needs. Bring this checklist, ask the hard questions, and start small. For the wider landscape, see our guide to AI in customer service and the AI customer support software overview.