Is your AI support agent working? Six numbers to watch

Every AI agent vendor will tell you their product “deflects” a large share of tickets. Deflection is easy to inflate: a customer who gives up and closes the chat was technically deflected. To know whether your agent is actually helping, you need a small set of honest numbers, and the habit of reading real conversations.

Here are the six we recommend, and how to get each one.

1. Resolution rate, defined strictly

What it is: the share of conversations where the customer’s question was answered or their request completed, without a person.

How to measure it honestly: don’t count a conversation as resolved just because it ended. Count it when the agent answered from your knowledge and the customer did not come back with the same question, or confirmed the answer helped.

What to do with it: a low rate usually means missing knowledge, not a bad model. Look at number 3.

2. Handoff rate and handoff reasons

What it is: the share of conversations passed to your team, and why.

Why it matters: a healthy agent hands off the right conversations quickly. If the handoff rate is near zero, check whether the agent is guessing when it should escalate. If it is very high, the agent is missing knowledge or actions.

What to do with it: group handoffs by reason every week. The top reasons are your roadmap.

3. Knowledge gaps

What it is: questions the agent could not answer from your content.

Why it matters: this is the most actionable number you have. Every gap is a document, FAQ entry or page you can add.

In Convosphere: unanswered questions surface as knowledge gaps, so you can see which topics to add content for. Add the content, re-sync, and watch the gap close.

4. Answer accuracy, from a weekly sample

What it is: of the answers the agent gave, how many were correct and complete.

How to measure it: there is no shortcut. Each week, read a random sample of 20 to 30 conversations and mark each answer as correct, partly correct or wrong. Check that the quoted source actually supports the answer.

What to do with it: wrong answers usually trace back to outdated or conflicting content. Fix the source, not the prompt.

5. Customer sentiment

What it is: how customers feel during and after the conversation.

How to measure it: Convosphere tracks sentiment across conversations on Professional and above. Look at the trend, and read the most negative conversations each week. They are rarely about the AI itself; they are usually about a policy or a process the agent had to explain.

6. Cost per resolved conversation

What it is: what you pay for the agent, divided by the conversations it actually resolved.

How to calculate it: take your plan price for the month, add any other AI costs, and divide by resolved conversations from number 1, not by all conversations. Compare it with the cost of a person handling the same conversations, including the time your team spends reviewing and improving the agent.

Why it matters: this is the number your finance team will ask for, and it only means something if the resolution number underneath it is honest.

A 30-minute weekly review

You don’t need a data team for this. Once a week:

1. Check resolution and handoff rates for the week (5 minutes)
2. Read the top five knowledge gaps and assign content to fill them (10 minutes)
3. Read a sample of conversations and mark accuracy (10 minutes)
4. Skim the most negative conversations for process problems (5 minutes)

Teams that keep this habit improve their agent every week. Teams that only look at a deflection dashboard usually don’t.

---

Try it: Start free and see what your customers actually ask. Or read how to set up human handoff properly.