AI agent
Answer quality
Chatisto automatically scores AI answers for relevance and faithfulness, flags hallucinations and shows which topics are missing from your knowledge base.
Every AI answer that uses your knowledge base is checked automatically by a second AI acting as a judge. Results appear in Quality in the main menu. Evaluations run every hour.
The three scores
Each answer is scored from 1 to 5 on:
| Score | Question it answers |
|---|---|
| Context Relevance | Did the AI find the right content for this question? |
| Faithfulness | Is the answer supported by your content, or did the AI make something up? |
| Answer Relevance | Does the answer actually address what the customer asked? |
The Overall Score combines them, with the trend against the previous period (7, 30 or 90 days).
Flags
Answers with problems are flagged:
- Missing Knowledge: the content needed to answer doesn't exist in your knowledge base.
- Hallucination: the answer includes claims your content doesn't support.
- Off Topic: the answer drifted away from the question.
Open a flagged conversation to see the question, the answer, the scores and the judge's reason.
Knowledge Base Gaps
Chatisto groups missing-knowledge flags into topics, with how often each came up, sample questions and a recommendation. This is your to-do list: add content for the top gaps and the scores go up.
A 10-minute weekly routine
- Open Quality and set the period to the last 7 days.
- Fix the top 3 Knowledge Base Gaps by adding text or documents in Settings → AI Chatbot.
- Read the Hallucination flags. Usually the content is ambiguous; rewrite it. If the AI invents links, turn on link filtering.
- Check again next week.