How to ask the AI judge why it assessed a criterion that way?
A dialog assessment is made by the checklist's AI agent — the "AI judge". When a verdict on a criterion looks odd, you can ask the AI Assistant to ask the judge itself: which operator message it relied on, which rule decided it (the criterion text or a knowledge-base answer) and which knowledge bases it queried.
The AI Assistant shows the stored AI verdict (and the human edit, if there is one) next to the judge's answer. The assessment is not changed — nothing is written.
The feature is off by default, and every question is a separate paid request to the checklist's AI agent.
When you need this
- AI failed a criterion, but you see no violation in the dialog — you want to know which quote the judge saw.
- You suspect the judge took a rule from a knowledge base that is not about this topic.
- You are about to fix the checklist or the knowledge base and want to understand what confused the judge.
What's important to know
- It is a new request to the same AI agent. The judge does not see its conversation from the assessment — only the same dialog, the current criterion text and the stored verdict. If the agent's instructions, the criterion or the knowledge base changed after the assessment, the judge answers against the new ones.
- The verdict may differ. The judge may now reach a different conclusion than in the stored assessment — the AI Assistant warns you. The assessment is not changed: a person decides (Adjust AI assessments).
- One request — one criterion. To look into several criteria, ask about each one separately.
- Daily limit. Questions to the judge are limited per instance per day (UTC) — 30 by default. When the limit is used up, the AI Assistant says when it resets.
- Costs. Each question is a request to the checklist's AI agent (model tokens). Such requests are not counted as "AI answers", but when the plan's AI answer limit is used up, the judge cannot be asked either.
- Only in the AI Assistant's write mode. In read-only mode, in Ask mode and in Plan mode before the plan is confirmed the judge cannot be asked — the assistant tells you what to change.
Before you start
Step-by-step instructions
1. Turn the feature on for the instance (root)
- Go to Menu → Settings → Instance settings.
- Expand the Instance Agent category.
- Turn on Instance Agent: Ask AI QA judge.
- If needed, change Instance Agent: AI QA judge questions per day — how many questions to the judge the whole instance may ask per day (1 to 1000; an empty or invalid value means 30).
2. Grant the permissions to roles
- Go to Menu → Settings → Users → Roles and open the role.
- In the Quality Assessment module, grant edit rights for the Assessments section — only users who can edit assessments can ask the judge.
- In the AI assistant (Instance Agent) module, turn on Allow the agent to create, modify, and delete data (write mode).
- Save the role.
3. Ask the judge in the chat
- Open the AI Assistant in Agent mode.
- Say explicitly that you want to ask the judge, for example:
- "Ask the judge why criterion "Refund by the knowledge-base rules" failed in assessment 4055";
- "Ask the AI judge about assessment 4055: which message did it rely on?"
- If you do not name the criterion, the assistant takes the only criterion AI failed; if there are several, it asks which one.
To a general question like "why did AI take points off?" the assistant answers from the stored assessment and does not ask the judge on its own; at the end it may offer to ask the judge. The separate paid request goes only when you ask for it explicitly or say "yes" to such an offer.
What the answer contains
- Criterion and assessment — the criterion name, the assessment number.
- Stored AI verdict — passed or not, points, the AI comment; if a person changed the assessment — their edit and comment.
- The AI judge's answer — its explanation (the assistant shortens a long answer, keeping the key quotes). If the judge re-evaluated the criterion instead of answering the question, its verdict now comes first.
- Messages the judge relied on — the numbers and the beginning of the dialog messages.
- The judge's knowledge-base requests — what the judge searched for and what it got; a request to a knowledge base not attached to this criterion is marked. Then the knowledge-base fragments, if any came back.
- Caveats — for example, that this is a new request, that the verdict now differs, that the criterion or the dialog changed after the assessment, that the judge did not query the attached knowledge base.
- "The assessment was not changed. Questions to the judge today: N of M."
Limitations
| Situation | What the assistant says |
|---|---|
| The feature is off on the instance or the role cannot edit assessments | That it cannot ask the judge now, and who turns it on |
| Read-only mode / Ask mode / Plan mode before confirmation | That write mode / Agent mode / plan confirmation is needed |
| The daily limit is used up | How many questions per day are allowed and when the limit resets |
| The judge is already being asked about this criterion | To wait for the answer to the previous question |
| The AI judge did not answer or the request failed | The assessment was not changed, try later; the assistant does not ask again in the same message |
| AI has not evaluated this assessment yet | To run the AI evaluation first |
| The criterion was deleted from the checklist or AI did not evaluate it | That there is nothing to ask about |
What happens next
- The assessment, the criteria and the checklist settings do not change; the judge's answer stays in the AI Assistant chat history.
- If the answer shows an AI mistake, adjust the assessment manually, and fix the cause (criterion text, knowledge base) in the checklist.