---
title: "How to ask the AI judge why it assessed a criterion that way"
description: "The AI Assistant asks the checklist's AI agent why it passed or failed a criterion in an assessment: enabling it, permissions, how to ask, what the answer contains and its limits."
---

# How to ask the AI judge why it assessed a criterion that way?

A dialog assessment is made by the **checklist's AI agent** — the "AI judge". When a verdict on a criterion looks odd, you can ask the **AI Assistant** to ask the judge itself: which operator message it relied on, which rule decided it (the criterion text or a knowledge-base answer) and which knowledge bases it queried.

The AI Assistant shows the **stored AI verdict** (and the human edit, if there is one) next to the **judge's answer**. **The assessment is not changed** — nothing is written.

The feature is **off by default**, and every question is a **separate paid request** to the checklist's AI agent.

## When you need this

- AI failed a criterion, but you see no violation in the dialog — you want to know which quote the judge saw.
- You suspect the judge took a rule from a knowledge base that is **not about this topic**.
- You are about to fix the checklist or the knowledge base and want to understand what confused the judge.

## What's important to know

- **It is a new request to the same AI agent.** The judge does not see its conversation from the assessment — only the same dialog, the **current** criterion text and the stored verdict. If the agent's instructions, the criterion or the knowledge base changed after the assessment, the judge answers against the new ones.
- **The verdict may differ.** The judge may now reach a different conclusion than in the stored assessment — the AI Assistant warns you. The assessment is not changed: a person decides ([Adjust AI assessments](/en/quality_assurance/how-to/adjust-ai-assessment.md)).
- **One request — one criterion.** To look into several criteria, ask about each one separately.
- **Daily limit.** Questions to the judge are limited per instance per day (UTC) — **30** by default. When the limit is used up, the AI Assistant says when it resets.
- **Costs.** Each question is a request to the checklist's AI agent (model tokens). Such requests are not counted as "AI answers", but when the plan's AI answer limit is used up, the judge cannot be asked either.
- **Only in the AI Assistant's write mode.** In read-only mode, in Ask mode and in Plan mode before the plan is confirmed the judge cannot be asked — the assistant tells you what to change.

## Before you start

- [x] The **Quality Assessment** module and **Instance Agent Integration** are enabled.
- [x] A root user turned on **Instance Agent: Ask AI QA judge** (step 1).
- [x] Your role can **edit assessments** in the Quality Assessment module and has **Allow the agent to create, modify, and delete data (write mode)** for the AI Assistant; **Agent Read-Only Mode** is off.
- [x] The checklist has an AI agent, and the assessment has an AI verdict on the criterion.

## Step-by-step instructions

### 1. Turn the feature on for the instance (root)

1. Go to **Menu → Settings → Instance settings**.
2. Expand the **Instance Agent** category.
3. Turn on **Instance Agent: Ask AI QA judge**.
4. If needed, change **Instance Agent: AI QA judge questions per day** — how many questions to the judge the whole instance may ask per day (1 to 1000; an empty or invalid value means 30).

<!-- screenshot: instance settings, Instance Agent category, the two new rows -->

### 2. Grant the permissions to roles

1. Go to **Menu → Settings → Users → Roles** and open the role.
2. In the **Quality Assessment** module, grant **edit** rights for the **Assessments** section — only users who can edit assessments can ask the judge.
3. In the **AI assistant (Instance Agent)** module, turn on **Allow the agent to create, modify, and delete data (write mode)**.
4. Save the role.

### 3. Ask the judge in the chat

1. Open the **AI Assistant** in **Agent** mode.
2. Say explicitly that you want to ask the judge, for example:
   - "Ask the judge why criterion "Refund by the knowledge-base rules" failed in assessment 4055";
   - "Ask the AI judge about assessment 4055: which message did it rely on?"
3. If you do not name the criterion, the assistant takes the only criterion AI failed; if there are several, it asks which one.

To a general question like "why did AI take points off?" the assistant answers from the stored assessment and does not ask the judge on its own; at the end it **may offer** to ask the judge. The separate paid request goes only when you ask for it explicitly or say "yes" to such an offer.

<!-- screenshot: AI Assistant chat — question and the judge's answer on a criterion without a knowledge base (scenario 1) -->

## What the answer contains

1. **Criterion and assessment** — the criterion name, the assessment number.
2. **Stored AI verdict** — passed or not, points, the AI comment; if a person changed the assessment — their edit and comment.
3. **The AI judge's answer** — its explanation (the assistant shortens a long answer, keeping the key quotes). If the judge re-evaluated the criterion instead of answering the question, its verdict now comes first.
4. **Messages the judge relied on** — the numbers and the beginning of the dialog messages.
5. **The judge's knowledge-base requests** — what the judge searched for and what it got; a request to a knowledge base **not attached to this criterion** is marked. Then the knowledge-base fragments, if any came back.
6. **Caveats** — for example, that this is a new request, that the verdict now differs, that the criterion or the dialog changed after the assessment, that the judge did not query the attached knowledge base.
7. **"The assessment was not changed. Questions to the judge today: N of M."**

<!-- screenshot: the judge's answer on a criterion with a knowledge base — knowledge-base requests and fragments (scenario 2) -->

## Limitations

| Situation | What the assistant says |
| --- | --- |
| The feature is off on the instance or the role cannot edit assessments | That it cannot ask the judge now, and who turns it on |
| Read-only mode / Ask mode / Plan mode before confirmation | That write mode / Agent mode / plan confirmation is needed |
| The daily limit is used up | How many questions per day are allowed and when the limit resets |
| The judge is already being asked about this criterion | To wait for the answer to the previous question |
| The AI judge did not answer or the request failed | The assessment was not changed, try later; the assistant does not ask again in the same message |
| AI has not evaluated this assessment yet | To run the AI evaluation first |
| The criterion was deleted from the checklist or AI did not evaluate it | That there is nothing to ask about |

## What happens next

- The assessment, the criteria and the checklist settings **do not change**; the judge's answer stays in the AI Assistant chat history.
- If the answer shows an AI mistake, [adjust the assessment](/en/quality_assurance/how-to/adjust-ai-assessment.md) manually, and fix the cause (criterion text, knowledge base) in the checklist.

## Related materials

- [Adjust AI assessments](/en/quality_assurance/how-to/adjust-ai-assessment.md)
- [View assessment details](/en/quality_assurance/how-to/view-assessment-details.md)
- [What is the AI Assistant](/en/constructor/explanation/what-is-ai-assistant.md)
- [How to configure access to the AI Assistant](/en/constructor/how-to/configure-ai-assistant-access.md)