Test a Checklist on a Real Dialogue

Before applying a new or changed checklist to bulk AI assessment, it's worth checking how it performs on an actual dialogue. This lets you see the AI's result right away and fix criteria wording before it affects real operator assessments.

When It Will Be Useful

  • You just created or changed checklist criteria and want to verify the AI assesses them correctly.
  • AI keeps making the same mistake on one criterion in real assessments — you need to understand why.
  • You want to make sure a criterion's wording doesn't confuse the AI before rolling the checklist out for bulk use.

What's Important to Know

  • Testing a checklist — running an AI assessment on one selected dialogue without saving the result to quality reports.
  • The test result shows the same data as a real assessment: passed/violated, scores, and an AI comment for each criterion.
  • Messages the AI used as evidence for a criterion can be opened directly in the transcript — with scroll and highlight.
  • Testing doesn't affect reports or statistics — it's only a check before the checklist goes into use.

Before You Start

  • You are logged in with administrator or integrator rights.
  • You have navigated to Menu -> Quality Assurance -> Checklists.
  • You have opened a checklist with at least one criterion added.

Step-by-Step Instructions

1. Choose a Dialogue to Test

  1. In the left column of the checklist editor page, click the search field and enter a Chat ID or dialogue name.
  2. Select the dialogue you need from the search results.
  3. The selected dialogue's transcript loads in the left column — it scrolls independently from the criteria column.

2. Run the Test

  1. Make sure the checklist (right column) has criteria added.
  2. Click the "Test Assessment Criteria" button at the bottom of the criteria column.
  3. Wait for the analysis to finish — the button temporarily shows "Analyzing dialog...".

3. Review the Test Results

  1. Each criterion shows its result: "Passed" or "Failed", a score, and an AI comment explaining the assessment.
  2. If the checklist has a dialogue summary prompt configured, an AI dialogue summary appears above the criteria list.
  3. Messages the AI used as evidence for a criterion are highlighted in the transcript.

4. Go to the Evidence Messages

  1. If a criterion has linked messages, a "N msg." badge and a "Go to messages" button appear under its result.
  2. Click that button — the transcript smoothly scrolls to the first linked message and highlights it, temporarily dimming the rest. The button label changes to "Shown".
  3. If there are several linked messages, use the ‹ N / M › stepper next to the criterion's result to step through each one.
  4. To clear the highlight and see the full dialogue again, click the same button ("Shown") again, or select a different criterion.

5. Adjust the Criterion Based on the Results (if needed)

  1. If the AI assessed a criterion incorrectly, click "Edit" next to that criterion and refine its description or check instructions.
  2. Save the changes and run the test again to check the new result.

What Happens After

You can see whether the AI assesses the checklist's criteria correctly on a real example, and adjust the wording before the checklist is used for bulk assessment. Test results are not saved to quality reports.

How to Verify Everything Worked

  • The test result for each criterion matches what you expected from that dialogue.
  • Messages highlighted as evidence actually relate to the criterion they're shown under.
  • The AI comment explains the assessment clearly, without confusing criteria.

Other Ways

  • You can test the checklist on several different dialogues in a row — click "Change" next to the selected dialogue to pick a different example.