Skip to main content
An eval flag is a marker on one reply (a turn) in a conversation. It tells you the reply had a problem, for example the answer was not backed by your content. Each flag has a severity: High, Medium or Low. This page is for agents and supervisors who review conversations. To choose which flags your institution uses, see Set Up Eval Flags.
A turn is flagged only after it has been graded. Grading runs in batches, so a new conversation can show Clean for a while. The Needs review tile shows the conversations that still have turns waiting to be graded.

​
Find flagged conversations

1

Open the Conversations tab

Go to CX Ops > Observe and click the Conversations tab.Four tiles sit above the list:
  • Conversations: all conversations in the date range.
  • Eval flags: conversations with at least one open eval flag, of any severity. It also shows the share of text conversations.
  • Needs review: conversations with turns that are not graded yet.
  • Slow replies: conversations where the average reply took over 10 seconds.
Click a tile to show only those conversations. The list title changes to match, for example Conversations with an open eval flag.
01 Conversations Tiles

The Eval flags tile selected, showing conversations with an open flag

2

Read the conversation list

Each row is one conversation. The columns are Customer, Conversation, Turns, Topics, Eval flags, Groundedness, Avg reply, Last turn and Review.
  • Eval flags shows Clean, or the names of the flags the conversation has, worst first.
  • Review shows Pending while a flag is open, and Reviewed once all its flags are resolved.
  • Use the All, Text, Audio, Image and Document boxes to pick the kind of message.
  • Search by name, number or conversation ID.
The list shows up to 100 conversations a page. Use the page numbers at the bottom to move on.
02 Conversation List

The conversation list with its column headings

3

Sort a column

Click any column heading to sort by it.
  1. First click sorts in the usual direction for that column. Names and IDs go from A to Z. Turns, Eval flags, Avg reply and Last turn show the biggest or newest first. Groundedness shows the lowest first.
  2. Second click reverses the order.
  3. Third click goes back to the original order.
An arrow next to the heading shows which column is sorted.
Sorting only reorders the conversations on the page you are looking at, not the other pages. When sorting by Eval flags, any High flag ranks above any number of Medium flags. Conversations with nothing to compare stay at the bottom.
4

Narrow the list with Filters

Click Filters above the list. A panel opens with six lists on the left:
  • Eval flag: pick flags by name. They are grouped as High, Medium and Low.
  • Severity: High, Medium or Low.
  • Status: Has an open flag or All flags resolved.
  • Topic: what the customer asked about. The 30 busiest topics are shown. Search to find any other.
  • Function: the actions the bot ran, for example a booking.
  • Transaction flag: the flags set on the customer’s lead in the Transactions Console. A conversation matches when its lead has that flag switched on right now.
Tick one or more values and click Apply. The number next to each value shows how many conversations in the date range have it.
03 Filters Eval Flag

Filtering by eval flag

04 Filters Transaction Flag

Filtering by transaction flag

The Filters button is only on the Conversations tab, not on the Overview tab.

​
How filters combine

  • Values in the same list: a conversation needs to match at least one of them.
  • Values in different lists: a conversation needs a match in every list.
The filters you chose show as chips under the list title. Click the cross on a chip to remove that filter, or click Clear all to remove them all. Clear inside the panel unticks everything before you apply.
05 Filter Chips

Filter chips under the list title

If a filter matches a very large number of conversations, the list shows “This filter matches more than 100,000 conversations in the period. Narrow the dates or add another filter.” Shorten the date range or add a second filter.

​
Resolve a flag

1

Open a conversation and a turn

Click a customer name to see the turns of that conversation, then click a turn to open it. The window that opens is called Inspect turn. It shows the reply time, the quality checks and the eval flags on that turn.
2

Resolve the flag

In Inspect turn, each eval flag is a chip showing its severity and name. Hover over a flag to see why it was raised.
  1. Once you have dealt with a flag, click Resolve on it. The chip fades and shows ✓ Resolved · Undo.
  2. If the turn has more than one open flag, click Resolve all (N) to resolve them together.
  3. Changed your mind? Click ✓ Resolved · Undo to open the flag again.
Your name is saved as the person who resolved the flag. The Eval flags tile only counts open flags, so the number goes down as you resolve them. A conversation shows Reviewed once all its flags are resolved.
06 Turn Resolve

One flag resolved, one still open

If Resolve fails, a red message appears under the flags. If it asks you to sign in again, sign in and try once more.

​
Flag stages

Severity is High, Medium or Low. Your institution decides how serious each flag is, and the colours are the same everywhere in the console.

​
If something looks wrong

Set Up Eval Flags

Add, change and switch off your institution’s eval flags.

Transactions Console

Work with the leads behind each conversation.