Review Conversations

Taking time to read through the conversations that users have had with your assistant is the best way to understand how users are interacting with your assistant. Too often, teams developing assistants get caught up in watching certain metrics (e.g. “what percent of users express intent X?”) to see the big picture.

Reading through real user conversations has benefits at every stage of a project. These include:

• Adding NLU data that reflects how real users talk (you’ll be surprised!)

• Discovering and prioritizing new goals that users expect to accomplish with the help of the assistant

• Finding new ways users deviate from the happy path and adding stories to handle them

Rasa X was built first and foremost to be a tool to help you learn from real conversations. The Conversations page has many different features to help you understand your users and figure out what to do next.

Conversations Inbox#

In the conversations inbox, you can review all of the conversations your assistant has had with users. This includes conversations from all external channels and from the Share your Bot link and the Talk to your Bot screens.

To best understand how your users interact with the assistant, take some time to read different conversations.

In any conversation, you can view the messages the assistant and user has sent, as well as the predicted intents and actions and their confidences. For more information, click on a message or action. You’ll see what the generated story up until that point looks like, as well as the values of any slots filled at that point in the conversation.

Filtering Conversations#

The conversations inbox provides a number of different filters to help you find interesting conversations. Filters are available in the top left corner of the screen. You can filter by conversation length, channel, confidence, actions, intents, slots, and review status. For more information about using filters, see Finding Relevant Conversations.

Tracking Review Status#

You can triage conversations as you go through the inbox by clicking the Mark as reviewed, Save for later, or Delete icons. By default, the conversations inbox will show all of your conversations. You can use the ‘Review status’ filter to restrict the view to reviewed, saved, or unread conversations.

Using Conversation Tags#

Conversation tags are a great way to track the process of reviewing conversations and classify conversations. Click the gear icon on any conversation to create a new tag or select one you’ve already created.

When working with a team of reviewers, it’s often helpful to have an idea of who is assigned to which conversations, to avoid re-doing other team members’ work. You could use conversation tags to achieve this by tagging conversations via the API to distribute them to different reviewers with name tags.

In addition to keeping track of the review process, you can use tags to keep track of success and failure, or any other information about a conversation you want to store and filter on later. For example:

• General Success or Failure tags for when conversations go well or not

• Conversation tags for different goals users try to accomplish

Tags give your team a shared way to categorize conversations, which gives you a common language to discuss what works and what could be improved.

Flagging Messages#

When reviewing conversations, you’ll undoubtedly come across conversation turns that have gone wrong.

When you come across these, flagging messages is a good way to keep track of issues without interrupting your reviewing flow. If you’re tagging conversations, flags can also be a way to provide more specificity to what behavior led to a certain tag.

Each message in a conversation has a flag next to it when you hover near it. Clicking this flag will turn a message red. In addition, the conversation in the list on the left will also receive a flag.

You can use flags to indicate whatever you like, but it’s best to agree as a team on what they mean. You could use one or more of the following ideas, or come up with your own:

• Flag messages where the bot selected the wrong action, despite a correct NLU prediction

• Flag user inputs that the bot can’t currently answer, but should in the future

Once you get to fixing problems, flags make it easy to find which conversations have problematic behavior, and which messages within a conversation are causing issues.

Sharing Messages#

Sometimes instead of flagging a message to come back to, you’ll want to share a message with someone else from your team. This is common in a few different situations:

• You notice a bug in one of your actions, and want to open an issue or ticket in your repository

• You want to ask your colleague if they think a certain behavior is desirable

• You find something funny you want to share with your team

When you select a message (by clicking on it), that selection is reflected in the URL. To point someone on your team to a specific message, select it and copy the URL from your browser’s toolbar.