notice
This is documentation for Rasa X Documentation v0.41.x, which is no longer actively maintained.
For up-to-date documentation, see the latest version (1.1.x).
Test Your Assistant
Tests are key to building reliable software. As you continue to improve your assistant, having a comprehensive set of tests ensures that you aren’t introducing regressions. Rasa X makes it easy to turn successful conversations into tests.
Why Test?
Professional teams don’t ship applications without tests. When you change your code or training data, you want to always answer two questions:
Do all of the most important conversations still work correctly?
Did my changes improve the ability of my model to generalize to messages and conversations it hasn’t seen?
When you start building your assistant, it is straightforward to talk to your assistant to test that a change works. However, as your assistant becomes more capable and complex, you end up either having the same conversation with your assistant over and over again to test it, or even worse: you stop testing parts of your assistant and have no idea if they still work.
Instead of manually trying out conversations, you should use tests to ensure that your assistant always behaves like you expect during every step of your development.
When to Create Tests
When you start to write stories, you should also start creating test stories. These example conversations test for correct NLU and dialogue management predictions.
As you have conversations with your assistant in Rasa X, you can save them as tests with the click of a button. Once you share your assistant with others, you can also save their successful interactions as tests in Rasa X.
By the time you go into production, you should have dozens of test stories covering the most important conversation flows.
How to Create Tests
Talk to Your Bot
When you are trying out a change in Rasa X and it works as expected, take that extra step and click the Save test story button to create a test that will make sure you always check that your assistant works like this in the future.
Conversations
You can save real conversations as tests in Rasa X, so successful conversations can become tests right away. When you review a successful conversation, click the Save test conversation button to create a test that checks that your assistant can still handle this conversation correctly in the future.
note
When you save test stories from Rasa X, you won’t be able to see them in the
UI. However, you can see them in the /tests directory of your project. If you add
your changes using Integrated Version Control, you will be able to see them in the
target branch on your remote Git repository.
When to Run Tests
Every time you train a model, the very next thing you should do is test it. This is easy to do using rasa test in the Rasa Open Source CLI:
Once you have Rasa X deployed to a server, it is worth investing in a Continuous Integration (CI) pipeline to automate your testing. Integrated Version Control in Rasa X can trigger tests in a CI pipeline that automatically run whenever you push changes.
For more information on continuous integration and deployment that you can run whenever you push changes from Rasa X, check out Setting up CI/CD.
What to Remember
If you want to be certain how your assistant will behave in a specific situation, you need to add a test for that scenario. As they say, untested code is broken code.