Skip to main
  1. Blog
  2. Admin role
  3. Copilot author role
  4. Copilot Studio
  5. Extensibility

Automate testing of your Power Virtual Agents chatbots with the PVA Test Framework sample solution

, Principal Program Manager, Power CAT, Friday, 11 August, 2023

When enterprise customers deploy large or complex chatbots using Power Virtual Agents, they need to ensure that the bots behave as intended.

This involves testing user utterances to confirm that the bot understands them correctly and triggers the appropriate topics. It also entails verifying that the full conversation leads to the intended outcome, such as resolution or escalation. Customers might also want to ensure that their chatbots meet quality requirements before deploying to production, which can be done automatically using Continuous Integration/Continuous Delivery (CI/CD) approaches like Azure DevOps.

The Power Virtual Agents teams have recently released new code samples to help automate chatbot tests, including testing utterances, validating topic triggering, intent candidates (multiple topics matched), playback of full conversation transcripts, and integration into a CI/CD pipeline.

What can be tested with the PVA Test Framework sample solution?

The PVA Test Framework is a sample solution available on GitHub that showcases how to execute tests against a Power Virtual Agents chatbot through APIs, using the Direct Line channel. It verifies that the bot performs as expected in various scenarios, such as:

  • Testing the Natural Language Understanding model (topic triggering)
  • Validating the multiple topics matched options (“did you mean…”)
  • Performing scale load testing
  • Testing full end-to-end conversations
  • Testing adaptive cards
  • Including a CI/CD test step in the deployment pipeline to prevent deployment if tests fail

How does the PVA Test Framework sample solution work?

The PVA Test Framework tool is a console application that can run tests against the Power Virtual Agents Direct Line API. It uses JSON as the format to play conversations back and verify outcomes, and it enables you to connect to Dataverse to download past conversation transcripts to run them again.

How to use the PVA Test Framework tool

There are different ways you can use the PVA Test Framework sample solution. The easiest one is to run it locally with the Command Prompt in Windows.

You first need to download or build the PVATestFramework.Console.exe file.
It then works as a console application, with command lines.

To run tests, you need conversation files in a .JSON format. You can either easily create these files from scratch, by writing your test cases in a .CHAT file format and converting them to .JSON, or you can download conversation transcripts from Dataverse.

For Natural Language Understanding testing, where you test many user utterances and verify that the bot answers as expected, it’s usually easier to start from your own set of user and bot utterances in a .CHAT format.

This command transforms a .CHAT file into a .JSON file:

.\PVATestFramework.exe convertChatFile --path test-set-sample.chat --outputFile test-set-sample.json

Here is what conversations look like a .CHAT format:

Each test conversation is separated by the <EOC> tag (end of conversation).
Messages following the user: tag are sent by the tool to the bot, and the messages received back from the bot are compared with the expected response, documented after the bot: tag, to determine if the test passes or fails.

user: Good morning
bot: Hello, how can I help you today?
<EOC>
user: When are you closed
bot: I'm happy to help with store hours.
<EOC>
user: Find me your nearest location
bot: I'd be glad to help find a store near you.
<EOC>
user: Where can I find your address?
bot: I'd be glad to help find a store near you.
<EOC>
user: I'd like to order a product
bot: I am happy to help you place your order.
<EOC>
user: what are you store hours and locations?
suggested: Lesson 2 - A simple topic with a condition and variable|Lesson 1 - A simple topic||None of these
bot: To clarify, did you mean:
user: Lesson 1 - A simple topic
bot: I'm happy to help with store hours.
<EOC>
user: What is weather today?
bot: I'm sorry, I'm not sure how to help with that. Can you try rephrasing?
<EOC>
user: What is weather today?
bot: I'm sorry, I'm not sure how to help with that. Can you try rephrasing?
<EOC>

Once converted to .JSON, you can run tests by using the below sample command. You can obtain the chatbot token endpoint from the Mobile app channel in Power Virtual Agents.

.\PVATestFramework.exe test --path test-set-sample.json --tokenEndpoint https://c53bf00279234d1cb5ae3265093d59.e1.environment.api.powerplatform.com/powervirtualagents/botsbyschema/cr507_testChatbot/directline/token?api-version=2022-03-01-preview --log --verbose

Test outcomes are then available in a CSV format that you can then use for further analysis.

Example analysis of the outputs of the PVA Test Framework:

You can learn more about the PVA Test Framework sample solution’s different settings and deployment options in the GitHub documentation.

Frequently Asked Questions

What is the cost of using the PVA Test Framework sample solution?

Conversations generated using the PVA Test Framework sample solution would typically generate billed sessions. See: Power Virtual Agents pricing.

How is the PVA Test Framework supported by Microsoft?

Although the underlying features and components used to build the PVA Test Framework are fully supported, the code itself represents a sample implementation of these features.

Our customers and community can use and customize these features to meet their requirements. Any issue with the PVA Test Framework sample solution should be raised on the Power Virtual Agents Samples GitHub repository and not through Microsoft Support.