Simplifying QA: Reducing Conversation Testing Time by 60%

Struggling with time-consuming bot testing? Let me take you through the redesign of Yellow.ai’s testing process, where we automated the creation and testing of test cases, significantly speeding up bot deployment and boosting company’s efficiency.

Imagine a world where conversation testing is effortless. We turned a tedious manual process into seamless automation, diving deep into language intricacies, user and business needs. The result? A design so intuitive it lets teams focus on what they do best — creating amazing conversations.

If you are in a hurry and cannot read the whole case study, you can check the final output here:

Final output

Before we dive into the details of this project, let me set the stage with some background on the company, the team, and the project itself.

About the company

Yellow AI helps companies deliver 24/7 customer support, automate employee interactions, and streamline marketing efforts through personalized campaigns, all while saving time and cost.

About the team

I took on the role of the sole designer working on this project, driving the design process from concept to completion. I collaborated with multiple stakeholders in the process : 

Product managers : Collaborated to understand nuances, ideate, test, and refine.

Other designers : Sought feedback to align designs with best practices.

Developers : Integrated technical constraints from the start to ensure feasibility.

What, where and how?

Even if you’re new to this , don’t worry — I’ll walk you through what conversation testing is, how it fits into bot development, and why it’s essential for delivering a flawless product. The illustration below depicts our process, starting from when we receive client requirements through to the final hand-off. It also highlights the manual testing phase, which we’re actively working to automate for greater efficiency.

This issue was significant enough to catch the attention of the product manager, who recognized its impact on the business. The management team then prioritized finding a solution to address these inefficiencies and improve the overall process.

Increase human errors ⚠️

Delayed handoffs 🕥

Loss of revenue 📉

Let me take you through the process of how did I approach this problem.

Few nuances to be familiar with

I’ll take you through the basic nuances that you need to be familiar with for this case study

Utterances : These are the inputs a customer types into the chatbot. The developer selects a pre-made response for each of these inputs.

Instances : A collection of related utterances grouped together, identified by a specific hashtag.

Test flows : The journey of a particular scenario, detailing how the entire flow progresses from the initial utterance to the final outcome.

Design mode : This is a unit where the Business analysts design the flows as per client requirements

Studio module : This is a unit where developers take flows from design mode and build then

Testing the written utterances is in the scope. Now, we will ideate on how to do that.

Me and the PM started ideating on this. We had a couple of ideas on how to group the utterances

Grouping the utterances in a file format in which the user can see all the main utterances and when expanded can see the sub-utterances. They can add the main utterance/ sub-utterance through the buttons

Idea 2

Since utterances are going to be so many, we can use instances as an identifier and we can try not to fit everything into one group. So, I have come up with an idea of introducing a sidebar

The testers need to write each and every utterance which is also too much work. So, we had discussions with devs if we can automate that also and yes, we can do that. Now we started ideating on that

Ideation

We started ideating accordingly and the below are a couple of ideas which have been designed. These proper ideas can only be explored if designed

Idea 1 :

Have it in the design mode and have users test the flows by typing in the chatbot only. I made a quick demo for this idea as many people wanted to explore this. Below is a demo of this idea :

Demo of the idea

In this, we imported flows as we wanted to test them out by typing. So, flows are imported here unlike the above options

But what happened to this idea?

In this, running everything manually is difficult

And to test all cases at once, visualizing that, it’s a lot of cognitive load on the user

This also creates confusion amongst the flows created during BRD and the flows tested. 

Final solution

With all the problems faced before, we wanted a solution which will solve for all of those. Just rewinding the problems :

Missing out on important actions

Excessive cognitive load

Here’s the proposed solution giving the user option of intents/ utterances:

In this, we have given option for importing intents which are already present which is a new way of doing this (or) they can upload the utterances file which they have written and they can add on to that. Introducing this in the first version helps users not totally go away from their habit (or) they will not be forced to do something new in a doubtful manner. So, we’re getting them used to this slowly.

We have tested this with 15 users internally who are currently testers in the organization.

Mode of testing : Online

Process of testing : Some tasks were given and we have observed the users nudging them with questions/ help needed

Metrics 

Following are the metrics used for measuring the output of design :

Time on task

System usability scale

Average testing time has decreased from 35 days to 12 days

Number of bots that went to this page are 102

I had to better understand all the terms and start with this project. So, it’s good learnings in terms of getting to know the workflow and understanding of the whole terminology fast and implementing it

Since this is B2B, I thought there’s less exploration in terms of delightful UI but that can be improved. Delightful UI and interactions can be worked upon

I learnt better collaboration.