New Testing Feature: ChatFunnels Experiment

ChatFunnels recently launched a new software feature called the ChatFunnels Experiment.

This testing platform allows you to make data-driven decisions and provide value to your organization.

Testing your chatbots is crucial in order to drive demand. 

In the recent Demand Gen Summit, Mark Bryson, the consulting manager here at ChatFunnels, showcased the ChatFunnels Experiment.

This software runs A/B tests on your bots, so you can stop guessing and start testing!

Why A/B Testing?

A/B testing, also known as split testing, is the process of comparing two variations of bots to determine which one drives better results.

Half of the website visitors will be presented with version A of the bot and the other half with version B. You set a custom timeframe and then the software tracks which version yielded higher results when the test concludes.

Within our product, you can also run multi-variate tests. 

The performance of the bot depends on the key metric you choose to test.

Key Metrics

Our testing platform will measure five key metrics:  

• emails captured 
• meetings booked 
• primary goals reached 
• secondary goals reached 
• routed to live chat 

You need to choose the key metric that will determine which bot will be declared the winner.

Throughout the test, you and your team can stay updated on how the bots are performing.

Let’s break down the best practices in using the ChatFunnels Experiment testing platform.

1. Define Your Goal

Before iterating your bot and starting a test, you need to define what the goal of that bot is.

Chatbots are here to make your life easier and create a smoother customer journey. Consider how you want your bot to do that.

Is it meant to increase engagement or live chat? Should it be booking more demos? Does this bot serve as more FAQ support? Maybe it’s meant to aid in lead qualification?

You must test with a goal in mind.

2. Decide Where to Start the Test

The best practices when choosing where to perform the test is to consider the following:

• pages with high web traffic
• high value bots
• bots on high intent pages (such as pricing page)

Those type of bots are what really matter to accomplish the goal you previously set.

What Should You Test?

There are many aspects of the bots that you can tweak and improve with the hopes of driving better results, but what should you test?

Here are our recommendations:

• opening message
• prompt size
• triggers (such as time on page, scroll percentage, etc.)
• question order
• live chat routing
• many vs. few answer options
• tone (casual vs. professional)
• qualifying question types (such as “How big is your organization?” “What is your role?” “What CRM do you use?”)
• colors, emojis, avatars

3. Set Up the Test

Log into your ChatFunnels account and navigate to the testing platform. Click on ‘New Test.’ Then name your test and select a bot to get started.

Then you can iterate the bot to create the second version that will be tested against the original.

It will then prompt you to select a success metric and a duration for the test, which can be time based or automatic. You can also select whether the winning bot will be promoted manually or automatically upon conclusion of the test.

You then will fill out a description of the test, which is crucial for the organization of future tests and reporting.

Then the test is ready to go!

Throughout the test, you can monitor the progress by clicking on the view test report.

4. Interpret and Document Results

The winner will be declared based off statistical significance, which ensures that the higher performing bot is the one that wins.

Helpful Tip

It’s important to write up an experiment outline. Months after conducting the test, chances are you’re not going to remember in detail the purpose of the test, what exactly was tested, and what the results were.

Documenting test results will help you stay organized.

We also recommend including a flow chart of both version A and version B, which again, helps you analyze what worked and what didn’t.

For a more in depth explanation of how to set up your experiment outline, check out Mark Bryson’s presentation from the Demand Gen Summit.

Conclusion

This new testing platform, The ChatFunnels Experiment, is quite the game changer for our product. Now you can stop guessing, and start testing!

Iterate your bots and let the tests determine which version yields higher results!