Testing and Optimizing Bots for Conversion

Overview: Learn how to improve your conversational marketing through testing with Mark Bryson consulting manager at Chatfunnels.

Transcript

Hey there and welcome to the demand summit. My name is Mark Bryson. I am the consulting manager here at ChatFunnels. And today I’ll be talking about testing and optimizing chatbots for conversion. So, what is demand generation in the context of conversational marketing? I checked Webster’s and there was a blank spot for that entry. So I had to come up with my own definition for that term. And it is demand generation is the process of attracting and getting potential clients to submit their contact information through chatbot conversation.   

In this presentation, I’m going to cover why A/B testing is important, what should be tested, best practices that you can implement and take home, and interpreting results. So why A/B testing? You must begin A/B testing without any assumptions. Just because an opening message with a value prop worked for the startup next door doesn’t mean it will work for you as well. Here at ChatFunnels, we have a mantra of ‘stop guessing and start testing’. It may very well be the case that the value prop converts better than any other opening message but implementing it without A/B testing isn’t the way to go about it. Once you understand this simple framework, you can adopt a systemic approach to testing instead of guesswork. Discover what engages your audience, quickly identify what works best, and then reap the rewards of higher conversion.  

What should be tested? You could test parameters like the opening message, which can include CTAs, such as the value prop or a solution to a pain point, the prompt size, like a mini prompt, or maybe something that takes up half the page and grabs more attention. Triggers such as time on a page, scroll percentage, question order, this can be a really helpful one to determine at what point customers are comfortable sharing their contact information. Live Chat routing, many verses few answer options, tone, which can be fun verse professional, qualifying question types, like how big is your organization? What is your role? What CRM do you use?  

There are many other parameters that you could try messing around with such as the colors, the avatars, and even little things like emoticons can set the tone for how you’re interacting with your customers. And you can eventually impact the likelihood of them sharing their contact information with you. Now let’s get into the best practices.   

First off, you’ve got to define your goal. Before you start testing, you need to figure out what matters most. Is it increased engagement, more live chat, booking demos, better FAQ support, or lead qualification? Testing without goal setting is a pathway to nowhere. You need to have a roadmap of what you’re trying to accomplish, and then set your goals and define your tests accordingly. And then consider where to start the test. Usually pages with high web traffic or good starting points as well as high value bots, which have purposes of leading to prioritize goals or bots on high intent pages, such as the pricing page.   

Now let’s jump into the ChatFunnels’ bot testing platform and see what it looks like to set up a bot test. So, we’ll start by creating a new test, giving it a name and start selecting our parameters. So, we’ll select a bot. This is where you can make your changes to the CTA or any other aspects such as routing. Select a success metric. So once again defining what you’re going to test on and what’s going to be considered a success is critical. We’ll go with emails captured, followed by selecting a duration. So, you can do a time based or an automatic. Time base can be something based on seasonality or some other marketing campaign. Or you can do automatic which automatically applies statistical significance to the conclusion to determine the conclusion of the test. 

 I usually recommend doing this. Choosing a start date. And then you can have the bot software automatically promote the winner of the test. So, we have original bot which is concierge bot A versus the modified bot concierge bot verse B and then go ahead and fill out the description of the purpose of the test. And you’re ready to go.  

So once this is live, you can monitor the progress by clicking on the view test report here. And since we just created this test, we don’t have the statistics yet, but I’m going to switch over to a page to see what it looks like to view an ongoing experiment. So here we see the dashboard of a live experiment where you can track the progress of which bot is performing better and keep track of what’s going on in the test. It shows bot prompts, conversations, meetings booked, primary goals, secondary goals, with your key metrics. And this allows you to decide if you want to end the test early and go ahead and switch over to whichever bot you feel is performing best. Or you can wait it out and make sure you attain statistical significance before making any changes. 

 To conclude your experiment, interpreting results is critical to making the best decision. Of course, statistical significance is the best way to ensure you’re making the right decision. And there’s multiple ways to do this. I showed you one in the ChatFunnels‘ product that takes care of it for you automatically. If you don’t have a product in your chat bot software to do this, you can go to abtestcalculator.com. And that’s where this screenshot has come from. You can manually enter the number of instances for variation B as well as the conversions for each one. And it will provide you with a calculation that’s really handy and quick to determine at what point you’re at in just statistical significance.   

Again, making sure that you define your success based on the metric that you’re measuring such as gathering emails, booking meetings, or just simply having more conversations. And lastly is documenting and testing your results. This step often gets overlooked and can end up in a lot of wasted time, because you don’t have a roadmap for following through with your experiment. And also, you don’t have a way to record your findings in an organized way so that you can look back on them and apply the findings to your chat bots.  

So, I’m going to take a minute and show you one way to document your test results and your experiment outline. So, this is our experiment documentation that we use here at ChatFunnels. And if anyone ever wants a template copy of this, I’m happy to share it. This particular one was for a demo bot conversion experiment. So we have an overview of the purpose of the test, results at a glance, which we’ll fill in once we have finished the test, the launch date, the experiment tactics, bot A’s opening message, bot B’s opening message, and the purpose and reasoning why we came up with those messages. 

 If you don’t do this, it’s hard to remember, especially in an experiment that lasts maybe two to three months. What you were thinking and why you set it up that way what you were trying to measure. But by recording all of this, you can look back and make sure that you’re organized, and you know what’s going on with your experiment. Audience parameters, experiment constraints, these are important to determine. So again, it’s very easy to forget what settings you had with each experiment with each bot. And the purpose behind it if you don’t write it down. Defining primary metrics, again, very important. This is ultimately what you’re measuring and what you’re declaring success off of, secondary metrics. It also makes it a lot easier when the experiment concludes to go back. And quickly just fill in this template and that way you don’t miss anything.   

Here’s a nice little visualization of the funnel to see version A versus version B of these different measurements here such as conversations, emails captured, meeting booked, and you can swap these out for the things that you’re measuring. And then finally a space for final results where you can summarize your recommendations from what you found. And then another good idea is to include a flowchart of what you did. What version A was for the bot and what version B was. And that way you can replicate it or share it with other team members or organizations.  

Again, this is just a great way to stay organized. And make sure that you’re accomplishing what you set out to. Well, thank you all for joining me. It was great. I hope that you have great success in demand generation through conversational marketing. And I look forward to talking with you again and again. Don’t hesitate to reach out to me if you’d like a copy of any of these materials. Or to discuss ways to help your organization be more successful.