The Testing Methodology Mindset with Ben Ashworth

Overview

In his presentation, Ben shares the equation to test the methodology mindset. He shares how to apply and implement it in your company with his own examples.

Speaker

Ben Ashworth is Vice President of Marketing at Paytm, India’s leading mobile payments and financial services company. 

Quotes

“Let’s implement that testing methodology mindset to be able to continue to drive that revenue growth.” 

“That conversion rate allows you to take care of your revenue by implementing the testing methodology mindset.” 

Key Points

1. An equation to tackle the marketing ecosystem
2. The positive effects of implementing the testing methodology mindset
3. Taking input from the market and leveraging the data and then establishing new baselines in revenue growth

Transcript

All right. Happy Wednesday, all of you Demand Gen specialists. Super excited to be here with you today at the 2022 Spring Demand Gen Summit.

My name is Ben Ashworth. And I’m excited to present to you today, the testing methodology mindset. Now we don’t have a lot of time so I’m going to go ahead and just dive right into it.

Okay. So as digital marketers and or demand gen specialists we are asked all the time to increase volume, create efficiencies, improve quality, drive consideration, and capitalize on intent. This is a lot to take in. In fact, there’s tens, if not hundreds of different job titles that are put into the marketing ecosystem to tackle all of these things. But if I were to consolidate this down to one simple equation for the business, it would be traffic times your conversion rate times lifetime value, equals the revenue. Everything on the left side of the equation, all marketers play some sort of role here.

And today, I want to specifically look at conversion and how this can really be the special sauce to be able to drive that conversion growth or revenue growth for your business. So, if you even just move that dial a little bit on the left side of the equation with that conversion rate, it will have a huge impact on the right side of that equation with revenue. And this is what I would like to say is a way for you as a digital marketer to take control of your revenue growth by implementing this testing methodology mindset.

All right, so let’s dive into this a little bit. So, the conversion lifecycle may look very familiar to you. This is just a traditional lifecycle that you may see from a product or marketing campaign lifecycle. Or whatever it may be, but they all kind of follow this same methodology of there being an introduction phase, there being a growth stage, there being a maturity phase, and then there being a decline phase. Now as you implement the conversion lifecycle, and the testing methodology mindset itself. The idea is there’s always going to be new ideas, things that we could be able to implement, and we can help mitigate and continue to drive that out going into a decline state. So maybe it’s a marketing campaign that you are pushing or a specific email series that you have going out the door as a part of that marketing campaign.

Now, the idea might be you have a subject line that you’ve been using for some time, and you’ve established that baseline of what that open rate might be for that specific email. Well, why not go in and test a lot of different variations to see what is going to beat out that current baseline that you have? And so that variant then puts it into a new conversion lifecycle, going back into the introduction, the growth stage, and you’re inevitably going to hit that maturity stage. But as you continue to implement that testing methodology mindset, you will establish that new baseline, you’ll interview and introduce new variants, time and time again, that’s going to continue to allow you to grow those marketing campaigns up into the right. So, this actually reminds me very much so of a quote that I would submit is one of the most important to me as a digital marketer and or demand gen specialist and that is continuous improvement is better than delayed perfection.

This was said by Mark Twain, and I couldn’t agree with this more. A lot of times we are in our organizations. And we always kind of expect to delay projects and or to delay campaigns and or to delay X, Y and Z because it’s not perfect.

So, what does this quote even mean? Continuous improvement is better than delayed perfection?

Well, a lot of times, we have a lot of different KPIs that we are marching towards. But let’s just assume a couple of those major metrics that we’re looking at is our output of the work that we are producing and how quickly over time are we able to produce that output. Now this graph right here indicates that this is potentially a culture a company that looks to perfection in everything that they are doing, and they may be able to get 85% of the way there within just a couple of weeks, but maybe they spend the next two months to make it 5, 10% better before they decide to go to market. This is where an MVP or a minimum viable product is so pertinent, as a digital marketer, even with our campaigns, because if we just introduce new variants and continually improve, we will take advantage of all of that output during that time when we are trying to create something that was perfect where we wouldn’t have garnered anything.

And in fact, I would submit that the output over time will be significantly better than what you would have been able to conjure on your own goodwill with what you would consider as perfect in this given case because you are leveraging the market and the data did dictate the output rather than your own intuition and or subjectivity to dictate what that output might be. So what, essentially who does this really well, who is able to go into a conversion lifecycle and introduce new variants time and time again to continually improve their experience?

Let’s take a look at Apple back in 1997. Woohoo. Isn’t that just the prettiest site you have ever seen? Some of these ads are absolutely phenomenal. Introducing cyber drive registered today for a free CD ROM. I bet a lot of you folks don’t even know what a CD ROM is anymore these days. I can hardly remember what that is. So Apple, this is what they were back in 1997. If we take a look at how Apple has been able to essentially improve over time to get to where they are today. It’s leaps and bounds different. There’s not a lot of noise. They’ve changed the way that they’ve been able to lay out the sights very focused and very, very focused in terms of what they’re trying to drive as their value props and their call to actions and I would submit to you Apple didn’t just do this overnight. In fact, they continually make iterations all the time. I guarantee you’ve noticed this every single year, Apple comes out with a new phone, even if there is only a subtle change. They’re taking input from the market. They’re leveraging the data they know to make those subtle changes in introducing these new variants, then establish new baselines in the revenue growth. So I want you to take this into consideration as you’re doing your next marketing campaign.

So who doesn’t need this? Well, there’s a site called the San Francisco fog cam, or you can just go to fog cam.org. This is what it looks like back in 1994. They essentially, you know, give residents of San Francisco an idea of what the fog is going to be looking like in their city for that day. And this is what it looks like in 2022. So, it actually hasn’t changed at all. So maybe they just have the perfect site and they never have to change it. I don’t know. But for the rest of us, let’s implement that testing methodology mindset to be able to continue to drive that revenue growth.

So I wanted to share with you the six step process of the testing, quality mindset, okay, so it always starts with being able to measure if you cannot measure the quality of what your current campaign is, or your website conversion rate or whatever it may be, is your success metric. There is no reason.

Okay, the next step is to be able to analyze, evaluate the data. Allow the data to tell the narrative of what you should be doing next. Then strategize. Ask the right questions, create hypotheses, try to disprove those hypotheses by being able to design those tests and being able to implement those tests and making sure that the juice is truly worth the squeeze of you putting that time and energy and effort into it and then one of the most important crucial elements is to share those learnings amongst your entire organization. I can’t tell you how often I’ve run into these experiences of where I or those on my team have been chasing our tails, where we may have potentially introduced a new idea of something that we wanted to test. And it was tested maybe one two years ago. But we totally forgot the results of those tests and it wasn’t documented anywhere and those learnings were never shared. And so we ended up implementing that same test anyway, over and over and over again. And that and that essentially makes it so that we have wasted our time to that point, just to learn something that we’ve already learned historically in the past. So please, as you’re implementing this testing methodology mindset, it is pertinent to be able to share those learnings throughout your organization.

So, let’s go ahead and put these six steps into practice. So, I have a spreadsheet here that I put together just called the testing methodology framework, and please feel free if you’d like to get the link to the sheet. You can reach out to me after this presentation, to be able to learn more about it but the idea is this. It’s really kind of looking at immediate buckets that you want to be able to look at in terms of being able to set up a test, execute a test, find statistical significance, establish either a new variant or keep the baseline that you have, and then move on to the next thing.

So, the first piece is to go through what I call the testing methodology brief. You then want to be able to measure you have to be able to measure both the control i.e. your baseline, as well as the test so the variations that you introduce in that ecosystem results it’s absolutely pertinent to be able to report those results. And this is again, this is a great way of being able to disseminate this information throughout the organization as well. So let’s go ahead and let’s walk through one of these here really quick.

So first and foremost. Name your test, make the test name very simple or whatever that might be. So if it’s the subject line test for a specific email campaign, just know what that email campaign might be, and then put the test name in there, who’s the owner, who’s the team that’s going to be owning this right? A lot of the time for us it’ll be the marketing team. It can very much be the product team. It can be other teams associated within the organization that have proposed a test that we may want to put to practice. The next piece is to be able to just jot down the questions here that are prompting this test. So why are we losing so many customers that x spot in our acquisition funnel? Questions to answer for the customer? What do you do? What will I get? Why do I need it? Will it work? How do you do it? Is it worth it? Do I need it now? So those are things that you may be asking yourselves or saying hey, let’s actually create a narrative around how our customers might want to know why they need our product or service right now. Then generate a hypothesis, this is your traditional if then statement.

Okay, changing this element from blank to blank will increase or decrease a specific conversion metric and or success metric that you may be looking at. And this is honestly this is what you are trying to disprove as you implement this test and you put it in place.

So here’s a proposed test description. What is it that you are changing? Is it a redesign of an element on a page or is it the location of a specific element that you may have on a page? Is it the style? Is it the offer itself? Is it the call to action? Is it a copy, update and so on and so forth. As you can probably see, the furthest to the left. This is probably where you will see the biggest impact of a test we just fully revamp and redesign something that’s where you’re probably going to that’s probably where you’re going to see the most impact. Rather than just implementing some simple copy, update. Identify the page or the campaign or the creative asset that you’re looking to want to test and be able to document that here. Identify specifically what is the conversion metric, then I’ve added a whole list of selectors of different conversion metrics for my company, things that we look to as our core KPIs that we ultimately are driving towards, and then take a look at the confidence interval of the experiment. So, if it’s something that you want to just be able to run and get a quick answer to, maybe you only need 85% confidence of saying, Okay, this is probably directionally where we should go. I don’t recommend that always if it’s not where you want to garner statistical significance to get to that 95% confidence interval. So just identify what your confidence is in the test and I kind of broke down your belief for people within my organization, say my confidence is very low, it’s low, it might be high, medium or high.

Looking at the effort to implement, this is absolutely crucial, especially as you’re working with a product team. If it’s something where you’re trying to put together a full revamp of a particular, you know, element of your experience, then you definitely want to get the input of your product team to be able to do so. If it’s just you know, simple email updates, those types of things. Maybe it’s a lot less effort to be able to implement that maybe it’s an overhaul of your email where you do need to pull in you know, HTML resources to be able to help you with that and engineer and get their feedback of understanding what is the effort to to implement this and then identify Okay, for this particular campaign and or page like how many visitors a day is that currently getting? What is the current as well? What is the current conversion rate based on that specific success metric? And then identify, what do you expect by implementing this test that impact will be to that conversion rate that you have again, I kind of broke this out into different buckets, so between one to five, six to 10% 11, to 15%, and so on and so forth. This is a good way just to get that gut feel from leveraging this testing methodology framework itself to know whether or not it’s even worth it for you to be able to implement this test.

Now, if you want to dig deeper into this and more the mathematics behind this, I would recommend you look up MDE which is the minimal detectable effect, the minimum detectable effect. So as you implement this test, kind of what sort of impact you expect for that to have. And that’s definitely going to play a factor into your specific statistical significance that tests that you will be running as well as how long it’s going to take for you to run that test. Here’s an example of being able to show some of those data points as I’ve been running that test.

So we look at the control again, this is our baseline. What is the total number of visitors that may have been visiting that page, in this case? 12,000. How many of those visitors converted based off of the success metric that I had established here? In this case, it’s 275, which gives me a 2.29% conversion rate Okay, so we take a look at the test variation itself, how many visitors 11,500, how many conversions 325. We then see that conversion rate moving to 2.83%. Now, again, make sure that if you have reports, share those links of those reports. Make sure people internally can be able to see those. And then also, there are calculators out there. In fact, I can share those with you as well if you’d like to reach out of where it will actually give you an expected Test, test duration for what you’re implementing at this time and how long it’s going to take for you to be able to run that test.

Also, As you’re running this test there’s also calculators out there that will give you that will also tell you whether or not your test is statistically significant if there’s enough variation between your control and your test variation itself as to whether or not you have confidence as we kind of talked about earlier. Being able to get to that threshold of what I’d recommend if that 95% confidence. Of course, it’s always important just to say, Hey, this is the date, we’re going to start it so that way you know it’s going to take this amount of time for it to be completed. Identify whether or not you have statistical significance. You know, a lot of times I think one of the rules of thumb, I always say is, if you’re holding dynamite, get rid of it immediately. So, you don’t have to run it for the entire two weeks, if you just see a significant downturn and the variant that you’ve introduced, but now it may take a long time for you to be able to say it’s just statistically significant. Take a shorter amount of time, kind of based off of some of those variables. And again, I have calculators that I can share with all of you to do so. Identify what is the new baseline, is it what we currently have in the control or is it the new variation you’ve introduced? Identify that improvement and then whether or not that test is complete. This is a great way again to be able to share throughout your organization and get everybody into that testing methodology mindset even if it is only for your marketing team and or demand gen team that you are working with.

So going back to the presentation here as we are wrapping this up, so going back to this equation, traffic times conversion rate times lifetime value equals that revenue. That conversion rate allows you to take care of your revenue by implementing the testing methodology mindset. I want to instill that in all of you and I hope each of you are able to take this back to your teams and really start to implement this, and I wish you the best of luck in your demand gen growth.

Thank you very much. If you have any questions, you can find me on LinkedIn. Here is my link to contact me.

Thank you.