A Culture of Testing: Establishing a Successful Team and A/B Testing

Experimentation and A/B testing are essential tools of the modern business world. They allow companies to make decisions based on actual data instead of opinions or assumptions (we discuss the challenge of dealing with HiPPO here).

But establishing an effective culture of testing goes well beyond simply performing a few tests. It involves the creation of dedicated teams, the establishment of solid processes and the commitment of the entire organization.

At Adviso, we did exactly this for one of our most loyal clients. This client now has over 635 tests under their belt. But Rome wasn’t built in a day.

To be successful, we needed to overcome many challenges, and in this article we’ll share some tips on how to build and cultivate a culture of testing at your company.

But before we begin… Is testing right for you?

*Disclaimer: This article is intended for leaders, managers and executives at digital organizations. If you’re an expert in digital and you’re looking for arguments that will inspire change “from the bottom up,” we’ve already tackled a number of issues in this area which we won’t be covering again here.

Determine whether testing is right for your company

Testing doesn’t work for every organization, but all orgs should at least consider testing. And remember that you don’t have to be a major player like MAMAA (Meta, Alphabet, Microsoft, Amazon and Apple) to benefit from this approach.

To determine whether a culture of experimentation is good for your company’s growth and its employees, I suggest answering these 5 simple questions. If your responses are all positive (✔️Yes), experimenting is definitely of benefit to you. 

If at least one of your answers was a ❌No, then a culture of testing, as embodied in current best practices, won’t be right for you. However, you can still use the concept to find inspiration for future innovations and improve internal processes.

If all of your answers were ✔️Yes, here are a few additional details:

—> If you have around 1,000 conversions per month, you can already consider implementing testing! Try making some “bold” changes (meaning tests in which many elements have changed compared to the original) on your main pages.

—> If you have around 10,000 conversions per month, you definitely need a culture of testing. You can perform some tests on all your conversion flows pretty extensively.

—> If you have over 100,000 conversions per month, I find it hard to believe you haven’t implemented testing yet. You need to talk to an expert, now. (Seriously, because you’re leaking money.)

Understanding the importance of testing

Before you can build a testing culture, it’s essential to understand why it’s so important.

Here’s a quick comparison of an organization that grew by implementing a testing culture vs. another without a testing culture (source).

Comparative chart: With and without a culture of experimentation

Recruiting and training team members

The success of your projects is based on three indispensable elements: a compelling vision, clear goals and a team (or individual) that is capable of delivering them.

As a leader, your role is to bring vision—to open doors, educate team members and supply needed resources. We’ll discuss this in more detail in the last section of this article about leadership.

Your vision should be translated into SMART (specific, measurable, assignable, realistic, time-bound) goals. To get started, I propose the following goal: Develop, launch and analyze two to three A/B tests within the next six months. This might seem easy, but if you’re just getting started with experimentation for the first time, you’ll need to install a new tool (like A/B Tasty, Convert or some other A/B testing tool) as well as perform quality assurance on your data to make sure it’s reliable. Consider this stage like preparing the soil before you plant your seeds.

Lastly, you need a solid team. It’s essential to recruit and train team members that possess the skills and qualities you need. This is the cornerstone of any successful testing initiative. At minimum you will need people with skills in development, design, marketing and analytics. The good news is that all of these skills can be provided by as few as two people.

One of these people will be a developer at an intermediate or higher level, with solid experience in front-end development, who is able to integrate the recommended changes and tests. This person is the one who ensures that what is put online makes sense and works as intended.

For the other team member, the ideal would be to hire an expert in CRO (conversion rate optimization) and testing. In simple terms, this individual would be a mix between a product manager, a researcher or analyst and a UX designer. This person will be solely responsible for managing all of the experimentation by performing research to produce hypotheses for testing, designing the variations to be integrated, analyzing the results and adjusting the road map following the obtained results. This role is for one of those jack-of-all-trades people that are becoming more and more available and accessible.

Once again, if you decide to go ahead, you can rely on internal resources, external resources or a mix of both. What really matters is that your team prioritizes actions and decisions that are based on data.

Establishing processes and standards

Once the team is in place, it’s time to set up your processes and standards to guide the experimentation.

Here is basically what you need to do:

Clarify your KPIs: The best way to guarantee that every experience contributes to the strategic goals of the company is to map your KPIs and how they relate to each other. This lets your teams see what metric they need to take action on to generate revenue. Below is a classic example of a goal tree for e-commerce.

Example of goal tree mapping for an e-commerce site

Evaluate and design the right tests: Designing robust tests requires detailed planning, from test design through to data collection and analysis. First, your teams need to determine if their hypothesis is testable (if there is enough traffic to the page to obtain significant results) and estimate potential revenue. This allows you to confirm in advance whether the resources needed to test the hypothesis are worth investing. Once the hypothesis has been validated, your teams should move on to the test design phase, which involves creating a document that summarizes the reasons for the test, how the test can be configured on the testing platform and what to integrate into the site.

Record what you learned and iterate: Lastly, it’s essential to rigorously document the results. Test after test, you will gradually amass a knowledge base and wealth of history. This will facilitate capitalizing on your learnings and making informed decisions in the future. This is best handled directly within your IT systems to promote collaboration between developers (ex., Jira). But you could also create your own system using solutions that are either free (like Google Docs) or low-cost (like AirTable). If you’re looking for free templates, take a look at Reforge.com.

Spread the word: For your culture of testing to take hold, it needs to break out of the silo that is your experimentation team. Remember that experimentation is above all a way of deducing and quantifying the effects of your actions on your business revenue. What you will find is that by making your results public and talking about them, you’ll get more and more attention from other people, whether VPs or employees, who are looking at your new way of approaching growth. There’s nothing like a simple slide to show how you generated over $50K in one month with just a single test (I’ll let you do the math for the annual amount of revenue).

P.S. If this figure seems big, I can assure you that it’s an extremely conservative number for the average e-commerce site. But you can always get in touch with one of our experts to evaluate what the potential would be for your digital platform.

Show leadership and engagement

“Rather than trying to prevent all errors, we should assume, as is almost always the case, that our people’s intentions are good and that they want to solve problems. Give them responsibility, let the mistakes happen, and let people fix them. If there is fear, there is a reason—our job is to find the reason and to remedy it. Management’s job is not to prevent risk but to build the ability to recover.”

These are the words of former Pixar CEO Ed Catmull, taken from his book Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration. As he says, leadership is about creating an environment where mistakes play a role, which is also a fundamental aspect of organizations that adopt a culture of experimentation.

Senior managers have an obligation to play an active role in transitioning to a culture of experimentation.

Of course, this also involves allocating the resources needed to implement such a culture.

And of course, this also requires leaders to demonstrate their commitment to this approach.

But if there’s just one thing you take away from this article, it’s that you also need to foster a culture of tolerance towards failure.

Let me explain.

Experimenting means, by definition, that tests won’t always be successful. And that’s normal, in fact it’s even healthy! If you set up a culture of experimentation and you notice that one of your experts heading up the tests only ever “wins,” that’s probably because they are not testing the right things (or worse, they aren’t correctly analyzing the results).

As a leader, you want to create an environment where employees have room to innovate and learn from their failures. As in life, mistakes are opportunities for learning and growth.

Conclusion 

To sum up, setting up a culture of experimentation is a demanding process, especially at the beginning, but it’s also gratifying. It can fundamentally transform the way in which a company makes decisions and obtains value from its data.

By establishing a culture of experimentation, companies can equip themselves with true superpowers, such as:

✔ Understanding the tangible effects of their actions. Develop a heightened ability to determine the cause-and-effect relationship on company revenue. By better understanding the impact of your actions, teams can orient their efforts towards activities that generate more value.

✔ Eliminating discussions that bring no added value. Reduce the number of useless meetings and discussions by employing tangible data. Good data can end a conversion and allow faster, more informed decision making.

✔ Increasing motivation. Increase each team member’s sense of ownership as well as their motivation, particularly by being able to directly observe the results of their work. This strengthens the sense of being part of the company and the engagement of employees towards the achievement of organizational goals.

✔ Improving continually. Be able to learn from your wins and losses in a significant way. Experimentation lets you draw important lessons on what works for your prospects and users in order to direct your energy towards working on more effective solutions.

As shown in this article, adopting a testing approach isn’t easy. Organizations need to invest in their technology and human resources. Testing tools can represent a significant monthly cost. You also need to manage your data, and this requires introducing your employees to an approach that is focused on both data and users. But despite the work involved, the potential advantages of this approach strongly outweigh any risks in most cases.

To estimate what you could gain by adopting or optimizing your culture of experimentation, don’t hesitate to discuss your situation with our experts.

Let’s get testing!