7 Critical Mistakes in Growth Experimentation

Experimentation is one of the prerequisites for growth. But it's also one of the hardest processes to get right. There's so much information about growth experiments out there – so talking about it in general terms doesn't work.

That's why we wanted to focus on the most common growth mistakes people make.

And our guest Heinz Grünwald did a great job at covering them all in a structured way.

Heinz has co-founded several companies in the e-commerce and SaaS space. He has coached and consulted early-stage startups on their growth strategies as part of the Swiss venture capital fund CV VC. As a Content Marketing Director at KAYAK.com, he executed one of the biggest content marketing efforts in Europe in terms of impact and budget.

As you can see, Heinz has worked on growth experimentation with numerous teams and shared the most common mistakes he's seen in his practice.

When to experiment?

"Growth experiments are at the core of growth. Growth experiments are a systemic way for testing a strategy to scale a business," Heinz started. This clearly explains why experimentation is usually at the top of the priority list for most startups.

And experimentation has the potential to help companies in different phases – no matter if it's an early-stage startup or a C-series company that wants to scale. It helps teams bake growth into the product itself. And it gets them to "focus on the creation of compounding growth loops that allow your business to lower the time to ROI and therefore to speed up growth."

But Heinz is quick to admit that if you still haven't found product-market fit, "you first need to focus on the validation of the product and the market."

At this stage, you probably have a small user base and almost no traffic. “You don't need to think about growth," Heinz concedes but quickly adds: "You still need to think about experimentation." It's just that you'll be relying on qualitative feedback and validation rather than large-scale growth experiments.

Most common experimentation mistakes

Heinz and I spent most of our time discussing the critical experimentation mistakes startups make. And these are the seven he wanted us to focus on.

Mistake 1: Starting too early

There are a few different reasons why starting early can be detrimental to experimentation: "If you start too early with growth experiments, you will not have enough data, [the tests] will be inconclusive. You will waste all those resources."

But on top of not having enough data, there might be organizational issues, too: "The organization and the team have to be ready for it. [...] The whole team needs to get used to this process. So it's a matter of the product has to be ready, and the team has to be ready." This all ties nicely to mistake number five that we'll cover further down.

Mistake 2: Expecting an impact right from the beginning

According to Heinz, there are very few cases when the experiments can bring substantial results from day one. This happens with experienced growth teams that have worked together before. And almost no one else.

But in all other cases, "you need to invest in the team set up, the education, the infrastructure, to run the proper tests." This means getting the right people involved, figuring out your tool stack, your measurement protocols...

Once all of this is done, Heinz advises conducting an A/A test. To do that, you need to launch two identical versions of your control. This is a way to test the infrastructure, process, and tracking.

"The pressure [of getting ROI immediately] could lead you to hack together things very fast. And then you have no proper process for the real stuff in the future." Getting things right is more important than doing them quickly, at least at the start.

Mistake 3: Lack of data collection

When you finally start experimenting, you can get in a world of trouble if you don't have the data set correctly.

This includes two key elements.

First off, making sure you're tracking everything properly. "It can be quite complex," Heinz says. But you need to figure out what sources of information you'll be using and how the data you're getting out of them can fit together.

The second element is an experimentation repository where you collect the data and the conclusions you made. There are some complex test documentation tools out there, but Heinz is partial to simple solutions: "There are always special tools for everything, right? But in general, a Google doc or a spreadsheet, or even a slide deck, is enough."

Mistake 4: Creating experiments with the wrong scope

Some experiments are simple, while others are more complex. But in any case, you need to try to arrive at a "minimum viable experiment" – an experiment design that's going to bring you results quickly but doesn't require excessive resources to set up or run.

Quite often, the main issue is defining the right scope for this experiment. "When an experiment is too big, then you're wasting unnecessary resources [...], or it takes longer than needed. And also, the more complexity you add, the harder it is to analyze."

So we should try to keep experiments smaller – especially at the start. But there can be issues with that approach, too: "If the change is too small, if you see no impact at all, obviously, you also waste your time." This is especially true for minor design experiments, like changing button color or the CTA position.

A good way to assess an experiment's viability is to first do a test run. Run it with your team or another easily accessible small audience. "You just collect a few hours of data or a day. Is this data you tracked of any kind of value?" By checking that up at the start, you invest one day to improve the experiment. But the improvement can be substantial.

Mistake 5: No growth mindset within the team

"It's important to bring everyone at the same level of understanding what growth is across the company. Even the people that are not involved in the growth process per se still need to understand what this team is doing."

Of course, this can happen gradually. But sometimes, the best insights can come from people who are not involved in the experimentation process directly. A customer success rep who understands what users are struggling with can provide an insight into their behavior.

The people on the growth team need a lot of education and alignment to reach their full potential. "They need to align on the process and the practices, even the terminology they use. Because there are several approaches to growth you could follow. [...] In general, there's a lot of commonality across these different approaches. But you just need to agree within your team that you're talking about the same things."

This can be especially crucial for experienced growth marketers who are working together for the first time. Each one will come in with their set of practices or their own approach to prioritizing ideas. And while there's no one right way of doing experimentation, the right thing to do is just decide what to stick to.

Mistake 6: Overlooking data

"You have to analyze the data in the right context. And that's not always that obvious," Heinz explains. It's about looking at the metrics directly affected by the experiment and the other metrics linked to that.

Heinz gave an example of a Facebook ad approach they tested with his team at KAYAK. The big change made all engagement metrics plummet down straight away. "Then, after three days, we analyze the numbers, and they really all stayed red. Just one number turned quite impressively green. And that was the return on ad spend. Which is, by the way, the only number we cared about." It turned out the change impacted the audience quality, and it was for the better.

Make sure you look at the big picture and analyze all the data, especially the elements that will provide additional context to your interpretation.

Mistake 7: Being afraid of failed experiments

The last mistake came up while we were discussing the difference between failed experiments and experimentation mistakes. These two are not the same.

"It's pretty hard to not have failed experiments," Heinz says with a smile. And they are not something to be avoided at all costs.

Failed experiments are much better than inconclusive ones. It's because they don't teach us anything. "The worst experiment is an experiment that has a neutral outcome. If it's a positive outcome, and you improve something in the business, great. If it's a negative outcome, and you really know why it was negative, great, you have a learning that you might need in another experiment, or you create a new one out of the learning."

So failed experiments are part of the game, and you shouldn't avoid high-risk experiments because you're afraid they might fail.

Instead, invest in them. They are high-risk, but they also might be high-reward.

Heinz prefers to hedge his bets and add high-risk experiments to his list in a calculated way: "It's like an investment portfolio. You have a few [planned experiments that are] sure – that's some security. But you need to also have some higher risk in there. When you evaluate the experiments, you also need to factor this into planning."

Look at the quarterly goals you want to hit. Plan most of your experiments to be high-confidence ones that can take you to 75% of your goal. And then sprinkle in some outlier ideas – most of them will fail, but if just one pans out, you'll hit it out of the ballpark.

The biggest mistake is not to experiment at all

This discussion was not designed to dissuade you from experimenting. Start, and you'll quickly start improving. So gather some of your teammates and plan an experiment you can start this week!