CRO is a Process, Not an Event
Do you know how the term “conversion rate” came to be? Like most things, it has an origin.
Bryan and Jeffery Eisenberg coined the term “conversion rate optimisation” in 1998, partly by serendipity, partly through intent.
Back in those days, it was a pre-problem, and so the category didn’t exist. The term “conversion rate” didn’t even exist, let alone optimise something that was non-existent. In fact, the only reason the search term even remotely appeared in Google was because people were searching for currency conversion rate, not website conversion rate.
“We knew conversion rate from sales was an important metric. We were always doing optimization and search optimization was growing as well. Conversion rate was a ranking term so we combined the two. We were writing about this and similar terms for years before anyone else was even talking about it” said Bryan, and so, they chose to rank for this - “conversion rate optimisation”. But, wrongly, with a Z instead of an S. Not using proper Queens English, I see.
Some say creating this term was nothing more than gaming the Google algorithm designed to create a business. But they were, rightly, trying to assimilate a term that described their efforts to grow businesses and support user experiences while doing so. Yet, their intent had an unexpected outcome.
Despite Bryan always citing that “conversion is a process, not an event,” the term conversion rate optimisation has since stuck. It has become popular amongst brands for its outcome and simplicity—too simple perhaps. It’s not just about optimising a single conversion rate, but the process of improving customer experience iteratively, ideally with validation.
CRO is an Event, Not a Process
Yet, that’s not how people see it. Society sees the event, not the process. They see the 2% that converts, not the 98% that doesn’t. If you optimise the conversion rate, you inherently ignore the needs of the majority.
Is there a blame here? Most tend to lie fault with the definition, but I blame the metric. Seeing conversion rate as a metric or a measure of success is why it has failed us.
Fast-forward fifteen years from Bryan and Jeffery’s digital Frankenstein. In 2013, the confusion of optimising a conversion rate was evangelised further. on why conversion rate is a—quote—“horrible” metric to focus on. Horrible is an intense term, Dan.
The reasons are obvious: it’s aggregated, not controllable, and does not reflect true performance.
That won’t stop the blog posts from peddling conversion rate as an event with the likes of “What is the average conversion rate?” - the answer being 1.4% for all Shopify stores apparently []. Or “What is a good conversion rate? Lo and behold, it’s above 3.3%. Such irreverent content that lacks context; trying to aggregate a measure that varys from product, to device, to source, to demographic, to gender, to weather, to pay day to hour per day.
While conversion rate optimisation might be seen as an event, if you asked any conversion rate optimiser, no one will say they optimise conversion rates. They will talk about the process not the measure of success.
This is because today, experts and practitioners know that conversion rate optimisation involves what Bryan and Jeffery designed it to be, unfortunately, warped by the simplicity of society. For years, there have been movements in the CRO industry to reinvent the term. I’ve heard it all, from CX (customer experience), to GO (growth optimisation) to BGO (business growth optimisation). These calls are cries for help and this has been going on at conferences, in blogs, and podcasts for a long time yet never quite landed.
The common denominator? The removal of the term “conversion rate” and the disassociation of the event from the process.
Beyond Conversion Rates: Fostering Genuine Customer Experience
Conversion rate optimisation fails us because it focuses on a metric—one that is, as Dan described it all those years ago, an aggregated, macro, uncontrollable, retrospective, binary measure of success that can easily be gamed.
Sure, we can increase a conversion rate - we could:
- just reduce all prices by 10% or give free shipping for our first 5 orders. Done.
- stop sending paid traffic to the website so that only those who know about us will visit. Completed.
- only send desktop traffic to the site, never mobile. Obviously.
- prioritise search results for our cheapest items. Next challenge.
See? Easy.
But can you grow something more meaningful? Something related to average order value? Dare I say, to customer satisfaction? To returning? To loyalty? Something that moves away from being a business KPI, a target to work towards, and more towards a genuine reflection of good user experience? This is what conversion rate optimisation should be.
The Cobra Effect
Pitfalls of Conversion Rate Targeting: Lessons from Goodhart's Law
Instead of what it should be, and what is seen by experts all around the world, conversion rate optimisation’s success has morphed it into something else. It’s moved down the dwindling haunted road from a process to an event to a target.
When a measure becomes a target, it ceases to be a good measure
This is Goodhart’s Law.
It was a term coined in 1975 by none other than Mr. Goodhart suggesting that “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes” []. Fun fact: it was later built upon to inform AI development in the Machine Intelligence Research Institute by a certain David Manheim. No relation [].
This law is everywhere. In politics, if success is measured by approval ratings, it becomes a popularity contest rather than driving meaningful results. In GP patient admissions, success is determined by the number of appointments completed in a day; the quality of patient care might decrease as a result.
Yet there is no better reflection than the anecdotal and questionable validity of the “Cobra Effect” story.
Under British rule, the government was concerned about the number of venomous cobras in Delhi. So, they offered a bounty for every dead cobra, where large numbers of snakes were killed for the reward. Eventually, however, enterprising people began to breed cobras for the income. When the reward program was scrapped, cobra breeders set their now-worthless snakes free, the wild cobra population further increased [].
Conversion rates are our cobras. Where, instead of cobra-breeders, we have pop-up breeders.
Continuous optimisation towards a single measure - sorry, target - pushing users down a cul-de-sac of “please convert today”. Don’t lie, we’ve all experienced the “10% off now” pop-up shoved in our face as soon as we land on most sites. Inappropriate behaviours caused by inappropriate targets creating this myopic form of digital directness.
In short, how brands measure directly reflects how they sell. And currently, they sell to do nothing but convert.
How do we solve this?
If the problems of conversion rate optimisation, are that it aims towards a target of something that is macro, binary and out of your control, it makes sense to focus on a measure of success that is micro, scaleable, and controllable.
Towards metrics that are micro, yet specific.
The concept of treating every customer the same is silly. It sounds silly. It feels silly. Everything becomes the same because brands treat everyone the same. This manifests itself in how brands currently measure success on websites: using conversion rate.
The metric used to understand success is highly aggregated at a macro level. It’s an average of averages of averages. It’s made up of so many conversion rates (plural) that trying to shift that behemoth metric is akin to pulling a U-turn on the Titanic – it’s going to take a lot more than a boatload of topless coal trimmers.
The result of optimising for an averaged metric is an averaged approach. The result of optimising with an averaged approach is an average result. Homogenisation begets homogenisation.
Much to Goodhart’s law, by optimising metrics that are aggregated, the output, too, becomes aggregated. It is why in ecommerce that we see page template optimisation based on the homepage, the product listing page, followed by the product detailed page, then the basket and finally the checkout.
A true enemy of personalisation.
The king of which, Netflix, announced as of 2024, they will no longer report on its average revenue per member (ARM), believing this is an irrelevant statistic to measure success. [] They cite it being an irrelevant measure due to its macro view of what it indicates; the clue is in the name average revenue per member, one that holds so much volatility over a period of time
It’s also a measure of success that’s indicative of business performance, not necessarily customer performance. Sorry, averaged business performance, not average customer performance. And so, moving towards something more symbolic of customer satisfaction, like viewing time, is a movement towards a move away from the macro aggregation of a one-way business metric and towards a measure that highlights the specificity of the business model.
Towards metrics that demonstrate scale and momentum
Conversion is not a binary point of success or failure. It is a progression of performance. A momentum.
As a brand, my job is to build customer momentum, persuade, create desire and elicit these emotions to allow you, the user, to take action. It’s not to just go straight for the jugular or to flip a switch.
And yet - why is it measured as such? There’s a reason why the majority of marketing models have a funnel usually in the form of four letters that tenuously spell out a loosely put-together acronym: the, , all stand up.
Put somewhat more succinctly, the process of buying online is about an upward trajectory not an end outcome. Translated into KPIs, it’s about scalable performance, not a binary result.
Looking at a not-so-similar industry, we can see what changing the success metric did for the world of football (he writes as Germany are battering Scotland in the Euros). was a stat created by Opta for use in football. A metric that indicates the quality and performance of the game based on a series of attributes fused together. These might include the angle of a shot, possession of play beforehand, the distance of a shot, and; less so who is taking the shot. It “provides an antidote to the disease of randomness which permeates football” (). Sounds familiar. In other words, it’s a statistic that is better representative of 90 minutes of a game than just what might be a lucky 5-1 win (granted, Germany isn’t lucky in this game). It’s a statistic that helps understand the performance of the game, not the end result.
Like ecommerce - still, unfortunately - football is built on opinion. OK, that’s what makes it fun. But, the data revolution requires more rigour when commercials are at stake.
Online needs to move towards something more indicative of movement, momentum, performance, quality, and play, less so the binary outcome of whether someone did or didn’t do something. This metric should be one that scales as behaviour does.
Towards metrics that are within your control
Conversion rate is not entirely within your control. If you’re selling umbrellas and it’s sunny, guess what will happen to your conversion rate? It’ll get rained on. The adverse, too, if it rains, the sun will ironically shine on sales.
There are far too many uncontrollable factors to determine whether someone will or will not do something online. When you put it like that, it makes it kind of silly that we look at conversion rate as a measure of success for websites now, doesn’t it?
In the 90s, Donald Wheeler wrote about “the key to managing chaos” and spoke directly about casual relationships. Specifically, “You cannot improve by listening to the voice of the customer. You can only improve a process by listening to the voice of the process”[].
These are fancy terms to suggest that focusing on the result—a demand from the customer or management—is not nearly as impactful as understanding how the process actually works: casual relationships.
Losing weight is a great example. You are in control of the inputs or calories in and calories out. If you understand how those inputs impact the result, your weight loss process, you will lose more weight. Of course, there are plenty of inputs within each primary input—the balance of carbohydrates, protein and fats, water intake, type of calorie—I could go on, but I’ve clearly not lost much weight in over 12 months.
This is a reason why Amazon only measures success on what they can control, and what they input; something talked about in great detail under Working Backwards []. The calories in and the calories out of ecommerce—not the weight loss. This might look like (and does in accordance with Amazon’s WBRs - Weekly Business Review):
- An increase in the number of detail pages, while seeming to improve selection, did not produce a rise in sales; the output metric
- the percentage of detail page views where the products were in stock and immediately ready for two-day shipping, which ended up being called Fast Track In Stock, did produce a rise in sales, the output metric
These are written as hypothesis in alignment with their ruthless test and learn culture at Amazon, but the principle remains: Focus on the input, not the output. This will prevent a Goodhart’s Law type of situation.
Towards Intent Metrics
If the micro beats macro.
If the demonstration of measures that scale beats an outcome that is binary.
If controllable inputs beats uncontrollable outputs.
We need a measure that is reflective of the nuances (micro measures of success) of the performance (the scaleable momentum shifts) of a user (not the business); not just the end outcome.
We took great inspiration from the Germany-Scotland game and created a measure that helps understand the quality of the play as opposed to the quantity of the outcome. A sentence that permeates the world in “quality beats quantity,” which, if you stand behind, will almost certainly have you sitting on the edge of your seat for this one.
Introducing Intent Metrics.
We collect 250x different intent signals—the nuanced behaviour of what users do online—and model those together, outputting them in a series of predictive metrics.
Breaking these metrics down into the known metrics with a predictive layer, we find:
- expected conversion (xC) - the likelihood that a user will convert
- expected add to cart (xATB) - the likelihood that a user will add to their cart.
From there, at ϴ¼, we utilise metrics indicative of behaviour that are more brought forward than the macro level.
- expected exit (xE) - the likelihood that a user will exit their session
- expected return(xR) - the likelihood that a user will return to the site.
And finally, we land on a measure that helps understand the core of what selling is all about: nurturing.
- expected progression (xP) - the likelihood that users will progress to the next stage of the buying journey. We call these “buying stages” and identify them as browsing, refining, evaluating, deciding and committing. They are also similar to those 4-letter, 1980s acronyms thought up by genius marketers.
Not to mention in all of these, the benefit of these being expected measures. One of the consistent flaws of conversion rate is that it is a retrospective target. A measure that’s already happened, and so, not something that you can influence there and then. Intent metrics have the added benefit of being predictive. They highlight what a customer might do, not something they’ve just done, meaning you can intervene in real time. Whilst predictive analytics was on the rise - at the peak of expectations in 2018 []- their use cases in ecommerce have been limited. The introduction of real-time analytics has brought this somewhat to life, in combination with using first-principle thinking (thank you, Germany vs Scotland).
Conclusion & next steps
When Netflix announced a move away from reporting on subscriber growth to something more indicative of success, they used two words in that were the most vital of the announcement: “We’ve evolved”.
Ecommerce needs to evolve.
The flaws within conversion rate are evident for all to see. It is an aggregated, macro, uncontrollable, retrospective, binary measure of success that can easily be gamed, where optimising reflects how you will sell. Cue the 10% off popups as soon as someone lands on your site.
The introduction of Intent metrics is a movement towards the opposite. It is a reflection of the performance, not just the result. An appreciation of the progression and momentum build of prospects, not just a binary outcome. A systematic approach to identifying controllable inputs as opposed to reviewing the uncontrollable outputs. Most of all, it is a focus on the predictive, not the retrospective.
