The Metric You Optimize Changes the Experiment You Design

October 2026 | Applied Economics | Experimentation | ← Back to Blog

How long an experiment runs, what unit it randomizes, and how much traffic it needs all depend on one earlier choice: the metric. This post walks through how the metric shapes the observation window, the unit of randomization, what you have to measure, and the sample size, and how variance reduction with CUPED fits in.

The Metric Comes First

The metric you want to improve decides how long your experiment runs, what unit you randomize, and how much traffic you need.

It is tempting to start from a familiar setup, a user-level test split 50/50 for two weeks, and decide at the end which metric to read. But each of those choices depends on how the metric behaves: how quickly it moves, whose behavior drives it, and how noisy it is. So the design starts one step earlier, with the business objective. What decision will this experiment inform? The metric is whatever measures progress on that objective, and once it is chosen, most of the design follows from it.

Decide what you want to improve before deciding how to test it.

Duration and Observation Windows

Some metrics move quickly and show up right away. Clicks, conversion, and page load time respond within a session, so a short test can measure them, although it is still worth running at least a full week to cover day-of-week patterns.

Other metrics only reveal the effect weeks or months later. Retention, repeat purchase, and lifetime value often need a longer observation window. When running the test that long is not practical, two tools help. A long-term holdout keeps a small group on the old experience after launch, so the long-run effect can still be measured. A surrogate metric is a short-term outcome used to estimate the long-term one. It needs to be validated for that purpose. A metric that predicts long-term behavior in general does not automatically capture how a particular treatment changes it.

The Unit of Randomization

When users do not affect each other much, a metric like conversion can usually be tested by randomizing users. Each user's outcome depends mostly on their own treatment, so a treated user and a control user can be compared directly.

Marketplace metrics are different. Liquidity, how reliably buyers and sellers find each other, depends on both sides of the market at once. Treated and control buyers draw on the same supply, so a change that helps treated buyers find listings can leave fewer listings for control buyers. The control group is then no longer an untouched comparison, and the test can overstate the benefit compared with giving every buyer the treatment. More generally, the direction and size of the bias depend on how the treatment changes shared supply and demand. An earlier post on treatment spillovers covers how to check for this.

The design then has to account for the interference. Options include randomizing time windows in a switchback experiment, whole markets or regions, or in some settings both sides of the market. The trade-off is fewer independent units, which means less statistical power.

Profit Versus Revenue

The objective also changes what you have to measure. A simple revenue metric needs price and quantity, plus any refunds or cancellations, depending on how revenue is defined. A profit metric needs the cost of each transaction as well: cost of goods, shipping, and payment fees. Discounts, refunds, and returns also have to be tracked, because they reduce the revenue actually earned and can add costs of their own. That data has to be logged at the transaction level before the test starts, because it is hard to reconstruct reliably afterward.

The difference shows up in the result, not just the logging. Take a discount test. Measured on revenue, it can look like a clear win, because volume rises. Measured on profit, the same test can look very different: every extra unit sells at a lower margin, and some of the discounted purchases would have happened anyway. An earlier post on the revenue maximization trap shows how far apart the two objectives can be.

Noise, Traffic, and CUPED

How much traffic a test needs depends on how noisy the metric is. A noisy metric bounces around a lot from one user to the next, so a real change is harder to tell apart from chance, and you need more users to see it.

Conversion is a yes-or-no metric: each user either buys or does not. Profit per user is often much noisier. Most users contribute nothing, a few contribute a lot, and margins vary from order to order with shipping, discounts, and returns. So a profit test can need substantially more users than a conversion test to detect a change of similar importance. The exact number depends on how spread out each metric is, its baseline level, and the smallest change worth detecting.

CUPED is one way to bring that number down. Much of what a user does during an experiment can be predicted from what they did before it. Someone who spent a lot last month will probably spend a lot this month too. CUPED adjusts each user's outcome for what their past behavior already predicts, which takes some of the noise out of the comparison. Less noise means fewer users may be needed to detect the same effect.

It has two conditions. It helps only when past behavior actually predicts the outcome, so it does little for new users with no history. And the past behavior has to be measured before the experiment starts, so the treatment cannot affect it. Tracking a steadier metric alongside profit, or capping a few extreme values, are other common options. Capping needs care, because changing how the outcome is measured can change the question the test answers.

A Practical Design Sequence

Putting the pieces together, a design can follow this order:

Define the objective. What decision will the result inform?
Choose the metric that measures progress on that objective, and the guardrails that protect what it misses.
Set the observation window. How long until the effect shows up?
Assess interference. Does treating one unit change another unit's outcome? If so, choose a unit of randomization that contains it.
Determine the data you need, including transaction-level costs if the objective is profit.
Calculate the sample size from the metric's variance and the smallest effect worth detecting, after any variance reduction.

The Takeaway

Decide what you want to improve before deciding how to test it. The metric answers most of the design questions for you. The ones it leaves open, such as how interference travels in a particular market, become much easier to see once the metric is fixed.