Metrics are essential for analyzing experiments in Eppo, as they provide quantitative measures to evaluate the performance of different variations in an A/B test. In this guide, we walk you through simple metrics: an aggregation over an entity. An example of a simple metric at the User entity level would be Revenue: this is the sum of money spent by a user.
Anatomy of a metric
Simple metrics turn an event stream (created from a Fact SQL Definition) into a single number for a variation in two steps:
- First, the events are aggregated at the entity level (such as a User) using an aggregation
- Second, the values are averaged across entities
For example, to compute a revenue metric, first we sum the amount spent by each user. Second, we cake the average across all users to obtain the average revenue per user. In SQL terms, a simplified version of this two step process looks like
WITH user_summaries AS (
SUM(spent) AS user_spent
GROUP BY user
WHERE spent_ts BETWEEN experiment_start AND experiment_end
AVG(COALESCE(user_spent, 0)) as metric_estimate
LEFT JOIN user_summaries ON user
GROUP BY variant
Metric aggregation types
In Eppo, you are able to create a wide variety of metrics because Eppo supports many aggregation functions:
Let's discuss each of them in more detail.
Sum computes metrics that are typically interpreted as averages per entity. If the fact value is NULL, it is discarded.
Examples: average revenue per user, sign-up rate, minutes streamed per user, average order value.
Unique Entities computes the number of unique entities with a non-null event. If the fact value is NULL, it is discarded.
Examples: % of users with a video watch, % of visitors who viewed an article, % of users who entered checkout.
Count leverages SQL's
COUNT to compute a total count of events per entity. If the fact value is NULL, it is discarded.
Examples: videos watched per user, articles viewed per visitor, orders per user.
Retention metrics measure the proportion of entities who have at least one fact value appear after a fixed number of days (X) from experiment assignment. For example, a 7-day retention metric on the website visits fact might measure the proportion of users who visit a website at least 7 days after being assigned to the experiment.
For example, if , Eppo records a retention event for an entity when
Conversion metrics measure the proportion of entities who have at least one fact value appear within a fixed number of days (X) from experiment assignment. For example, a 7-day conversion metric might measure the proportion of users who sign up for a free trial within 7 days of being assigned to the experiment.
For example, if , Eppo records a conversion event for an entity when
Threshold metrics measure the proportion of entities who meet a user-specified
COUNT of a fact within an optional time-period. For example, you might want to understand what share of users in your experiment spent more than $100 within 7 days of assignment into an experiment.
Creating a metric
Now that we understand how metrics are defined withen Eppo, we are ready to create a metric. Note that in order to create a metric, you first need to have set up a Fact SQL Definition to point to the underlying data for your metric in your data warehouse. Let's walk through an example to create a User Revenue metric based on a fact table that lists purchase events at the user level.
- Navigate to
+Metric, then select
Useras the subject of the metric
User is the default entity in Eppo, but you can also create a custom entity and select it here.
- Select a fact
This should be one of the facts that you created in the step above, and should correspond to a metric that you want to track in an experiment.
- Select an aggregation
The aggregation will aggregate over whatever the fact is measuring on a per-entity basis. So for example, if you select a
Revenue fact and the
SUM aggregation, the metric will be "average revenue per user;" if you select a
Name of Article Viewed fact and the
COUNT DISTINCT aggregation, the metric will be "Unique articles viewed per user."
- Set outlier handling
Eppo handles outliers through a technique called winsorization. Lower and upper bounds by percentiles for winsorization can be defined by the user for every metric. For example, in the screenshot below, we are setting the upper bound for winsorization at the 99.9th percentile. This means that any user with a value above the 99.9th percentile will be rounded down to the 99.9th percentile value.
Note that winsorization is only utilized for
COUNT aggregations. This is because conversion and retention metrics are binomial variables that are not prone to influence from outliers. As a result, winsorization is redundant for these metric types.
- Set a default precision target
The precision refers to the uncertainty you want to be able to measure in an experiment, as measured by the width of confidence intervals. You can set a default at the metric level, which will be used to measure progress if this metric is the primary metric for an experiment.
- Set formatting options
Here, you can select whether to format the metric as a number or as a percentage.
Eppo allows you to further refine metrics by adding a timeframe. For example, we may be interested in a metric that only considers purchases within one week of the user's assignment to an experiment.
Consider adding a timeframe metric to experiments where you believe the intervention has a short term effect. For example, suppose sending a promotional email boosts engagement for one week. If we do not add a timeframe and run the experiment over 4 weeks, then the experiment effect may be diluted by a factor up to 4.
Metric properties give you another tool to filter events based on a metric property. Suppose you work at a streaming platform that streams both movies and series and watch time is your primary metric. However, you may also be interested in understanding the impact of experiments on movie watch time and series watch time separately. In this case, you can add show type as a category and create separate metrics for movies and shows.
When creating such a metric, specify that you want to use a metric property and select which values are valid.
Navigate to the Metrics tab on the left-handside menu.
Navigate to the metric you want to edit
Click on the three dots on the right hand side of the row
Select Edit Metric from the dropdown
Navigate to Metrics on the left side menu
Click on the three dots on the right hand side of a metric
Select Delete from the dropdown menu
Note that if you delete a metric that is being used in an active experiment, that may affect the experiment currently being run.