Your first experiment analysis
Follow this quickstart to get your first experiment set up on Eppo.
Overview

- 1. Prep your data warehouse and set up experiment assignment
- 2. Create Eppo account
- 3. Connect your data warehouse
- 4. Create an Assignment SQL
- 5. Create a Fact SQL
- 6. Create metrics
- 7. Create an experiment
- 8. Add metrics to experiment
1. Prep your data warehouse and set up experiment assignment
This quickstart assumes that you already have assignment logs tracked in your data warehouse, either from Eppo's randomization solution or an existing internal or third party solution. In particular, Eppo requires that you have the following tables in your data warehouse:
An assignment table that indicates which experiment subjects were assigned to which experiments and variants at which time. It should contain columns that roughly correspond to:
timestamp subject_id experiment variation 2021-06-22T17:35:12.000Z 165740867980881574 adding_BNPL_experiment affirm where
subject_id
is a unique identifier for the entity on which you wish to experiment (in most cases, this will be eitheruser_id
orsession_id
).
An event table whose rows are logs of specific events that occurred at specific times. It should contain columns that roughly correspond to:
timestamp subject_id event_type event_value 2021-07-17T18:57:13.000Z 49980400511307080 Revenue 45.5695 2021-07-17T18:57:13.000Z 2281323415877132491 Subscription 1
If you do not have a experiment assignment tool integrated, please refer to the experiment assignment section for more instructions.
If you have a third-party feature flagging tool integrated but do not have an assignment table set up in your data warehouse, please follow the instructions here to log your assignment data.
2. Create Eppo account
Create a new account at https://eppo.cloud/. Eppo implements Auth0 and you should be able to sign in with any of the providers that Auth0 supports, including Google.
3. Connect your data warehouse
Follow the linked warehouse-specific guides to connect Eppo to your data warehouse.
4. Create an Assignment SQL
Now that Eppo is connected to your data warehouse, the first thing we are going to do is create an Assignment SQL. Assignment SQLs define which experiment subjects will be assigned to which experiment and variation, and at what time. They're basically just pulling data from your assignment table.
Navigate to Definitions on the left hand menu and click +Create Definition SQL, then click Assignment SQL
Select the subject of the Assignment SQL
Entities are the randomization units of your experiment. By default, entities in Eppo are User, but you can also create your own custom entities and attach Assignment SQL's to them.
Name your Assignment SQL
Write SQL in the SQL editor to pull assignments from data warehouse and click Run
Recall in the Set up experiment assignment section that you should have an assignment table in your data warehouse with certain column types.
In this step, you're going to write SQL to pull that data.
Then click Run, and the rows from that assignment table should appear in the bottom left.
- Annotate the columns that you've selected from the data warehouse
In case there's any ambiguity as to which properties the columns correspond to, we annotate them here.
- Make note of your feature flag name and variant names
Note the value of the FEATURE FLAG column; in this example it's new_user_onboarding
- this is your feature flag name.
Note the values of the VARIANT column; in this example it's control
and treatment
- these are the names of your variants.
You will need these names later.
- Adding optional dimensions
Your feature flag tooling may have logged additional data about the user, like what country they're from or which browser they're using. You can annotate these additional dimensions here, and they will show up under the Dimension SQL tab.
- Click Save & Close
You've created your first Assignment SQL!
5. Create a Fact SQL
Fact SQL's define events, like sign-ups, activations, or orders. Together, the Assignment SQL and Fact SQLs give us a picture of what happened to different segments of users shown different variants of the experiment.
Navigate to Definitions on the left hand menu, click +Create Definition SQL, then click Fact SQL
Select User as the entity
User is the default entity in Eppo but you can also create your own custom entity and select that here.
Name your Fact SQL
Write SQL in the SQL editor to pull events data from the data warehouse
Recall in the Set up experiment assignment section that you should have (potentially multiple) event tables in your data warehouse with certain column types.
In this step, you're going to write SQL to pull that data.
- Annotate the columns that you've selected from the data warehouse
In the example above, Eppo has already automatically determined that the timestamp of creation
column is TS
and the relevant entity id
column is USER_ID
.
- Add Facts
For every event that you want to track, you should add its corresponding column in the data warehouse as a fact.
In the example above, we would like to measure the effect of the experiment on revenue
.
In the data warehouse, every time there is a revenue event (that might be someone purchasing something), that event is logged as a row in the data warehouse.
This event is translated into an integer value, which is the revenue amount, that can then be used as an Eppo fact and tracked in an experiment.
- Click Save & Close
You've now created your first Fact SQL. If you have more events that you would like to track in your experiment, you can repeat the steps in this section to create additional Fact SQLs.
6. Create metrics
- Navigate to Metrics, click +Metric, then select User as the subject of the metric
User is the default entity in Eppo, but you can also create a custom entity and select it here.
- Select a fact
This should be one of the facts that you created in the step above, and should correspond to a metric that you want to track in an experiment.
- Select an aggregation
The aggregation will aggregate over whatever the fact is measuring on a per-entity basis. So for example, if you select the Revenue
fact and the SUM
aggregation, the metric will be the total revenue for each user; if you select the Upgrades
fact and the SUM
aggregation, the metric will be the total number of upgrades for each user, which should only be 1 across the board.
Eppo supports the following aggregations:
- SUM
- COUNT DISTINCT
- COUNT
- RETENTION
- CONVERSION
- (Optional) Select a Filter
When you created an assignment SQL above, you may have also created additional dimensions, i.e. country or browser. These dimensions are now available under the Definitions > Dimensions SQL.
Here, you can filter on any of those dimensions. In the example above, we want to track the revenue metric by country.
- Select a minimum detectable effect
The minimum detectable effect refers to the smallest effect you want to reliably detect in experiments. The higher the minimum detectable effect you set, the longer the experiment will take to reach conclusive results.
7. Create an experiment
Navigate to Experiments in the left-hand menu and click +Experiment
Fill out the Create Experiment Form
Give your experiment a name, start and end date. You can optionally provide no end date if you would like the experiment to run indefinitely. Refer to the best practices guide if you're not sure how long you would like the experiment to run.
- Navigate to the Set Up tab and click the Configure the Experiment button
- Select the Assignment SQL that you created previously
- Input feature flag name
When you created your Assignment SQL, you should have made note of your feature flag name. In our case it's new_user_onboarding
.
- Input what percentage of traffic you want randomized into the experiment.
If you input 100%, 100% of the relevant entities (users in this case) will be included in the experiment.
- Add the variants
When you created your assignment SQL, you should have made note of your variant names. In our case here it's control
and variant
.
You can go ahead and add each variant into the experiment here.
- Select the traffic allocation you would like
The default is an even split between all the variants (including control), but you can also customize the allocation for unequal splits.
- Click Save Changes
8. Add metrics to experiment
- Navigate to Experiments and click the Overview tab
Under Decision metrics, you will see that guardmail metrics have already been included automatically.
- Click +Add metric button
On the left hand of the modal, you will see a list of metrics that have been created that are attached to this set of experiment subjects.
You can select one of them to add to the experiment
- Click Save
Your experiment is now in progress!
Your experiment will likely take a few days to start outputting results. Go here for an overview of how to interpret your experiment.