New in November, 2017:
- Value Control reporting shows the results of the last 30 campaigns together, giving more meaningful figures.
- Experiments are updated for 7 days after they finish, to ensure that all the data has been synced to Emarsys.
- Test groups have ineligible contacts removed from their total figure.
- Sub-segment results for the test group are updated hourly.
- Improved error handling if results are currently not available.
As a quick reminder, here is the onboarding video about Value Control:
Emarsys Value Control is a control group testing feature that measures the effectiveness of your email campaigns. It does this by preventing a percentage of the launch list from receiving the campaign (the control group) and comparing their behavior over a predefined time period (the experiment) against the rest of the launch list who did receive the campaign (the test group).
Value Control requires Smart Insight data to be loaded for the seven days prior to an experiment, and for the entire duration of the experiment, otherwise it will not work.
The results of the trial period for each campaign can be found in the Analysis menu under Value Control Reporting. This presents the revenue and engagement figures of the test groups side-by-side with those of the control group, so you can see exactly what influence your campaign has had on their behavior. In this way you can see instantly if your strategy is successful or if you should look for ways to improve it.
This feature is only available for Smart Insight users (at no extra cost), and can be used for template-based or custom-HTML batch email campaigns.
It does not currently support the following:
- Email campaigns sent via Automation Center programs.
- Email campaigns that use A/B testing versions.
How does Value Control work?
Value Control takes as its baseline what the contacts in the launch list would have done anyway during the trial period. This is the behavior of the control group, and is measured in four metrics:
- Purchase rate (% of contacts who made a purchase)
- Average revenue per purchase
- Average revenue per person
- Engagement rate (% of contacts who visited the website)
The same metrics are measured for the test group for the duration of the experiment, and these two sets of metrics are then compared side by side:
Only responses which occurred during the timeframe of the experiment are counted. Value Control continues to update the graphs for seven days after the experiment has finished, to allow for any delays in data synchronization.
Working with Value Control
If you are a Smart Insight customer, Value Control is automatically available for you as a step in the email campaign creation wizard.
Turn on Value Control for a campaign by simply activating the toggle. You are then presented with these controls:
- Select how much of the launch list should be used for the Control group. These will be contacts selected at random from the email launch list and will not receive the email.
- Select how many days you want the evaluation period to run. A day is counted as 24 hours, not a calendar day.
- Decide whether or not the control group should also receive a copy of the campaign at the end of the trial period. This copy will be created immediately after the original campaign is scheduled and and will be scheduled for launch at the end of the evaluation period.
Note: If you do schedule a copy of the campaign for the control group, but decide that you want to edit or cancel it (for example if the results are not as expected), you can find this campaign in the Emails list and unschedule it at any time during the evaluation period, and make your desired changes.
Once you have saved your settings, Value Control is active and will randomly select the test and control groups when the campaign is launched. Please note that this means the Original target figure on the Scheduling will not take Value Control into account and will still show the full launch list. However, the Original launch list figure on the Results Summary page for this email will show the true number of contacts who received the email (i.e. only the test group).
You can then measure the added value the campaign has brought you on the Value Control Reporting page.
When & how to use Value Control
Value Control can be used on any batch email campaign. We recommend using for as many campaigns as you can, as it not only measures the additional value of that campaign, but the control groups also give you continuous insights in to the regular behavior of your different contact segments.
The contacts in a control group will be omitted only from that particular campaign and not from any other campaign unless specified. If you would like to exclude control group contacts from other campaigns during the test period, you will have to use the Save as contact list button to save them as a contact list, and then exclude that list manually from the relevant campaigns.
Here are a few other points to help you make the most of Value Control.
This is not A/B Testing
Value Control is not about responses, it is about value. You can use it to validate even the most basic assumptions, such as whether discounts actually trigger additional purchases or whether the same number of purchases would have happened anyway.
Keep the control group large enough to be significant
It is tempting to make the control group as small as possible, but this reduces the validity of the information you receive from the trial. We recommend to start with a 50/50 split and move from that only for fine-tuning campaigns.
Don’t confuse this with revenue attribution
Revenue attribution links purchases to email campaigns if the purchases happened x days after the click (according to your attribution window). While this is important, it does not show you how many people received the campaign and made a purchase afterwords without clicking – by going directly to the website, for example. Furthermore, some people would have made a purchase anyway and only clicked the campaign because it was there. Value Control is designed specifically to exclude these scenarios.