You can measure the impact of your Automation Center programs by using this node. The Measurement node allows you to assess the added value of a program by comparing the uplift it generates against your Key Performance Indicators (KPIs).
As a quick summary, here is the onboarding video for Value Measurement:
How does it work?
The Measurement node estimates the added value of your programs as follows:
1. It divides contacts into two groups:
a. 90% of contacts are assigned to the treatment group.
b. 10% of contacts are randomly assigned to the control group.
The group membership of the contacts is stored together with the programs.
2. Contacts in the control group will be dropped out of the program right after the Measurement node. These contacts will not receive further messages sent by this program or any other program with Measurement nodes. However, they will receive all campaigns sent by other programs.
When contacts enter the Measurement node, their behavior is measured for 5 days. This includes registering email opens, email clicks, website visits, mobile push opens, SMS clicks and purchase data for each contact. This 5-day long measurement window is fixed. The measurement period does not take the program length into consideration.
Before adding the Measurement node to your program, consider the following:
- You can add only one Measurement node to a single program.
- It is not recommended to insert channel nodes before the Measurement node.
- Once a contact has been assigned to the control group in a program, then this contact will belong to the control group in all programs using this node. This will help us calculate the combined effect of multiple programs.
You can check the estimated added value of your program on the Value Measurement page.
Tips for using the Measurement node
The Measurement Node can be deleted from the program at any time. This will enable the program to be sent out to all contacts in the program according to the scheduling settings.
Control group members will not return to the program after deleting the Measurement node.
Value Measurement page
You can check the uplift generated by your Automation Center programs and compare the results of the treatment group with that of the control group on the Analytics > Value Measurement page.
Some KPIs look at interactions made by the contact, for example "Share of Contacts with Email Opens". Because contacts who are in the control group can receive marketing communications through other programs without Measurement nodes, it is possible to draw comparisons between the treatment group and the control group for these interaction based KPIs as well.
This also allows us to measure the impact of your program with the Measurement Node compared to what would happen if you did not have this program.
On this page, you can:
- Specify the Date Range in the top left corner. This will filter the results from the selected programs.
- Display the metrics of a single, multiple or all Automation Center programs by selecting the programs you need from the Measured programs drop-down.
If multiple programs are selected, you will see aggregated results.
In the Overview widget, you can see the following metrics divided by Treatment and Control groups:
- Number of Contacts: the amount of contacts that are in the given group. The treatment group constitutes 90% of your whole contact base and the control group consists of the remaining 10%.
- Group Size: the percentage of the number of contacts compared to the size of your whole contact base that enters the program.
- Purchase Rate: the percentage of the contacts who made a purchase compared to all the contacts within the given group
- Engagement Rate: the percentage of contacts who made any type of your interaction (email open, email click, website visit, etc.) compared to all the contacts within the given group
- KPI Performance: the number of measured KPIs that are likely to have a positive effect
In each KPI widget, you can check the following metrics:
- Relative uplift - The treatment group’s relative performance compared to the control group’s performance expressed as a percentage.
- Likelihood notification - This estimate (expressed as a percentage) shows you how likely it is that the performance of the treatment group exceeds that of the control group. In other words, it shows the likelihood of the relative uplift being above 0.
- Measured results: - These numbers show what was measured during the measurement window for each group.
The metrics for each KPI are calculated from these numbers. For example, if the control group had 8,207.1 EUR of total revenue from 74 orders, then the average order value for the control group was (8207.1 / 74) = 110.9.
Calculations
From the measured data, the following summary statistics are calculated for each group (treatment and control):
- Number of contacts
- Group size (percentage)
- Number of purchases
- Total revenue
- Number of contacts who visited the website
- Number of contacts who opened an email
- Number of contacts who clicked in an email
- Number of contacts who opened a mobile push message
- Number of contacts who clicked in an SMS
-
Average order value
(total revenue / number of purchases)
-
Average revenue per contact
(total revenue / number of contacts in the group)
-
Share of contacts with purchases
((number of contacts with purchases / number of contacts in the group) * 100)
-
Share of Contacts with Website Visits
((number of contacts who visited the website / number of contacts in the group) *100)
-
Share of Contacts with Email Opens
((number of contacts who opened an email / number of contacts in the group) * 100)
-
Share of Contacts with Email Clicks
((number of contacts who clicked in an email / number of contacts in the group) * 100)
-
Share of Contacts with Mobile Push Opens
((number of contacts who opened a mobile push message / number of contacts in the group) * 100)
-
Share of Contacts with SMS Clicks
((number of contacts who clicked in an SMS / number of contacts in the group) * 100)
-
Purchase rate
((number of contacts with purchases / number of contacts in the group) * 100)
-
Engagement rate
((number of any type of interactions (for example email clicks, mobile push opens, etc.) / number of contacts in the group) * 100)
Relative uplift and likelihood calculations
The relative uplift for each KPI is calculated from the values of each group in the following way:
(treatment group value - control group value) / control group value * 100
This results in the relative percentage value by which the treatment group is better.
Using the Bayesian method of control group testing, we calculate the likelihood that the treatment group outperforms control - this estimate (expressed as a percentage) shows you how likely it is that the performance of the treatment group exceeds that of the control group.
Known limitations
- The Measurement node can only be used in recurring or batch programs, you cannot use it in transactional and Behaviour-focused (Interactions) programs.
- You cannot change the percentage of contacts assigned to the control and treatment groups.
- Value measurement reports are available only 3 days after a contact enters a program to ensure that conclusions are drawn by taking the right amount of data into account.