The following are the most common questions about Send Time Optimization.
We count any open as a success, no matter how long after the launch it comes, so a contact will continue to receive emails at that time, as long as they continue to respond. A lack of open will cause the algorithm to calculate a new time for that contact, based on its current statistical model.
It usually needs at least 10-20 launches to see the first stable improvements, until then, results may fluctuate, even deteriorate, so be patient. You will see the results in time.
No. We have found no significant difference during the pilot phase in unsubscribe rates between STO and non-STO campaigns.
Currently, no. Our algorithm analyzes contact events in order to determine the best possible send times. Excluding hours manually would decrease the efficiency and the performance of the algorithm.
The calculations begin as soon as Launch now is clicked, and control group email will be sent as soon as they are finished, that is, within a maximum of two hours. The first real STO launch will happen on the following even hour.
Therefore a delay of up to three hours after clicking the Launch now button is, in theory, possible.
No. Scheduled campaigns are launched from 00:00 through to 22:00 on the calendar day of the launch (in the timezone of the account), and manually launched campaigns are staggered over 24 hours from the time the calculations are finished. No other control over the timescale is possible.
In most cases, no. For time-sensitive emails, such as limited special offers or discounts, you should consider the duration of the offer before using this feature. If an offer is only available for a 24-hour period, for example, you will not be giving all your customers the same advantage if STO is used.
Each contact’s profile is updated every 2 days with the latest engagement information in order that we have the most recent data at our disposal.
If the Frequency Cap is active, it will apply to each launch at the moment of launch. For example, if an STO campaign is scheduled for a day when contacts whose Frequency Cap is set to 1 have already received a message, the STO child campaigns launched on that calendar day will not be sent to them. But they will receive the launches sent on the next calendar day.
Yes. In such cases the given launch will not be created at all, which means that there will be fewer than 12 launches in the campaign. But if you have a large contact list containing thousands of email addresses, and especially if these are geographically dispersed, this is highly unlikely to happen. You can always check the statistics breakdown for each STO campaign by going to Analysis > Emails and clicking the plus sign to see the results of the individual launches.
Even though STO algorithms basically use the same data that you can see in the Launch Time Advisor, STO does the math for you and calculates the timing automatically on a per-contact basis. When using STO, you can simply sit back and enjoy the results.
About the control group
We need a large enough control group in order to reliably measure results. On the other hand, we do not want to adversely impact your engagement results. We think that 10% is a good trade-off.
No. The control group is not optional. It is the basis of all monitoring and reporting, and it also saves you the time and effort you would need to invest in A/B testing.
Monitoring and Reporting
It is currently a limitation for this feature that you cannot see the Deliverability Reporting for the parent campaign (i.e. all 12 launches aggregated). You must open each launch individually to see the deliverability statistics.
For any given week, we compute the aggregated open rate for both STO and CONTROL launches, then calculate the difference as a percentage of the CONTROL open rate: (STO open rate - CONTROL open rate) / CONTROL open rate as %.
- If the two groups performed identically, e.g. both have a 15% open rate, then the added value of STO will be 0%.
- If the control group launches have a 15% open rate and the STO launches 16.5%, then the added value of STO will be +10%, since the +1.5% difference is 10% of control group's 15% open rate.
When aggregating weekly results, each campaign is attributed to the day on which its control group emails were sent.
Results will only be shown 48 hours after the launch of the first STO campaign, to allow for a meaningful amount of engagement results to be returned.
If the recipient list contains fewer than 100 contacts there is no control group, therefore the results of this campaign will not appear in the monitoring widget.
Differences in open rates between the control and STO group will naturally occur, and we need enough data to ensure that the results are statistically relevant. In order to have a fair comparison, we need to aggregate results on a weekly basis so that these natural differences disappear.
The science behind STO
Both. We cannot know for sure which is the best hour for any contact, so it is important to experiment with send times we have minimal or no information about. On the other hand, we obviously want to exploit the knowledge we already have and send at times which perform well with high probability. The algorithm finds the best trade-off between exploration and exploitation. The more historical data you have the better results you can expect as STO needs less exploration and can do more exploitation.
If you are an existing customer, we use your account’s aggregated historical data on open times to assign the most likely send time to your new contacts. If you are a new customer with no historical data, new contacts are randomly distributed among the launches if STO is turned on. That is why it is highly recommended to wait at least a month before activating STO.
The algorithm automatically adapts to consumer preference changes: for example, if the client moves to a country in a different timezone or their daily routine changes, the algorithm will gradually adjust the sending time as we receive this feedback.
Partially, yes. It still collects response data on an account level, so the underlying statistical model will be better when you next switch it on, benefiting future contacts.
STO currently considers a timeslot “good” if there is an open afterwards. Thus, it will continue to send emails at the same time while that launch is successful (i.e. it triggers an open). For any given sending time, a negative feedback is when the email is never opened. Only then will the algorithm update itself.
One possible reason is that the client was in the control group in that particular campaign and the control group was scheduled to be sent at 4:00 pm. If this is not the case, the algorithm aims to collect information about every possible send time to have the chance to find even better send times.
The better the current send time performs, the less frequently the algorithm will try different send times, but this will still happen occasionally. This also ensures that we can adapt to changes in the preferences of the contacts.
If there are two, or more, similarly performing send times the algorithm will send to these with similar frequency; it is not a goal to pick one of them. This ensures that we have the possibility to find out if one of them is actually better in the long run, even if both produce very similar results for a period.
For new contacts, we do not have information about their preferences but we still use best guesses based on aggregated data from other contacts in your account. However, for the first ever STO campaign we do not even have best guesses, so in this case the recipients will be randomly distributed among all the available time slots.
In the case of inactive contacts, we know that the times we used previously were not successful so we will try new sending times for them more frequently as these cannot be worse.
STO applies a method called a multi-armed Bayesian bandit algorithm, if you must know.