Data Quality:: Smart Insight Data Quality – Emarsys

You can check the status and monitor the quality of your Smart Insight data on the Smart Insight Data Quality page.

If you are using our Batch Sales Data Load solution, then you can check your sales data loads and data quality notifications on the Management > Sales Data Loads page.

We also suggest that you check back on your integration's health by opening the Predict Data Sources tools from Management > Predict Data Sources. Learn more.

Introduction

Data quality is one of the most important aspects of online marketing and e-commerce. Accurate, consistent and reliable data is the key to achieve your business goals. The Smart Insight Data Quality page provides you with up-to-date reports of your contact, product, order and sales data collected by Smart Insight. You can check the quality of your data uploaded to Smart Insight on this page.

The Smart Insight Data Quality page is refreshed after each data load.

The Overview tab

On the Overview tab, you can see reports of Contact, Product and Purchase data uploaded to Smart Insight and their quality. At the top of the page you can see the overall number of Contacts, Products, Orders and Purchased line items uploaded into Smart Insight.

Here you can see the following metrics:

Contacts - The total number of contacts, including identified, unidentified and anonymized contacts.
Products - The total number of products that have ever been uploaded to Smart Insight.
Orders - The total number of individual purchases made.
Purchased line items - The total number of lines (i.e. separate items purchased) in all orders.

The following icons indicate the status of your data quality in each section:

- Means that the specific metric is up-to-date and accurate (i.e. the value is below the specified threshold).
- Means that the specific metric’s threshold has been exceeded.

To display additional information about each metric, expand a specific line by clicking it. Here you can see:

A chart that shows you how this metric changed in the last 30 days,
Detailed description of each metric and
Recommendations describing how you can resolve an issue.

Contact data

You can see the following metrics in the Contact data section:

Last contacts load - The date of the last successful contact data load.

If this date is older than 1 day, it means that some of your contacts have corrupted data and your contact load is failing. If contact data cannot be loaded regularly, then contact information will become outdated and you cannot import new contacts into Smart Insight. This could be a possible reason for an increasing number of Unidentified contacts.

Unidentified contacts - The number of contacts who have not yet been added to your Emarsys database but who had made purchases according to your sales data.

You cannot send campaigns to these customers, regardless of the amount of their purchases.

You can export the data into a .CSV file. To do so, expand the Unidentified contacts line and click the Export button.

Contacts with negative turnover - The number of contacts whose total lifetime spend is negative, which means that these customers appear to have refunded more items than they have bought.

If there is a significant amount of contacts with negative turnover, then Smart Insight may be missing purchase data, which would adversely affect your lifecycle marketing.

You can export the data into a .CSV file. To do so, expand the Contacts with negative turnover line and click the Export button.

Outlier purchases - The percentage of purchases made by a single customer who has made the most purchases based on your sales data.

If this percentage is high, it could mean that all Unidentified purchases are attributed to the same person, which would distort your analytical reports.

You can export the data into a .CSV file. To do so, expand the Outlier purchases line and click the Export button.

Product data

You can see the following metrics in the Product data section:

Last product load - The date of the last successful product data load.

If this date is older than 1 day, it means that your product load is failing or you are not uploading your catalog to Smart Insight. If product data cannot be uploaded regularly, then product information will become outdated and new products will not be imported into Smart Insight. This could be a possible reason for an increasing number of Unknown products.

Unknown products - The number of products that have been sold but are not in the product catalog. These products will be displayed as Unknown Product on the Smart Insight reports.

You can export the data into a .CSV file. To do so, expand the Unknown products line and click the Export button.

Products with no category - The number of products that do not belong to any product category. These products will be displayed as Missing category on the reports and you will not be able to filter the customers of these products by category.

You can export the data into a .CSV file. To do so, expand the Products with no category line and click the Export button.

Products with no name - The number of products that have an empty name in the product catalog. These products will appear with an empty label on the reports and you will not be able to filter these products (or their customers) by name.

You can export the data into a .CSV file. To do so, expand the Products with no name line and click the Export button.

Purchase data

You can see the following metrics in the Purchase data section:

Duplicated item purchases - It means that your sales data file contains duplicate lines with the same contact ID, product ID, order ID, quantity, price and purchase date.

Importing the same purchases multiple times will distort your lifecycle analytics and buyer status calculations.

Whats the difference between Duplicated item purchases and Orders with duplicated line items?

The metrics Duplicated item purchases and Orders with duplicated line items are closely related:

Duplicated item purchases counts all lines with the same contact ID, product ID, order ID, quantity, price and purchase date in your sales data file.
Orders with duplicated line items displays (as described below) the number of orders with duplicate line items having the same order ID in your sales data file.

The following sales data excerpt contains 2 Orders with duplicated line item sand 5 Duplicated item purchases:

order, timestamp, customer, item, price, quantity
2123, 2022-09-08, abc@sap.com, xyz, 12.00, 2
2123, 2022-09-08, abc@sap.com, xyz, 12.00, 2
3123, 2022-09-20, xzy@sap.com, trw, 32.00, 1
3123, 2022-09-20, xzy@sap.com, trw, 32.00, 1
3123, 2022-09-20, xzy@sap.com, trw, 32.00, 1

You can export the data into a .CSV file. To do so, expand the Duplicated item purchases line and click the Export button.

Last purchase load - The date of the last successful purchase data load.

If this date is older than 1 day, it means that your purchase load is failing or you are not uploading your sales files to Smart Insight. If sales data cannot be loaded regularly, then customer lifecycle information will become outdated, which affects both your analytics and marketing activity.

Latest purchase date - The date of the most recent purchase.

If your latest purchase is more than 2 days old and you are uploading daily sales files, then your purchase load might be failing.

Oldest purchase - The date of the oldest purchase.

Smart Insight lifecycle calculations are optimized to be run on at least 2 years’ worth of purchase history, otherwise your lifecycle analytics might not be accurate.

Orders with duplicated line items - It means that your sales data file contains duplicate line items with the same order ID. This metric groups duplicate line items by their order IDs. For example, if you have two identical line items with the same order ID, it will count as one order with duplicate line items.

Importing the same orders multiple times will distort your lifecycle analytics and buyer status calculations.

You can export the data into a .CSV file. To do so, expand the Orders with duplicated line items line and click the Export button.

Future purchase dates - The number of purchases that have a purchase date in the future.

These purchases can distort your analytics results.

You can export the data into a .CSV file. To do so, expand the Future purchase dates line and click the Export button.

Missing purchase days - The number of individual days when no purchases were made. Missing days in purchases mean that either you have not uploaded any sales data for those days or Smart Insight could not load your uploaded files.

Missing purchases have the potential to render your customer lifecycle modelling incorrect and thus, adversely affect your lifecycle marketing.

You can export the data into a .CSV file. To do so, expand the Missing purchase days line and click the Export button.

Missing purchase periods - The number of time periods (one or more consecutive calendar days) when no purchases were made according to the sales data in Smart Insight. Missing periods in purchases mean that either you have not uploaded any sales data for that period or Smart Insight could not load your uploaded files.

Missing purchases have the potential to render your customer lifecycle modelling incorrect and thus, adversely affect your lifecycle marketing.

You can export the data into a .CSV file. To do so, expand the Missing purchase periods line and click the Export button.

Orders with multiple contacts - The number of orders that occur in the purchase history of multiple contacts at the same time. Smart Insight expects each order to have been placed by one person only.

If multiple people share the same order, then customer lifecycle modelling will be incorrect, which will adversely affect your lifecycle marketing and revenue impact reporting.

You can export the data into a .CSV file. To do so, expand the Orders with multiple contacts line and click the Export button.

Outdated purchases - Purchases that were made before the year 2000.

Purchases this old might distort your analytics results and are irrelevant for Smart Insight lifecycle calculations.

You can export the data into a .CSV file. To do so, expand the Outdated purchases line and click the Export button.

Editing the thresholds

You can adjust the threshold for each Smart Insight data quality metric. When the threshold of a metric exceeds the specified value, a warning will be displayed next to it.

To edit the threshold of your Smart Insight data quality metrics, proceed as follows:

1. Click the Edit Thresholds button at the bottom of the page.

2. Set the required values.

You can disable warning messages for specific metrics by checking the Ignore checkbox. If you activate this feature, then you will not receive warning messages when the selected metric exceeds the specified threshold.
If you select a metric to be ignored, then the export option will be disabled. In case of an individual metric the icon is replaced by .

3. Once you are ready, click Apply at the bottom of the page.

The Data Loads tab

For more information, see Data Loads.

Sending email notifications

You can subscribe to email notifications that you will receive when there is a new Smart Insight data quality warning or a data load error occurs in your daily load on the Notifications > Settings > Data Management Notifications page.

For more information, see Custom notification settings and Data management notifications.