Keeping your product data consistent between Emarsys and your webshop is essential to ensure that the content of your messages and their recommendations matches what is actually available online.
Please read this article before you set up your regular data synchronization schedule, and make sure that your product data file is formatted according to the specifications and guidelines described below.
Important: Your product data file always replaces the entire product catalog each time it is imported. All data is overwritten, rather than updated, and if products are left out of the file, they will no longer be available in the Emarsys product database.
Before you start
What is product data?
Product data is all the information on all the products you sell, and have ever sold, on your website. It is usually kept in one file (also known as the product catalog) on a server hosted by you and synchronized daily with Emarsys.
Your product data file can be monitored and modified on the Management > Predict Data Sources > Product Data page.
This is how the elements of a product page would translate into Emarsys product data fields.
Why is product data important?
Your product data is required by most of the Emarsys products and features, including Predict and Smart Insight.
Accurate product data enables you to target specific customer segments on the basis of which products they bought, viewed or left in their shopping carts, or send them dynamic, tailor-made product recommendations.
How much effort do I need to invest in this?
This depends on the product data file format you are going to use. We support two formats:
- Emarsys CSV format
- Google Product Feed (GPF)
If you use the Emarsys CSV format you will need to generate an output file that conforms to our specifications, described below and in our CSV guidelines. Depending on the size and complexity of your product catalog, it may take up to a few hours for your data specialist to deliver this file. Then, if our validation engine detects some issues, you will need to fix these first, which may take another couple of hours of your data specialist’s time.
Here is an estimate of how much time you may require to set up your product data file:
Action |
Role | Max effort |
|---|---|---|
| Generating product data | Data specialist | Up to 6 hours |
| Validating your product data file | Data specialist | 5 mins |
| Fixing product file, if required | Data specialist | Depends on the number and severity of issues, but usually no more than 2 hours |
| Scheduling regular updates | Marketer/Data specialist | 5 mins |
If you already have a Google Product Feed (GPF), you may be able to skip these steps as your GFP could be readily usable in Emarsys. However, make sure that your GPF contains all the required fields. If you are unsure, just contact Emarsys Support and we will check your file.
WEBP image format is not supported.
Do not use Chroma subsampling on your product image files. This compression technique is not supported and your images will not be show in recommendation widgets.
Who do I need to involve from my side?
- If you use the Emarsys CSV format, you will need a data specialist to provide the product data and, optionally, a product owner to oversee the project.
- If you reuse your Google Product Feed, all you may need is a link to your GPF file, which is something your data specialist should know and be able to provide.
What should I include in my product data file?
We strongly recommend to include all the products that have ever been sold on your website, even if they are no longer available (simply set the corresponding value in the available field).
Including past products and categories will help to build a more accurate and robust affinity model from your web behavior, and will reduce the risk of your product data becoming corrupted or inconsistent.
Orders also typically contain a number of service items such as delivery costs and method, packaging, free gifts or promotions, etc. Emarsys will not be able to distinguish between these and actual products in the order, so you should make sure that they are all listed in your product data file as unavailable products. This will prevent them from ever being offered as recommendations.
How often do I need to upload product data?
We recommend to upload your product data at least once a day. You may want to consider doing it more frequently, if important fields such as price or availability are likely to change during the day.
General rules for your data file
- Your product data file should be a simple .csv table, with each row containing all the attributes of a given product, such as the item ID, the URL of the product page, a link to a product image, etc.
- Take into consideration that values are case sensitive. The identifiers
Women_Dress_100andwomen_dress_100are treated as individual. - Use a CSV export library that is compatible with the standard CSV format.
- Use a comma to separate fields in a row.
- In fields containing multiple values, use a pipe (|) to separate the values. Example:
…,John Lennon|Paul McCartney|George Harrison|Ringo Starr,... - When a field does not have any value, leave it empty. Example:
value1,,value3,value4 - Do not use newline characters, either unquoted or quoted, in the fields.
- Use UTF-8 encoding.
- Include the field names in the first row (header) and make sure that all the subsequent rows contain the values corresponding to the fields listed in the header.
- Do not use more than 255 columns.
- Do not use field values longer than 64 KB.
- Redundant (starting and trailing) white spaces are automatically trimmed unless they are quoted (e.g. 1, 2, 3 is the same as 1,2,3).
- Even though your product data file normally contains information that is publicly available on your website, you may want to add a level of security by password-protecting the storage where you upload it. Emarsys fully supports uploading product data files to a secure, password-protected storage, based on your location.
The product file should contain only letters, numbers, underscore, dash and slash characters. Slash is solely for the g/ prefix used in the Parent/Child feature.
Personalization does not support and cannot process null values in the Product data files. In case of an occurring null value, personalization would use the last not-null value, which is not correct.
As a workaround, you can either use a numeric 0 or an empty string '' character.
Need an example? Click here to see how a product data file looks like in the Emarsys CSV format or download a sample file.
File size
- If your product data file contains more than a million items, please contact Emarsys Support before importing it.
- If your product data file runs to several hundred megabytes in size, you may consider compressing it.
- You can upload gzip compressed product data files to Emarsys:
- Compress the file with GZIP and update the URL so that the file has the .gz suffix.
Product categories
In the category field, you submit the product category path of your items according to the product categorization system of your online store.
Please pay particular attention to the consistency of product categories in your web shop and in your product data file.
A common cause of errors (and therefore delays) when onboarding product data is that the product data file does not have the same understanding of categories as the Web Extend data collection script category command.
A category path describes the location of each product in the category hierarchy. It is a >-separated list of subcategories that starts with the top-level category.
For example, a small bicycle store may have the following category hierarchy:
Bikes
Road Bikes
Mountain Bikes
Parts
Road
Tyres
Chains
MTB
Tyres
ChainsIn this structure, Bikes and Bikes > Road Bikes are both valid category paths, while Road Bikes in itself is not, since it is not a top-level category.
The category path for bike1, an awesome road bike, is Bikes > Road Bikes, which can be submitted in your product data file like this:
item, title, category,...
bike1, Awesome Road Bike, Bikes > Road Bikes, …Emarsys Predict supports an unlimited number of category levels, while Smart Insight can handle a maximum of five levels. If there are more than 5 levels, Smart Insight keeps the top 5 and ignores the rest.
Items with multiple categories
Sometimes you may want to categorize products into multiple categories. For example, our bike shop sells pedal1, a universal hybrid pedal, which works on both road bikes and mountain bikes. As such, it belongs to both Parts > Road > Pedals and Parts > MTB > Pedals, so pedal1 belongs to a list of categories. The elements of lists should be separated by a | (pipe) within the category field.
item, title, category, ...
pedal1, Universal Hybrid Pedal, Parts > Road > Pedals | Parts > MTB > Pedals,...Standard product data fields
The Parent/Child catalog feature allows you to handle two-tier product structure and use that data for recommendations and personalization. For more information, see Parent/Child catalog feature.
Field |
Required | Type | Example | Description |
|---|---|---|---|---|
item |
Yes | String | BOOK0012 | - The unique ID used in your web store to identify the product. |
- Do not use the same item_id more than once in your product file.If the same item_id is used multiple times then only one of those lines will be processed, the rest of the duplicates will not be considered and a yellow warning message will notify you on this in the Product Catalog page. | ||||
| - Do not include SKU-level product variants in the product data file. A good rule of thumb is to consider an item unique if it has a unique URL in your online store. | ||||
- Do not include semicolons (;) in this field, as they are not supported. | ||||
title |
Yes | String | The Hitchhikers guide | - The name of the product. |
| - Do not include any promotional text. | ||||
link |
Yes | URL | http://www.buy.me/hitch/ | - The URL of the item's product page. |
| - Start with the protocol, such as http or https. | ||||
- URLs (for example, in the link field of the product catalog) should be properly URL encoded with the following exception: fields used in Predict are encoded automatically, so you don’t need to encode these.Note: Please use only ASCII characters to avoid compatibility issues with iOS devices. | ||||
image |
No | URL | http://www.buy.me/hitch_thumb.jpg | - The URL for the product image. Note: Please use only ASCII characters to avoid compatibility issues. The linkfield of the image URL should be properly URL encoded. |
| - Start with the protocol, such as http or https. | ||||
| - Our best practice recommendation is to use an 800-pixel wide image to ensure that it looks good on all displays. | ||||
| - Do not use images larger than 1 MB or 1280 x 1280 px. | ||||
- For high-resolution product images, use the zoom_image field. | ||||
| - If you only have high-res product images, we can create the thumbnails for you automatically. Learn more. | ||||
| - WEBP image format is not supported. | ||||
zoom_image |
No | URL | http://www.buy.me/hitch_fullres.jpg | - The URL for the high-resolution product image. |
| - Start with the protocol, such as http or https. | ||||
| - WEBP image format is not supported. | ||||
category* |
No | Click here for details | Fiction > Sci-Fi | English literature > Sci-Fi | The list of category paths in which the given item belongs. |
available |
No | Boolean | true | - The current availability of the product in your store. |
- Use either lower case (true) or all-caps (TRUE). | ||||
| - Make sure to keep this field up to date. | ||||
| - If possible, include all the items you have ever offered on your website. This will improve the analysis of historical data and makes it possible for Emarsys to recommend available alternatives for items no longer for sale. | ||||
- Submit products currently unavailable as false. | ||||
description |
No | String | Douglas Adams' seminal work of fiction. | A brief description of the product. |
price |
No | Float | 13.99 | - The current product price you display to your customers in your web store. |
| - Do not include the currency. | ||||
| - Always use the period (.) as the decimal mark. | ||||
| Localized price values are strings in ISO localized monetary format usually with 2 digit precision and rounding. | ||||
msrp |
No | Float | 17.99 | - The original price of the product (i.e. the price before discounts). |
- msrp must be higher than price. If msrp is equal to it or lower, it will not be shown in email recommendations. | ||||
| - Do not include the currency. | ||||
| - Always use the period (.) as the decimal mark. | ||||
brand* |
No | String | Nike | The brand name(s) of the product. |
* In this field, you can use multiple values.
Loyalty fields
The Loyalty add-on enables managing loyalty campaigns to improve customer retention and customer lifecycle value. For more information, see Emarsys Loyalty - Overview and Implementing Loyalty.
The following fields are required to fully support catalog promotion:
| Field | Required | Type | Example | Description |
|---|---|---|---|---|
brand |
Yes | String | Calvin Klein | The brand of the item |
category |
Yes | String |
Shirts | The category of the item. An item can be in more than one category. In case of multiple categories, separate the categories using the pipe delimiter (|). |
Other predefined fields
Field |
Required | Type | Example | Description |
|---|---|---|---|---|
album* |
No | String | Let it be | The title of the album(s) on which the product was released. |
actor* |
No | String | Brad Pitt | The actor(s) associated with the product. |
artist* |
No | String | Janis Joplin | The artist(s) associated with the product. |
author* |
No | String | Douglas Adams | The author(s) associated with the product. |
year |
No | Integer | 1984 | The publication year of the product. |
* In these fields, you can use multiple values.
Custom fields
Besides the standard and predefined product data fields, you can also add other product information to your data file by using custom fields.
When deciding on custom fields, think about the kind of segmentation you would like to have available in your Emarsys account, then work back to identify the fields that you need in order to achieve those goals.
This is particularly relevant if you are using Smart Insight, as these fields will enable you to create segments based on product attributes.
Naming conventions and limitations
When naming your custom fields make sure to apply the following regulations:
Do:
- The names of custom fields have to begin with the
c_*prefix (e.g.c_colororc_size).
Do not:
- Custom field names cannot begin with a number.
- Do not include any of the predefined field names in any part of the names of custom or optional fields.
e.g. Avoid having a custom field name:c_available_colorsasavailableis a standard field.
If optional field names start with a predefined field name instead of thec_*prefix, then they will be added as localized fields to your product data file.
Custom fields are not validated by Emarsys.
Field localization
If you run a web store in multiple markets or languages, it may be necessary to also include the localized versions of standard product data fields in your product data file. Name your localized fields using the pattern [field name]_[language identifier], that is, the name of the standard product data field must be followed by an underscore and the language identifier (for example, title_it for the Italian title of a book).
All standard and predefined product data fields can be localized, except for item.
Localized fields will be of string type.
In case there are multiple localised catalogs remember that the localized category structure needs to match exactly the category structure and order in the default category field.
To learn more about localization, see our article on Localized Websites.
Disambiguation
Since several product attributes, such as categories, authors and actors, can be used to build statistical models, it is important to use them canonically. For example, the author J. K. Rowling should always be spelled the exact same way in the product data file. When there are more identical strings representing different values of a given attribute, they should be disambiguated consistently, for example J. Doe and J. Doe(2).
Optimizing product data images
Image optimization function at Product Data page ensures that your uploaded GPF product data image is automatically tailored to have the recommended size parameters. For more information, see Optimizing GPF product data image size.
Image optimization function may distort your image.
You may turn Image optimization function off, however, turning it off may cause that large size images may not appear. You can also reduce the image size by Resizing images with the Emarsys image proxy.
Resizing images with the Emarsys image proxy
Certain Emarsys apps, such as product recommendations, work properly with smaller images. If you only have high-resolution product images, Emarsys can create their thumbnail versions for you. To use this automatic image resizing service, simply replace the image URL with the following URL pattern in the image field:
https://imgfly.scarabresearch.com/w_<thumbnail image width in pixels>/<image URL>Make sure to start the URL of the original image with the protocol, which can be either http or https, and to use the standard URL syntax.
Our best practice recommendation is to use 800-pixel wide images to ensure that they look good on all displays.
Examples
This is how a product data file looks like in the Emarsys CSV format. Click here to download a sample file.
item,link,title,image,category,price,msrp,available,brand,description,zoom_image
103,http://www.lifestylelabels.com/steven-by-steve-madden-pryme-pump.html,"elegant red pump",http://www.lifestylelabels.com/media/catalog/product/cache/1/small_image/210x/9df78eab33525d08d6e5fb8d27136e95/s/t/steven-by-steve-madden-pryme-pump.jpg,"Shoes > Pump",299,0,true,"Steve Madden","Nothing will turn his head faster than you wearing the sexy Pryme pump from Steven by Steve Madden. This daring pump has a pretty patent leather upper with light shirring, a double stitch detail surrounding the collar, and a vampy almond shaped toe.",http://www.lifestylelabels.com/media/catalog/product/s/t/steven-by-steve-madden-pryme-pump.jpg
108,http://www.lifestylelabels.com/nine-west-women-s-lucero-pump.html,"glittering red pump",http://www.lifestylelabels.com/media/catalog/product/cache/1/small_image/210x/9df78eab33525d08d6e5fb8d27136e95/n/i/nine-west-women-s-lucero-pump.jpg,"Shoes > Pump",289.99,0,true,"Nine West","The Lucero pump from Nine West may just leave him at a loss for words. This flirty pump has a leather upper, a pretty almond-shaped toe with a slight V-cut vamp, leather linings, and a cushioned insole for long-wearing comfort.",http://www.lifestylelabels.com/media/catalog/product/n/i/nine-west-women-s-lucero-pump.jpg
112,http://www.lifestylelabels.com/womens-golf-shoes.html,"womens golf shoes",http://www.lifestylelabels.com/media/catalog/product/e/c/ecco-womens-golf-flexor-golf-shoe.jpg,"For Women > Shoes > Golf",159,0,false,"Ecco","Sporting with style, this is a durable and super-comfortable golf shoe built for performance.",http://www.lifestylelabels.com/media/catalog/product/e/c/ecco-womens-golf-flexor-golf-shoe.jpgThe above file without white spaces:
item,link,title,image,category,price,msrp,available,brand,description,zoom_image
103,http://www.lifestylelabels.com/steven-by-steve-madden-pryme-pump.html,"elegant red pump",http://www.lifestylelabels.com/media/catalog/product/cache/1/small_image/210x/9df78eab33525d08d6e5fb8d27136e95/s/t/steven-by-steve-madden-pryme-pump.jpg,"Shoes>Pump",299,0,true,"Steve Madden","Nothing will turn his head faster than you wearing the sexy Pryme pump from Steven by Steve Madden. This daring pump has a pretty patent leather upper with light shirring, a double stitch detail surrounding the collar, and a vampy almond shaped toe.",http://www.lifestylelabels.com/media/catalog/product/s/t/steven-by-steve-madden-pryme-pump.jpg
108,http://www.lifestylelabels.com/nine-west-women-s-lucero-pump.html,"glittering red pump",http://www.lifestylelabels.com/media/catalog/product/cache/1/small_image/210x/9df78eab33525d08d6e5fb8d27136e95/n/i/nine-west-women-s-lucero-pump.jpg,"Shoes>Pump",289.99,0,true,"Nine West","The Lucero pump from Nine West may just leave him at a loss for words. This flirty pump has a leather upper, a pretty almond-shaped toe with a slight V-cut vamp, leather linings, and a cushioned insole for long-wearing comfort.",http://www.lifestylelabels.com/media/catalog/product/n/i/nine-west-women-s-lucero-pump.jpg
112,http://www.lifestylelabels.com/womens-golf-shoes.html,"womens golf shoes",http://www.lifestylelabels.com/media/catalog/product/e/c/ecco-womens-golf-flexor-golf-shoe.jpg,"For Women>Shoes>Golf",159,0,false,"Ecco","Sporting with style, this is a durable and super-comfortable golf shoe built for performance.",http://www.lifestylelabels.com/media/catalog/product/e/c/ecco-womens-golf-flexor-golf-shoe.jpgNote that Emarsys considers both files exactly the same as redundant white spaces and redundant quoting are ignored. Also note that values containing the separator character must be enclosed in double-quotes (e.g. "Foo, Bar").
FAQ
Which formats are supported for the product catalog?
Product catalogs need to contain the data prescribed in our guidelines. We support both our own .csv format and the Google Product Feed format, but you will need to ask Emarsys Support to enable GPF.
Can I store my product catalog on an SFTP server?
Yes, you can use SFTP.
Currently, we support the following protocols:
- SFTP
- FTPS (FTP over SSL/TLS) are supported. These must start with sftp://.
How should I manage online and offline product data in Smart Insight?
If you have products that you only sell offline or online, then you can choose from the following options:
- You can prepare a single product feed containing both your online and offline products and use this catalog for both Predict and Smart Insight.
- You can prepare one product feed containing online products and use it for both Predict and Smart Insight and prepare another product catalog with offline products that you upload to the Smart Insight SFTP manually.
We recommend using this method only if your offline products do not change often (i.e. once every quarter).
- You can prepare 2 product feeds:
a. One product catalog containing online products that will be used for Predict.
b. One product catalog containing both online and offline products that will be uploaded to Smart Insight.
If you have many offline products in your product catalog, then we don't recommend including them in the product catalog used for Predict because it is updated every day (sometimes it might be updated multiple times a day).