This guide walks you through the process of importing new accounts into SAP Customer Data Cloud, and updating existing ones using an import flow. This guide supports the following use cases:
- Migration: importing users that are saved in a previously used system or file, into SAP Customer Data Cloud.
- Ongoing creation of new users: adding new users to your existing SAP Customer Data Cloud database, from a 3rd party system. This can be a one-time or recurring process.
- Updating existing SAP Customer Data Cloud accounts: you may use the import process to update existing accounts, instead of using accounts.setAccountInfo. The reasons for doing so are mainly as follows:
- To update accounts without triggering emails (e.g. welcome emails)
- To update accounts without triggering webhooks* (see below for more information)
- To override existing information instead of appending to the existing data
- To avoid setting default values automatically (e.g. login identifiers) and prevent applying other logic that is part of SAP Customer Data Cloud flows
The import process is performed using a JSON file and sending it to SAP Customer Data Cloud using IdentitySync, our dedicated tool for data transport. This guide walks you through the steps for performing a successful import.
IdentitySync performs data transports using building blocks called "components" to create a step-by-step process, whereby data (a file) is read from a data source, transformed to match the target platform (Customer Data Cloud), and written to the target database using a writer component. In the case of imports, the data should be mostly handled prior to launching the import process.
IdentitySync is launched from the SAP Customer Data Cloud Console, and includes a dashboard for creating and handling dataflows, a scheduler, and tools for monitoring and debugging. It also includes default dataflows, including a dedicated template for imports. You can also create custom scripts using a component called record.evaluate.
Generally, the import flow is described in the following diagram:
Site Data vs. Social Data
It is important to note that Site Data overrides Social Data when merging field values into the profile. Therefore, you should only include fields in the Site Identity that have been previously entered as site fields by the user. Otherwise, if a value was originally collected from a Social Identity and the user later changes it in their social profile, the updated value will not be reflected in their SAP Customer Data Cloud profile.
Accessing Social Data
In most cases, you should continue using any previously configured social network apps. If you create new social network apps, users who have previously given permission to your old app will have to give permission to your new app. In the case of Facebook users, they will not be able to log in at all if a new app is used.
In addition, if you have stored auth tokens for your users and can provide them in the import file, at time of import SAP Customer Data Cloud will automatically download the social data available for users whose auth tokens are still valid. Since authTokens expire after finite amounts of time, we will only be able to download social data for users who have logged in recently (exact timeframe depends on the social network). If you do not include authTokens in your import file, social data will be downloaded for users the next time they log in socially.
Retaining Social Identity Account Links
In order to retain the links between social accounts and site accounts, you should provide us with the providerUIDs from each provider mapping to a siteUID. When the user logs in socially, we will then be able to automatically link them to their site profile. Pre-mapping the providerUIDs to siteUIDs is a better user experience because it avoids asking the user to re-link their accounts when they log in socially. For more information, read about the Identity object.
The user's login identifier(s) is passed in the loginIDs array. SAP Customer Data Cloud supports username or email as the login identifier, as well as any unverified emails. In addition, users may log in using their phone number. For phone number login, an additional accounts.setAccountInfo call is required after the import to set the value of the phoneNumber param.
The import process uses the accounts.importFullAccount API method. Familiarize yourself with this API and its parameters, before starting the import.
Turn off automated email policies prior to starting the import. Otherwise, an error is thrown and the import is blocked.
Only the subscription updated webhook may be fired when importing an account. This will happen only for existing accounts (if a subscription is changed or a new subscription recorded). All other webhooks will not fire when using the import flow.
Importing of SAML and/or OIDC identities is not supported.
The SAP Customer Data Cloud data schema, Identity Store, holds the user data that was captured during interactions with SAP Customer Data Cloud screens and flows. The schema includes the following namespaces:
- profile - out-of-the box fields, that contain all the basic and some extended account information. See the Profile Object documentation for more information.
- data - any custom fields you create, for storing any custom data. If no custom data fields were created, this namespace is empty of fields.
- subscriptions - custom fields, specifically designed to hold subscription information to various communication channels. If no subscription fields were created, or if your licence does not include communication preferences, this namespace is empty of fields.
- preferences - if your implementation includes Consent Management, you will have access to the ‘preferences’ object. A preferences object contains the data pertaining to one consent statement.
- All fields that are required in your schema should be included in the import file. If you are using Customer Identity, and a user is missing any of the required fields, their account registration will stay in a pending state and they will be asked to complete the required fields the next time they log in.
- All required fields must be included in your Registration Completion screens for users to be able to complete their registrations.
- In the default import process, the import job ID is added to the user record for reference. When using this process, add a custom field to your schema for saving the job ID, such as data.gigyaImportJobId (the default name assigned in the import data flow).
- Data types of the source data fields should be matched to the types in the SAP Customer Data Cloud schema.
- Additional data can be imported into the Data Store.
Passwords must be hashed when imported. The following algorithms are supported: "md5", "sha1", "sha1_hashbytes", "sha256", "sha512", "sha512Hexa", "md5_double_salted", "md5_crypt", "bcrypt", "pbkdf2", "pbkdf2_sha256", "drupal", "symphony2", "php_hmac_sha256", "sha1_nordjyske_custom", "md5_variable_salt", and "custom".
If your source passwords are hashed using a different algorithm, you should host an endpoint that can be accessed by SAP Customer Data Cloud. On the first user login after the import, the Gigya service will pass the clear text password to the endpoint, which will return a hashed password.
You can use the following as a template for the custom endpoint:
Make you sure that GET requests are enabled on the endpoint and parameters are sent as URL parameters.
We recommend blocking access to all other applications except the SAP Customer Data Cloud service, by whitelisting the relevant IPs.
Once the endpoint is set up, accounts should be imported in the following format:
Note that the salt property, though stored as base-64, will be sent as clear text.
Following this setup:
- The first time a user logs in, SAP Customer Data Cloud will call the hashing endpoint with the clear text password (and the salt associated with the account if applicable).
- The hashing endpoint will hash the clear text password (with the salt if applicable) and return it to SAP Customer Data Cloud.
- SAP Customer Data Cloud will then compare the hash returned from the endpoint against the one stored in the user account.
- If they differ, the authentication has failed and the user is denied login.
- If they match, the authentication process has succeeded and the user is logged in. Additionally, SAP Customer Data Cloud will hash the clear text password with the pbkdf2_sha256 algorithm and store it in the user's account, so that on subsequent logins, the password submitted by the user will only be compared to the one stored in the SAP Customer Data Cloud account.
You can use the following structure as a basis for building your JSON import file. Remember to adapt the data to your needs.
Setting Up in IdentitySync
Importing accounts is performed using IdentitySync, SAP Customer Data Cloud's dedicated tool for data transfers. The import steps are as follows:
- Open the IdentitySync dashboard in the Console.
- Select Create Data Flow.
- Select Gigya with SFTP and choose the Import Full Accounts From SFTP template.
- The template flow appears in the editor canvas. Edit each step in the flow by double-clicking it and adding mandatory and other params. You can also:
- Rename step names
- Add components from the right-hand menu by searching for them in the search box, then dragging the selected component into the canvas.
- For the Import Account step, in addition to other parameters, select the import policy:
- insert (default): creates a new user. If the user exists, an error is returned.
- upsert: modifies an existing user or creates a new one.
- For detailed information on using IdentitySync, see IdentitySync documentation.
- Save the dataflow.
Tips and Tricks
The following are based on our experts' accumulated experience of performing imports:
- As opposed to legacy import tools, there is no need to pass an "accounts" field for all the fields saved in the accounts object, but rather, specify each of them separately as detailed in the accounts.importFullAccount documentation.
You can use the record.evaluate custom script component for generating random UIDs, based on the following code:
- Rename the IdentitySync steps to include numbering, e.g. instead of "Read files from SFTP", rename to "1. Read files from SFTP". This can help sort the step metrics by numerical order.
- Use the dedicated component datasource.write.gigya.importAccount, and not any other component such as write.gigya.generic or record.evaluate, since the dedicated component better handles requests rate and responses.
- Avoid using the same UID for numerous accounts in the same import job: Since calls to Gigya are done in parallel, the order of the accounts in the import is not guaranteed . Several executions with the same UID may cause data mismatch and account corruption.
- When importing a large batch, first test the import with a smaller batch of records. This should expose common data errors that can be corrected before importing the larger set.
Each imported file should have about 200K accounts at most, to make it easier to follow and monitor the import progress.
If data is not provided - it will not be set. An exception for that rule is groups of fields which go together: if data exists in one of the fields, the whole group will be affected. For example, if a loginID object for an existing account includes a username (and other loginID fields are empty), and the import of that account includes a loginID email (and other loginID fields are empty), the existing username will be deleted and the email written.
To prevent imported users from emails when the import completes, turn off automated email policies prior to starting the import.
- These are the most common reasons that a newly imported account may not function properly:
- The loginIDs array is not populated (the user has no ID for logging in)
- isRegistered was not set to "true", so the user is not considered a registered user
- The password hash was incorrect. Prevent this be checking the hash in advance, before starting the import
- Common error codes:
- 400003 - the user already exists in the system.
- 400006 - invalid parameter value.
- 503001 - this error may display if too many calls were done with the same account UID within a specific time period.