Epsilon Dataflow

Skip to end of metadata
Go to start of metadata

Integration with Epsilon is done by executing a dataflow that extracts data from Gigya and uploads a json file to SFTP. The process includes steps for field renaming, replacing non-printable characters with a space or removing them altogether, splitting the flow into two so as to create an empty file, uploading the JSON file, and, after the successful upload, uploading an empty file of the same name to indicate that the file upload was completed successfully. For a full explanation of these steps, see Script Repository.

Note that IdentitySync jobs are scheduled in UTC time. Therefore, the platform participating in the flow should be set to the UTC timezone to ensure that file requests are handled properly.

Following is a code sample of a dataflow for exporting data from Gigya to SFTP in an Epsilon-compatible format: 

{
    "id" : "",
    "name" : "Epsilon Dataflow",
    "steps" : [{
            "id" : "account",
            "type" : "datasource.read.gigya.account",
            "params" : {
                "select" : "UID,profile,data"
            },
            "next" : ["rename"]
        }, {
            "id" : "rename",
            "type" : "field.rename",
            "params" : {
                "fields" : [{
                        "sourceField" : "profile.gender",
                        "targetField" : "'profile.gender'"
                    }, {
                        "sourceField" : "profile.birthYear",
                        "targetField" : "'profile.birthYear'"
                    }, {
                        "sourceField" : "data.user.cableProviderName",
                        "targetField" : "'data.user.cableProviderName'"
                    }, {
                        "sourceField" : "data.user.location",
                        "targetField" : "'data.user.location'"
                    }, {
                        "sourceField" : "data.user.masterID",
                        "targetField" : "'data.user.masterID'"
                    }, {
                        "sourceField" : "data.personas.Food.aboutMe",
                        "targetField" : "'data.personas.Food.aboutMe'"
                    }, {
                        "sourceField" : "data.personas.Food.myFavorites",
                        "targetField" : "'data.personas.Food.myFavorites'"
                    }, {
                        "sourceField" : "profile.education.degree",
                        "targetField" : "'profile.education.degree'"
                    }, {
                        "sourceField" : "profile.education.endYear",
                        "targetField" : "'profile.education.endYear'"
                    }, {
                        "sourceField" : "profile.education.fieldOfStudy",
                        "targetField" : "'profile.education.fieldOfStudy'"
                    }, {
                        "sourceField" : "profile.education.school",
                        "targetField" : "'profile.education.school'"
                    }, {
                        "sourceField" : "profile.education.schoolType",
                        "targetField" : "'profile.education.schoolType'"
                    }, {
                        "sourceField" : "profile.education.startYear",
                        "targetField" : "'profile.education.startYear'"
                    }, {
                        "sourceField" : "profile.educationLevel",
                        "targetField" : "'profile.educationLevel'"
                    }, ]
            },
            "next" : ["remove"]
        }, {
            "id" : "remove",
            "type" : "field.remove",
            "params" : {
                "fields" : ["profile", "data"]
            },
            "next" : ["replace"]
        }, {
            "id" : "replace", // Replace the "\n" line break with a space, and remove all non-ASCII characters
            "type" : "field.replace",
            "params" : {
                "fields" :[
					{
						"field" : "*",
						"regex" : "(\\|)|(\n)|(||)",                
						"replacement":" "
					},
					{ 
						"field":"*",
						"regex":"[^\\x00-\\x7F]",
						"replacement":""      
					}
				]
            },
            "next" : ["json", "empty"] // Specify two next steps to split the flow for an SFTP upload scenario
        }, {
            "id" : "json",
            "type" : "file.format.json",
            "params" : {
                "fileName" : "gigya-${now}.json"
            },
            "next" : ["sftp"]
        }, {
            "id" : "empty",
            "type" : "file.empty",
            "params" : {
                "fileName" : "gigya.ok"
            },
            "next" : ["sftp"]
        }, {
            "id" : "sftp",
            "type" : "datasource.write.sftp",
            "params" : {
                "host" : "",
                "username" : "",
                "password" : "",
                "temporaryUploadExtension" : true,
                "remotePath" : ""
            }
        }
    ]
}
  • No labels