Import from SFTP to Gigya

Skip to end of metadata
Go to start of metadata

Following is a code sample of a dataflow for importing data from SFTP to Gigya. This sample flow includes extracting a CSV file from SFTP, parsing it to DSV, renaming the fields to a format that matches Gigya's database (in this case the fields are several examples of different newsletters), and finally uploading the transformed data into the Gigya account. You can use this flow as a template and change some of the steps in the script: such as extracting a JSON file rather than CSV. Note that datasource.write.gigya.account is the final step, so any transformation to the data, including transforming a string into Boolean or handling empty fields, should be handled in a previous step. 

Note that IdentitySync jobs are scheduled in UTC time. Therefore, the platform participating in the flow should be set to the UTC timezone to ensure that file requests are handled properly.

For a full outline of the dataflow steps that are available out of the box, see Script Repository

 {  
   "name":"Import from SFTP to Gigya",
   "description":"sftp > dsv > rename > account",
   "steps":[  
      {  
         "id":"sftp",
         "type":"datasource.read.sftp",
         "params":{  
            "host":"",
            "username":"",
            "password":"",
            "remotePath":"",
            "fileNameRegex":".*.csv"
         },
         "next":[  
            "dsv"
         ]
      },
      {  
         "id":"dsv", // Unless inferTypes is set to false, an attempt is made to parse strings to Boolean, long or double (in that order)
         "type":"file.parse.dsv", // If extracting a JSON file, use file.parse.json, instead. 
         "params":{  
            "columnSeparator":","
         },
         "next":[  
            "rename"
         ]
      },
      {  
         "id":"rename",
         "type":"field.rename",
         "params":{  
            "fields":[  
               {  
                  "sourceField":"EVENING_ABC",
                  "targetField":"data.newslettersEvening.abc"
               },
               {  
                  "sourceField":"EVENING_CORREO",
                  "targetField":"data.newslettersEvening.correo"
               },
               {  
                  "sourceField":"EVENING_CSVOCENTO",
                  "targetField":"data.newslettersEvening.csvocento"
               },
               {  
                  "sourceField":"EVENING_DIARIOMONTANES",
                  "targetField":"data.newslettersEvening.diariomontanes"
               },
               {  
                  "sourceField":"EVENING_DIARIOVASCO",
                  "targetField":"data.newslettersEvening.diariovasco"
               },
               {  
                  "sourceField":"EVENING_ELCOMERCIO",
                  "targetField":"data.newslettersEvening.elcomercio"
               },
               {  
                  "sourceField":"EVENING_FINANZAS",
                  "targetField":"data.newslettersEvening.finanzas"
               },
               {  
                  "sourceField":"EVENING_GRADA360",
                  "targetField":"data.newslettersEvening.grada360"
               },
               {  
                  "sourceField":"EVENING_HOY",
                  "targetField":"data.newslettersEvening.hoy"
               },
               {  
                  "sourceField":"EVENING_HOYCINEMA",
                  "targetField":"data.newslettersEvening.hoycinema"
               }
            ]
         },
         "next":[  
            "account"
         ]
      },
      {  
         "id":"account",
         "type":"datasource.write.gigya.account"
      }
   ]
}
  • No labels