idx.search REST

Skip to end of metadata
Go to start of metadata

Description

Searches and retrieves data from IdentitySync using an SQL-like query.

For an introduction to IdentitySync, read the IdentitySync guide.

Please note:

  • SQL queries are converted into Gigya's proprietary query language. SQL injection attacks are not possible because queries are both created by the customer and then converted by Gigya.
  • A short delay is possible between the writing of account data and its availability in queries.

Usage Examples

idx.search can be used to check on your job's status, as well as to retrieve saved dataflows, scheduling objects, scripts, and artifacts.

Examples:

TaskQuery Syntax
To retrieve all dataflows stored in the system
SELECT * FROM dataflow

To retrieve all schedulings that have been set for a specific dataflow:

SELECT * FROM scheduling WHERE dataflowId='ec736fd67e834895bf72493ec48e3c66'

To retrieve the job status of the latest 5 jobs that are on a specific dataflow:

SELECT * FROM idx_job_status WHERE dataflowId='ec736fd67e834895bf72493ec48e3c66' ORDER BY updateTime DESC LIMIT 5

To look at the details of a specific script:

SELECT * FROM script WHERE id='GigyaAccountsExtractor'

Request URL

http[s]://idx.<Data_Center_ID>.gigya.com/idx.search

Where <Data_Center_ID> is:

  • us1 - For the US data center
  • eu1 - For the European data center
  • au1 - For the Australian data center
  • ru1  - For the Russian data center

Parameters

RequiredNameTypeDescription
querystringAn SQL-like query specifying the data to retrieve. Please refer to the Query language specification section above.
openCursorBoolean

When set to true, the search response will include, in addition to the first page, another field named nextCursorId, which is used to fetch the next batch of results. This parameter should only be used on the first request and later should be removed from the request. When openCursor is active, the Limit clause sets the number of results returned in the batch.

Note: 

  • When using a cursor with a Limit set, the number of results in a batch is not guaranteed. 
  • You cannot use a cursor if you have a group by or when using start.
cursorIdstring

The cursor ID that contains the nextCursorId value received in the first search call.

Note: 

  • You cannot pass both cursorId and query on the same request - cursorId brings the next page for the search for which it was opened. Also, the time between search requests using a cursorId must not exceed 5 minutes.
  • Each request should contain a different cursorId obtained from the response of the previous request (not the first) using the nextCursorId field. The exception to this rule is when a request fails or when a particular result set needs to be resent; in this case, resend the same cursorID (as long as it has not expired) to receive its associated result set.


Query Syntax

Gigya queries use the same syntax rules as SQL, but not all standard SQL key words are available. Unsupported SQL syntax in the query string, such as "HAVING", will produce an error.

The query string clauses must appear in the following order (Queries ordered differently will produce an error):

SELECT ... FROM ... WHERE ... FILTER ... GROUP BY ... ORDER BY ... START / LIMIT

SELECT

The SELECT statement accepts a comma-separated list of fields or objects to retrieve.

Acceptable values for this statement are:

  • Field names: specify the complete path to retrieve the specific field.
  • Partial field names: specify only a part of the path to sub-objects, i.e., data.album, to retrieve everything below that path.

  • Object names: specify an object type to return all the fields in the object.

  • *: Use the wildcard to retrieve every field in the schema.

  • COUNT(*)

  • SUM(), MIN(), MAX(), AVG(), SUM_OF_SQUARES(), VARIANCE(), STD() -  Mathematical functions, must all be performed on the same numeric field. Fields with null values will be ignored. The name of the field on which the function is to be performed must be entered in the brackets.

Use AS to create an alias (temporary title) for the returned object or field. For example: SELECT id AS jobID FROM...

FROM

Name of the data source. Only one data source is supported.

The available data sources are:

Data SourceDescription
dataflowSearch the stored dataflows in the system.
schedulingSearch the scheduling objects that have been saved in the system.
idx_job_statusRetrieve the status of a specific transfer job or jobs (see JobStatus object).
scriptSearch the Script Repository.

WHERE

The "where" clause defines conditions for selecting items from the collection.

Note:

  • String values must be wrapped in double quotes, e.g., SELECT * FROM dataflow WHERE name = "John Doe".

  • Integer and other non-textual values must not be wrapped in quotes. e.g., SELECT * FROM accounts WHERE age = 42.

Supported operators:

  • >=<,  <=  = != - the left operand must be a data field name (with a proper suffix letter) and the right operand must be a constant of the same type as the field. For example: "WHERE profile.age >= 18 ".

  • AND, OR

  • CONTAINS, NOT CONTAINS - may be used only on text (string) fields and arrays. 

    • Text (string) fields - supports standard full text search capabilities. Contains is not case sensitive. The left operand must be a text field name and the right operand must be a constant string. You can search for a specific word within the string, for example: 'WHERE data.about_t CONTAINS "music" '. Underscores are treated as separators between words.
      If you want to perform a search on an encrypted field, you must enter the full string. Encrypted fields include the fields you encrypted and also fields that Gigya encrypts
    • Arrays -  the left operand must be an array field name and the right operand must be a constant of the same type as the array values. The array field name must have a suffix denoting the type of the arrays values. For example: 'WHERE data.hobbies_s CONTAINS "swimming" '.
    Note: You can only search words that are part of a sentence, you cannot search for parts of a word. 
  • IN() - only retrieve items if the field contains one of the list values. For example: 'SELECT * FROM accounts WHERE profile.firstName IN ("Frank", "Dean", "Sammy")' will return users with the specified first names. 

  • IS NULL, IS NOT NULL

  • NOT

  • REGEX('<pattern>') - defines a search term using regex formatting. The regex syntax can be found in: 'Regular Expression Language Reference.

ORDER BY

Use ORDER BY to enter a list of fields by which to sort the result objects.

LIMIT / START

Use LIMIT to specify the maximum number of returned result objects. If not specified, the default is 300. The maximum limit value accepted is 10000. If the search is sent with openCursor = true, limit will set the batch size. Please note, when using a cursor, the number of results in a batch is not guaranteed.

Use START (not an SQL standard clause) for paging. The clause specifies the start index from which to return result objects.

The 'select - from - where - order by' query creates an (internal) indexed list of objects. By using the start and limit clauses, you will receive a subset of this list, starting with the start index and ending with start+limit index. You can not use start and openCursor in the same query.

Note: When implementing paging, there is no guarantee against duplications or that recently added data will show up in the query results.

COUNTERS

Counter data is only available if the data source statement ("from") specifies it, i.e., "FROM accounts WITH counters". For example: ' SELECT * FROM accounts WITH counters LIMIT 5'. To view only the counter fields, 'SELECT counters FROM accounts WITH counters'. To specify a particular field in the counters object, use "counters.class" or "counters.path" etc. Note that counter data is not retrievable by searches until 1-24 hours after it is written.

Two SQL keywords exist for use with counter fields (they are not available with other fields): ifElement is used for setting multiple conditions on a single counter's different fields and filter counters is used to restrict the counters returned with an account object.

  • filter counters by class="<class name>" - Restricts the counter types returned with the account object to those listed in the filter statement. For example: filter counters by class="shares", returns only the shares counters.

  • ifElement(counters<if statement relating to different elements of a single counter>- This function allows you to apply conditions to several elements within the same counter, for example: ifElement(counters, class="purchases" and period = "total" and value > 100). When true the ifElement expression returns 1.

Authorization Parameters

Each REST API request must contain identification and authorization parameters.

See Authorization Parameters Overview for details. 

 

Response Data

FieldTypeDescription
errorCodeintegerThe result code of the operation. Code '0' indicates success, any other number indicates failure. For a complete list of error codes, see the Error Codes table.
errorMessagestringA short textual description of an error, associated with the errorCode, for logging purposes. This field will appear in the response only in case of an error.
errorDetailsstringThis field will appear in the response only in case of an error and will contain the exception info, if available.
callIdstringUnique identifier of the transaction, for debugging purposes.
timestringThe time of the response represented in ISO 8601 format. i.e., yyyy.mm.dd.Thh.MM.ss.SSSZ
resultCountintegerThe number of objects returned in the "result" array.
totalCountintegerThe total number of objects that satisfy the query in the DB. This is useful when fetching a limited amount using the " limit " parameter.
nextCursorIdstringUsed to fetch the next batch of results. This parameter is not returned on the last batch of results, its absence means that the result set is finished.
resultArray of JSON objects

An array of idx objects (full or partial objects, based on your selected clause), retrieved from Gigya's IDX Storage. 

A field that does not contain data will not appear in the response.

Response Example

{   
   "statusCode":200,
   "errorCode":0,
   "statusReason":"OK",
   "callId":"6f1ec074d2a84a999f526d849e56de7b",
   "time":"2015-07-26T12:39:30.922Z",
   "resultCount":1,
   "totalCount":1,
   "result":[   
      {   
         "id":"9f1ec074d2a84a999f526d849e56de7r",
         "dataflowId":"782ec074d2a84a999f526d849ert84j",
         "startTime":"2015-07-28T13:54:03.428Z",
         "updateTime":"2015-07-28T14:04:03.428Z",
         "endTime":"2015-07-28T14:04:03.428Z",
         "totalRecords":18750,
         "processedRecords":18750,
         "attemptNumber":1,
         "status":"succeeded",
         "exitMessage":"................"
      }
   ]
}
  • No labels