Data Store

Skip to end of metadata
Go to start of metadata

Overview

The Gigya Data Store (DS) is a flexible, fully indexed cloud database with a dynamic schema, that is designed to store generic data objects. Its flexibility includes the ability to configure the way data is stored by setting the default behavior for the ds.store method, and the ability to fetch data from the DS in one of two ways: search the entire databse, or, if the search involves a single known Object ID (OID), use ds.get for a fast data retrieval. Simple SQL-like queries are used when searching the DS. 

The DS service is delivered through a set of REST APIs. You may either use one of our Server Side SDKs to implement DS, or if there is no SDK available for your preferred language, you may use direct REST API  calls. For security reasons, the DS API is fully available via server to server API calls, but limited client-side access is available. Read below for more information about using the DS from a client application.

 
Note: The Data Store is a premium platform that requires separate activation. If DS is not part of your site package please contact your Gigya Account Manager or contact us by filling in a support form on our site. You can also access the support page by clicking "Support" on the upper menu of Gigya's site.

Major Features

  • Dynamic Schema - Meaning you don't have to know in advance how your data is going to look and it doesn't have to look the same for all objects. You may store any data you have with no constrains on its structure. There is no need to go through schema creation or modification when the data structure changes.
  • Fully indexed - Meaning that no matter what data you send you can always search based on any combination of stored fields in that data. You don't have to know what your queries are going to be or create the appropriate index in advance.
    An example of such a search: get the email addresses of all US users who graduated from any university between 2000-2005.

Data Objects

The DS stores JSON data objects, including full support for embedded objects and arrays. Different objects may contain completely different data fields. Each object has a unique Object ID ( oid ), you may retrieve an object by specifying its oid. Objects are grouped by type  for easier look-up and logical separation. Object types do not impose a schema, they are just used for logical separation of the data. See below for restrictions regarding client side use of data objects.

Note: Data Store JSON objects are limited to a maximum of 512K of data.

The following are the Object Data related API methods:

  • ds.store - stores an object in the DS
  • ds.get - retrieves an object from the DS
  • ds.delete - deletes an object from the DS
  • ds.search - uses an SQL-like query to search and retrieve data from the DS.

Applying a schema

Because the DS is schemaless you don't have to know in advance what data will be stored and objects can differ in structure. Nevertheless the DS allows specifying a schema defining a set of static data fields in the data, and their properties. The properties act as meta-data, guiding Gigya on how to handle the data in the specified fields. Other fields can be added to the DS and will be treated normally. Read more in the methods' reference:

Field Data Types

Stored fields are classified individually into one of the following types: integer, float, string, basic-string, text, boolean, binary or date. Since there is no schema, the type can be inferred automatically according to the field's data (once it is populated) or it can be specified explicitly using the 'type' parameter of the setSchema  method. The available types are:

  • integer - An integer number (may contain 64bit long values as well), allows exact matching or range searches.
  • float - A floating-point number, allows exact matching or range searches.
  • basic-string - A string that only allows exact matching. Limited to 16K in length and allows grouping in a search query.
  • string - A string that allows exact matching as well as full text search (partial searching based on keywords). Limited to 16K in length and allows grouping in a search query.
  • text - A string that allows for full text search (partial searching based on keywords). Unlimited in length. Does not allow grouping.
  • boolean - Contains 'true' or 'false' values.
  • date a date field, allows exact matching or range searches. The valid date format:
    • Standard ISO 8601 time format (like "2011.07.14T11.42.32.123+3", may be specified without a time at all or without a time zone)
    • "YYYY-MM-DDTHH.MM.SS.sssZ"
  • binary - The 'binary' field type cannot be inferred from the data and must be specified explicitly through the setSchema method. It can be used to hold large data objects such as base-64 images and large binary data. It is not limited in size or data format. However, it is not searchable using the ds.search method.

The following section is visible to internal users only.

Specifying Field Data Types

The field type can also be specified explicitly by appending the type to the field name in the following manner: <name>_<type> . For example, naming a field "created_d" specifies that the field contains date values.

The valid types are:

  • i - integer
  • f - floating point
  • s - string
  • t - text
  • b - Boolean
  • d - date
  • h - binary

 

End of internal only section.

When the system auto-detects a textual field value it defines it as a string. If you want the field to be treated as basic-string or text, define it so explicitly.

If you are not sure what should be the data type of the field value, it is preferable to omit the suffix and let our engine auto-detect the data type.

Relating Data Objects to Users

The DS allows storing information related to a specific user, as well as any generic data that is not directly related to a user.

For example, you may build a photo gallery using the DS. A user may own multiple albums: this means that albums are directly associated with the owning user, making them "user related data". A collection with any number of albums will be stored and associated with the owning user's UID. An album contains many photos, but photos are not directly associated with the user, just "owned" by the album that contains them; this makes them generic objects, which can also be stored in the DS.

To associate a data object with a user, pass the user's UID to the ds.store method.

Note: Once you associate a DS data object with a user's UID, deleting said user's account using accounts.deleteAccount will also delete all Data Objects currently associated with that user within the Data Store. Associating DS objects with a specific UID is a one-way operation*.

*Once an object has been associated with a specific user's UID within the Data Store, the association can not be removed.

Example

 The following example demonstrates storing information about a book. We use the ds.store  method and set the required method parameters with the following values:

  • type: "book"
  • data: { name_s: "Eat, Pray, Love", author_s: "Elizabeth Gilbert", ISBN_i: 0143038419 }
  • oid: "auto"

The REST request URL (after URL encoding) looks something like:

https://ds.gigya.com/ds.store?data=%7B%0A%20%20%20%20%22name_s%22%3A%20%22Eat%2C%20Pray%2C%20Love%22%2C%0A%20%20%20%20%22author_s%22%3A%20%22Elizabeth%20Gilbert%22%2C%0A%20%20%20%20%22ISBN_i%22%3A%20143038419%0A%7D&type=book&oid=auto&oauth_token=0000

Note that the method response includes the unique identifier (oid), generated by Gigya, for the newly added object:

{
        "statusCode": 200,
        "errorCode": 0,
        "statusReason": "OK",
        "callId": "31ba039fb8d340ceb2f43d52c89bf187",
        "oid": "12234"
}

Querying

Gigya provides an SQL-like query language for searching and querying the DS. Use the ds.search API method to query the DS: The method receives a 'query' parameter that accepts an SQL-like query string.

For details of the query syntax, see the ds.search documentation.

An example of a search REST request URL (after URL encoding):

https://ds.gigya.com/ds.search?query=SELECT%20album.photo.photoTitle_t%20FROM%20albums&oauth_token=0000

Notes:

  • It is always preferable to access data objects directly using their IDs (with ds.get) rather than using search. Accessing data objects directly using their IDs is faster because the process is highly optimized. 
  • After data is published to the DS, indexing is performed asynchronously and may take a number of seconds to complete.

Using the DS from a Client Application

For security reasons, we apply restrictions on the usage of the DS from client side, i.e. when calling DS methods through the JavaScript API or one of our Mobile SDKs.
Please note that some methods, such as ds.setSchema, are supported only for server-side SDKs.

Getting Data

You can only retrieve data which is associated with the currently logged-in user. When calling the ds.get method, the UID parameter is explicitly set the current user's UID.

Storing Data

By default, setting data in the DS is allowed only from a server application. To allow data to be set from the client side, you'll need to change the default settings. Each data field has a write access property, which is by default set to "serverOnly". You may change this property to be clientCreate or clientModify, by calling the  ds.setSchema from your server application.
Client applications can only modify field values if the field's write access property is set to clientModify.

Example:

We would like to be able to set data into the following two data fields from the client side: "address.city" and "address.state". These fields belong to the data type "otherInfo".
Before setting the data we need to specify that these two data fields should have a "clientModify" write access. To do this we first call the  ds.setSchema from the server application and pass the following schema as a parameter:

{
    fields: {
        "address.city": {
            "writeAccess":"clientModify"
        },
        "address.state": {
            "writeAccess":"clientModify"
        }
    }
}

The REST request URL (after URL encoding) would look something like:

https://ds.gigya.com/ds.setSchema?schema=%7Bfields%3A%20%7B%22address.city%22%3A%20%7B%22writeAccess%22%3A%22clientModify%22%7D%2C%22address.state%22%3A%20%7B%20%20%20%20%20%22writeAccess%22%3A%22clientModify%22%7D%7D&type=otherInfo&oauth_token=0000

Now we can set these fields from the client application, for example using the JavaScript API:

var params = {
	    data: {
           address: {
               city:"LA",
               state:"California"
           }
    },
    type:"otherInfo"
};

gigya.ds.store(params);
date.now();

 

Querying

When using the ds.search method from a client application, you are required to sign the request using the querySig and expTime parameters.

  • querySig : A HMAC_SHA1 signature. Follow the instructions in Constructing a Signature using the following base-string:  query  + "_" +  expTime , where  query  is your search query string and expTime  is the parameter described below.
  • expTime : The GMT time when the signature, provided in the  querySig  parameter, should expire. The expected format is the Unix time format (i.e. the number of seconds since Jan. 1st 1970). Gigya checks the time when the search request is received. If the time exceeds expTime, the request is considered forged.