Data formats and access

1. Format Overview

  • DataConnect.io uses the schema.org specification as its data output format.
  • The base wrapper format for dataconnect.io entities is the schema.org/DataFeedItem and includes these fields:
    • dateCreated - when the entity was created on the platform.
    • dateDeleted - when a deleted entity was deleted on the platform.
    • dateModified - when an entity was updated or modified on the platform.
    • item - the entity itself. This will be an instance of either schema.org/Product or schema.org/Order.

2. Supported Shopping Platforms

  • At the moment DataConnect.io delivers data from eBay and Shopify shops. More will be added each month.

3. Attributes in the normalised output

  • schema.org provides a very flexible set of data attributes, some of which are not always available from the data we fetch from individual shopping platforms. As a result, we define a minimum set of attributes, which you can expect to always be there. We also provide a list of other attributes which will be added to the normalised outputs, if they are available in the raw data we get from the shopping platform. Both these attribute lists are shown below.

3.a Minimum set of attributes for schema.org/Product entities

Please consult the schema.org/Product for more detailed descriptions of these fields.

  • _entityID - this is a custom field added by dataconnect.io. It represent the identifier of the entity on the shopping platform.
  • productID
  • name
  • category
  • description
  • offers - variants of the product which are on offer. Each is an instance of schema.org/Offer

3.b Additional attributes that will be normalised in the schema.org/Product entity, if they are present in the raw data we get from the shopping platform.

  • brand
  • alternateName
  • offeredBy
  • color
  • height
  • weight
  • gtin*
  • ìtemCondition
  • sku

3.c Minimum required for schema.org/Offer entities

  • price
  • priceCurrency
  • inventoryLevel
  • offeredItem (note that offeredItem is a schema.org/Product entity).

3.d Additional fields to be added to schema.org/Offer if present

  • acceptedPaymentMethod
  • availabilityStarts
  • availabilityEnds
  • deliveryLeadTime
  • sku
  • gtin*
  • offeredBy
  • seller

3.e Minimum required for schema.org/Order

  • _entityID
  • acceptedOffer - note that this is an instance of schema.org/Offer
  • orderedItem - note this is an instance of schema.org/Product
  • billingAddress
  • customer
  • orderDate
  • createdAt
  • updatedAt
  • orderNumber

3.f Additional fields to be added to schema.org/Order if present

  • confirmationNumber
  • discount
  • discountCode
  • discountCurrency
  • orderDelivery
  • orderStatus
  • paymentMethod
  • paymentDueDate
  • seller

4. Examples

Here are some examples of "raw" data entities as provided directly by the platforms eBay and Shopify, as well as the same entities after they have been through DataConnect.io's normalisation process.

Shopify

Product
  • Shopify product: https://gitlab.com/snippets/32931
  • Corresponding normalised output from dataconnect: https://gitlab.com/snippets/32930
Order
  • Shopify order: https://gitlab.com/snippets/32932
  • Corresponding normalised output from dataconnect: https://gitlab.com/snippets/32934

Ebay

Ebay Item
  • GetItemResponse entity: https://gitlab.com/snippets/32939
  • Corresponding normalised output from dataconnect:https://gitlab.com/snippets/33016
Ebay Order
  • GetOrders entity; https://gitlab.com/snippets/32936
  • Corresponding normalised output from dataconnect: https://gitlab.com/snippets/32937

Accessing data stored via server-example in RESTHeart

information_source For all calls to RESTHeart, you need to use basic authentication. By default, the username is quickstart. You can see the current configuration for it in the RESTHeart configuration file dist/restheart/etc/security.yml.

All data created by the QuickStart App is stored and can be accessed via its local RESTHeart instance. RESTheart provides a RESTful interface on top of a MongoDB. All data is stored in the DB quickstart. The naming schema for the MongoDB collections is as follows:

  • connections stores all created connections
  • products stores all normalised products
  • orders stores all normalised orders

All data is stored as an entity of type schema.org/DataFeedItem. The example below shows how a normalised eBay product looks after it has been delivered by DataConnect.io via the QuickStart app:

{
  "@context": {
    "@vocab": "http://schema.org/"
  },
  "@type": "DataFeedItem",
  "dateCreated": "2016-10-27T10:40:19.000Z",
  "item": {
    "@type": "Product",
    "url": "http://cgi.sandbox.ebay.com/iphone-5-something-/110185480624\n\t\t\t",
    "name": "iphone 5 or something",
    "brand": {
      "@type": "Brand"
    },
    "category": "Cell Phones & Accessories:Cell Phones &\n\t\t\t\tSmartphones",
    "offers": [
      {
        "@type": "Offer",
        "availabilityEnds": "2016-11-03T10:40:49.000Z",
        "availabilityStarts": "2016-10-27T10:40:49.000Z",
        "deliveryLeadTime": "1",
        "inventoryLevel": 0,
        "priceCurrency": "USD",
        "seller": {
          "@type": "Person",
          "alternateName": "testuser_sitest",
          "email": "viking@dataconnect.io"
        }
      }
    ],
    "productID": "110185480624",
    "_entityID": {
      "@type": "PropertyValue",
      "value": "110185480624"
    }
  }
}

All of the available fields can be used for filtering – let's take a look at some examples.

Get all normalised products for connection with id 9a64a4d6-2f4e-4e04-911e-6994e05577f0:

http -j -a quickstart:password GET 127.0.0.1:8080/quickstart/products?filter={\"connectionId\":\"9a64a4d6-2f4e-4e04-911e-6994e05577f0\"}

Get all normalised products with id 327475578523353100 for a connection with id 9a64a4d6-2f4e-4e04-911e-6994e05577f0:

http -j -a quickstart:password GET 127.0.0.1:8080/quickstart/products?filter={\"connectionId\":\"9a64a4d6-2f4e-4e04-911e-6994e05577f0\"\,\"entityId\":\"327475578523353100\"}

You can find more information about the RESTHeart API in the official documentation.