Caliper integration

Developer guide


Overview

This document is a Caliper sensor implementation guide for developers integrating with the Unizin Data Platform. It defines the validation performed by the Caliper endpoint, the endpoint's responses, and how to design payloads that make the most of the UDP's capabilities.

The Unizin Data Platform (UDP) ingests Caliper event streams through a Caliper endpoint exposed over the web (HTTPS only). For each Caliper envelope it ingests, the Caliper endpoint performs two major operations:

  1. Validates the HTTP request, Caliper envelope, and Caliper event(s)
  2. Routes events for processing before being persisted in the UDP Event store.

The Endpoint's HTTP response will vary based on the validation and routing it performs.

1. Message authentication

Caliper events posted to your Caliper endpoint MUST be authenticated with a token granted by Unizin. If a Caliper event envelope or Caliper event is posted without a valid authentication token, it is rejected by the Caliper endpoint.

Per the Caliper 1.1 specification, a tool's Caliper sensor implementation:

SHOULD ... set the Authorization request header field using the "Bearer" authentication scheme described in RFC 6750, Section 2.1. The b64token authorization credential sent by a Sensor MUST be one the Endpoint can validate although the credential MAY be opaque to the emitting Sensor itself.

Your Caliper endpoint will inspect the Authorization: request header for a Bearer token and compare it agains a list of valid tokens.

An example header for a Caliper message sent over HTTPS would therefore look like this: 

POST /caliper-endpoint HTTP/1.1
Host: server.example.com
Authorization: Bearer [some_granted_token]

1.1 Granting and revoking auth tokens

At present, Unizin's technical services team will grant and revoke authentication tokens on Institution's behalf. Tokens are granted on a per-tool basis. It is strongly recommended that institutions do not share the same token among multiple tool providers. If the same auth token is granted to multiple tool providers, then all are subject to the validity of a token that may need to revoked if a single tool provider were to abuse a token.

Request an Auth token.

2. Responses

In a prod environment, the Caliper's endpoint responses are terse to keep the endpoint performant at scale. We call these "common" responses (see below).

In a sit environment, the Caliper endpoint's responses are verbose (see below). The responses are intended to aid in debugging your Caliper event stream integration. If a valid HTTP request, the Endpoint's response will contain information about your envelope and event conformance with Caliper along with information describing how events were routed for processing.

Response HTTP status codes

The endpoint may return different status codes depending on the results of validation.

HTTP request errors:

Caliper conformance errors:

Success:

Response body

For any given request, the Caliper endpoint will respond in one of two styles.

Common response
In a prod environment, the Caliper endpoint will always and only return a common response. In a sit environment, a common response is given if and only if there is a validation error on the HTTP request itself (e.g., an invalid authentication token is used). In such cases, the Caliper endpoint rejects the request without inspecting the Caliper enevelope or its event(s). 

  {
      "error": "Unauthorized",
      "message": "Bearer token missing or not recognized"
  }

Verbose response
In a sit environment, the Caliper endpoint will always return a verbose response if the HTTP request is valid.

A verbose response is divided into two sections.

An accepted_events attribute contains information about all of the conformant events that were accepted and to which processing they were routed (these are called "topics"). An errors attribute contains information about the Caliper envelope and all of the events that did not conform with Caliper. 

  {
      'accepted_events': [
          {
              'id': 'urn:uuid:b46f370c-79e0-433f-9f2b-2d750155e338',
              'topics': { '': '', ... }
          }, ...
      ]
      'errors': {
          'envelope': [  ],
          'events': [
              {
                 'id': 'urn:uuid:b46f370c-79e0-433f-9f2b-2d750155e337',
                 'index': 0,
                 'errors': ['Event `actor` attribute must an object or ' 'string.'],
              }, ...
          ]
      }
  }

3. Validation errors: requests, envelopes, events

The UDP Caliper endpoint only accepts valid HTTP requests with content-type of JSON.

HTTP requests

Requirement Error message
Authentication token is present and valid Bearer token missing or not recognized
Content type is JSON Unsupported Media Type", "message": "application/json payload expected.

Caliper envelope

All events posted to the Endpoint must be delivered in a Caliper envelope. Caliper envelopes must conform to the version 1.1 of the IMS Global Caliper standard. The following validation is performed on Caliper envelopes.

Requirement Error message
sensor is present Event envelope must include a `sensor` attribute
sendTime is present and in ISO 8601 format Event envelope `sendTime` attribute must be an ISO 8601 time string formated like "yyyy-mm-ddThh:mm:ss.SSSZ".
dataVersion is present Event envelope `dataVersion` attribute must be a version like "http://purl.imsglobal.org/ctx/caliper/v1p1"
dataVersion is valid Event envelope `data` attribute must be an array with at least one element.

Caliper events

All events in a Caliper envelope must conform with the Caliper standard. The Caliper endpoint will reject any events that fail the following requirements.

Requirement Error message
id is present Event `id` must be present. Version 4 UUID is recommended.
@context is present and valid Event `@context` must be a version like "http://purl.imsglobal.org/ctx/caliper/v1p1".
actor is present Event `actor` attribute must an object or string.

Note: we recommend defining actor as a hash with an id and type attribute. Alternatively, actor may be defined as a string.
action is present
type is present 		"Event `type` and `action` must be present, and `action` must be valid for `type`.
eventTime is present and valid ISO 8601 format Event `eventTime` attribute must be an ISO 8601 time string formated like "yyyy-mm-ddThh:mm:ss.SSSZ".

4. Event processing

When Caliper events are accepted, the response includes information describing to which kind of processing the event was routed.

At present, two types of processing are supported:

Event record processing is applied to all events captured by the UDP’s Caliper endpoint. Event record processing entails taking a valid Caliper event and persisting it in the Event Store. Unizin defines two tiers of Event record processing.

(See further documentation on the UDP Caliper endpoint's event processing.)

5. General guidelines

Make your event payloads consistent

In practice, the UDP endpoint examines the attributes in event payloads to determine if they qualify for processing beyond standard processing. The UDP endpoint then routes an event to all of the kinds of processing for which it qualifies (our processing types are categorized into two main tiers).

To enable event processing beyond Silver tier event record processing, Unizin will build custom integrations for events streamed by your tool. To make our development efficient, we strongly urge you to:

Make your identifiers predictable IRIs or URNs

We suggest you define the identifiers included in your event payloads as Internationalized Resource Identifiers, or IRIs (RFC 3987). The UDP’s common modeling engine will expect identifiers to be IRIs. A known, predictable string pattern will allow for the UDP’s common modeling engine to strip the relevant piece(s) of the id that can cross-reference to existing data in the relational store.

Alternatively, if your identifiers are not normally IRIs, we suggest constructing them as URNs, which are useful for showing the object’s location in the app’s internal hierarchy.

For example: 

“edApp.id”: “http://unizin.instructure.com/”
“session.id”: “urn:umich:engin:leccap:session:T7Lzgsmtq0x2e1204jut6sop6”

Include LTI launch parameters in your payload

If your tool is launched via LTI, Unizin recommends including the LTI launch parameters passed to your tool via an event’s federatedSession object. Of particular interest are the LTI parameters representing the LMS course ID and the LMS user ID corresponding to the event. These values are useful in commonly modeling your event data with object data.

In the example below, the LMS course and user IDs are passed via messageParameters (this is arbitrary) and identified with the attributes lis_course_offering_sourcedid and lis_person_sourcedid, respectively. 

"federatedSession": {
  "type": "LtiSession",
  "id": "urn:session-id-localized/lti/oauth_nonce/5a81e2eb557427.65333446",
  "messageParameters": {
    "custom_canvas_course_id": "12345",
    "custom_canvas_user_id": "98765",
    "lis_course_offering_sourcedid": "test_99999003",
    "lis_person_sourcedid": "11235813",
    "resource_link_id": "a1b2c3d4e5f6g7h8i9j10"
  },
  "user": "https://university.edu/#profile:smith"
}