Overview
The UDP processes events before they become event records in the UDP Event store.
The UDP Caliper endpoint triages events into processing streams based on the event's payload. If an event qualifies for particular processing, it is routed to that processing stream. In its response, the UDP caliper endpoint provides the signature(s) of the processing stream(s) to which an event was routed. Events may qualify for more than one processing streams.
The UDP currently supports two categories of event processing, each with one or more processing stream.
Event record processing
Event record processing streams create event records in the UDP Event store.
Reduced latency processing
Reduced latency processing streams use data in the event payload to update data in the UDP's relational store. The effect is to reduce the latency of data in the UDP relational store.
Key concepts:
- Event record. The record schema in the UDP Event stoer (see here).
- Event processing stream. Defines a form of event computation that is applied for a particular purpose.
- Event processing stream signature. A unique string identifying an event processing stream.
- UCDM enrichment. The process of adding data to events such that they are aligned with the Unizin Common Data Model and the UDP relational store.
1. Event record processing
Event record processing is divided into two tiers – Silver and Gold. Events may only be routed to one processing stream in one tier.
1.1 Silver tier processing
Silver tier processing represents a baseline of event record generation. In Silver tier processing, attributes required by the Caliper standard are extracted and stored in the event record (see below). The event record also includes the raw event payload. There is only one processing stream at the Silver tier.
Generally speaking, any valid Caliper event that does not qualify for a Gold tier event stream is routed to the Silver tier processing.
Attribute extraction
The following Caliper event attributes are extracted from an event and become part of the UDP Event store record.
| Caliper attribute | Event record field |
|---|---|
uuid |
id |
type |
event_type |
action |
event_action |
| Event JSON | event |
1.2 Gold tier processing
Gold tier processing streams extend Silver tier processing in two ways.
First, Gold tier processing streams extract optional Caliper event attributes and store them in the event record.
Second, Gold tier processing streams enrich both the event record and event payload with data that aligns the event to the UDP's relational store.
Gold tier processing streams are tool specific. The Caliper information model offers tool developers many ways to represent the same types of entities in an event. Consequently, Gold tier event processing streams must have knowledge of the particular way a tool models its entities.
Attribute extraction
Gold tier processing streams extract the following Caliper event attributes from an event to become part of the UDP Event store record. All event attributes extracted at the Silver tier are also extracted at the Gold tier.
| Concept | Caliper Entity | Typical event attribute | Event record field |
|---|---|---|---|
| The unique identifier for the Learning Tool | SoftwareApplication | edApp |
ed_app |
| Actor who performed the behavior | Actor | actor |
event_actor_id |
| Course in which event occurred | CourseOffering | Varies by application | event_course_offering_id |
UCDM enrichment
Gold tier processing streams align event data to the Unizin Common Data Model. They achieve alignment by enriching events and event records with object identifiers from the UDP relational store that correspond to the identifiers for the same objects in the event payload.
In the UDP Event store event record, UDP relational store identifiers are prefixed with the string ucdm_.
When this event is routed to Gold tier processing, the common identifiers for the Learner and the Course offering in the UDP relational store are retrieved. These identifiers are added to the Caliper event payload and also written to the Event store.
Aligning Caliper events with the UCDM and with your UDP relational store allows event data to be joined with object data.
Event record enhancement
The following Caliper-to-UCDM Entity mappings are currently supported in Gold tier processing.
| Caliper Entity | UCDM Entity | Event record field |
|---|---|---|
| Actor | Person | ucdm_actor_id |
| CourseOffering | Course offering | ucdm_course_offering_id |
Event JSON enrichment
Caliper events are enriched with UDP relational store identifiers through the extensions node of the JSON event payload. For example, an enriched Canvas Live Event's extensions node will look like this after enrichment:
"extensions": {
"com.instructure.canvas": {
"hostname": "",
"request_id": "",
"user_agent": "",
"version": ""
},
"org.unizin.udp": {
"ucdm_actor_id": "67890",
"ucdm_course_offering_id": "12345"
}
2. Reduced latency processing
Reduced latency processing uses event data to reduce the latency of data in the UDP relational store.
Generally speaking, two strategies are applied to reduce data latency with event data:
- Event payload data is used to update the UDP relational store
- Events types/actions trigger API calls whose responses contain data used to update the UDP relational store
Below are the Reduced latency processing streams currently supported by the UDP.
| Application | Event type | UDP object data |
|---|---|---|
| Canvas Live Events | Enrollment updated | Workflow state and Role updated for Course section enrollment. |
| Canvas Live Events | Grade changed | ScoreValue updated for Learner activity result. |
| Canvas Live Events | Submission created | Learner activity result is created. |
3. Event stream signatures
Event stream signatures are strings. The strings uniquely identify a processing stream to which an event was routed. Event stream signatures are returned in the Caliper endpoint's responses.
Event stream signatures are unique to an Institution and a UDP environment. All institutions are identified by a shortcode equivalent to their top-level web domain (e.g., umich for the University of Michigan). Three environments will be supported in the UDP: "SIT" (for systems integration testing) and "PROD" (for production).
| Route signature | Processing type | Application |
|---|---|---|
STREAM-unizin-<INST>-<INST>-stream-Silver-<ENV> |
Silver tier | All |
STREAM-unizin-<INST>-<INST>-stream-GoldCanvasLiveEvents-<ENV> |
Gold tier | Canvas Live Events |
STREAM-unizin-<INST>-<INST>-stream-GoldEngage-<ENV> |
Gold tier | Unizin Engage |
STREAM-unizin-<INST>-<INST>-stream-GoldUMichLectureCapture-<ENV> |
Gold tier | U-M CAEN Lecture Capture |
STREAM-unizin-<INST>-<INST>-stream-RLEnrollmentUpdated-<ENV> |
Reduced latency | Canvas Live Events (Enrollment updated) |
STREAM-unizin-<INST>-<INST>-stream-RLGradeChanged-<ENV> |
Reduced latency | Canvas Live Events (Grade changed) |
STREAM-unizin-<INST>-<INST>-stream-RLSubmissionCreated-<ENV> |
Reduced latency | Canvas Live Events (Learner activity result created) |