Caliper Graph Demo

Purpose

This project is a Proof of Concept (PoC)/ demonstration for using the 1EdTech Caliper standard to capture and analyze real-time, time-series event data in education. It demonstrates how Caliper event streams can be combined with authoritative source-of-truth data (such as course and student information) to provide meaningful analytics.

Key Principle:

Caliper event data is most valuable when combined with the source-of-truth for course and student data.

In this project, a minimal subset of the EduAPI data model is used to represent the source-of-truth for users (students) and courses.

---

What This Demo Shows (Caliper Features)

This project demonstrates the following Caliper capabilities:

1. How to send Caliper events
   • Uses the Caliper JS library to generate and send events in the correct format.
2. How to receive Caliper events
   • Implements a Caliper event collector endpoint that ingests and processes incoming events.
3. How to validate Caliper events
   • Validates all events against the Caliper v1.1 and v1.2 schemas to ensure compliance.
4. How to authenticate Caliper events
   • Demonstrates an OAuth2-style authentication flow for event submission (demo only, not for production).
5. Examples for how to query and access the data to discover new insights
   • Shows how to access and query the stored event and entity data (e.g., in Neo4j) to derive analytics and insights.
6. How to blend the time-series data of Caliper with source-of-truth
   • Illustrates how to combine event streams with authoritative user/course data for richer analytics.

---

Architecture & Modes

There are two main modes for the source-of-truth (EduAPI) integration:

1. Mock EduAPI (Default)

• Purpose: Easiest way to get started.
• How it works:
  • Generates a set of mock users and courses on first run.
  • Both the pipeline (Caliper event collector) and the generator (Caliper event producer) use the same shared mock data.
  • No external dependencies required.

2. Advanced: Async EduAPI (Real-time Sync)

• Purpose: Demonstrates real-time synchronization with an external SIS or LMS.
• How it works:
  • Both the pipeline and the generator synchronize users and courses in real-time with an external source-of-truth (SIS/LMS) using the Async EduAPI protocol.
  • Requires: A compatible SIS/LMS backend implementing the Async EduAPI gRPC interface. See the Async EduAPI PoC demo for a reference implementation.

---

Data Storage

• This demo uses Neo4j (GraphDB) as the event and entity store.
• Note: Caliper is datastore-agnostic. Many real-world implementations use relational databases (with JSONB columns) or document stores like MongoDB.

---

Authentication

• The project includes a simple OAuth2-style auth layer for demonstration purposes only.
• Do not use this auth code in production!

---

Quickstart

1. Install dependencies

npm install

2. Configure environment

Edit the .env file to set your desired mode and parameters. Example:

# Use mock mode (default)
EDUAPI_MODE=mock
# Or use real-time sync (requires external SIS)
# EDUAPI_MODE=grpc

# Neo4j connection
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_password

# Other options: see .env for more

3. Start the pipeline (Caliper event collector)

npm run start:pipeline

4. Start the generator (Caliper event producer)

npm run start:generator

---

Switching Modes

• Mock mode (default):
  • No external dependencies. Uses generated users/courses in packages/eduapi-mock/.
• Async EduAPI mode:
  • Set EDUAPI_MODE=grpc in .env.
  • Requires a running compatible SIS/LMS backend (see above).

---

Troubleshooting

• Module not found errors: Ensure your IDE and TypeScript are picking up the path aliases. Try restarting your IDE and running tsc --build from the project root.
• Neo4j connection errors: Make sure Neo4j is running and the credentials in .env are correct.
• Mock data not found: The first run will generate mock users/courses in packages/eduapi-mock/. If you delete these files, they will be regenerated.

---

Project Structure

• apps/pipeline/ - Caliper event collector (ingests, validates, and stores events)
• apps/generator/ - Caliper event generator (simulates user activity)
• packages/eduapi-mock/ - Shared mock data and utilities
• packages/grpc/ - gRPC/Proto definitions for EduAPI
• packages/caliper/ - Caliper type definitions and utilities

---

Disclaimer

This project is for demonstration and educational purposes only. The authentication and mock data generation are not suitable for production use.

---

Project Structure

caliper-graph-demo/
  apps/
    pipeline/      # Caliper event consumer and graph pipeline
    generator/     # Caliper event generator/simulator
  packages/
    caliper/       # Shared Caliper types and library files
    grpc/          # Shared gRPC client/server logic and generated code
  proto/           # Protocol buffer definitions
  schemas/         # Caliper JSON schemas (v1.1, v1.2)
  README.md        # This file
  package.json     # Project scripts and dependencies
  tsconfig.json    # TypeScript configuration with path aliases

Key Folders

• apps/pipeline: Handles event ingestion, validation, and graph storage. Runs an Express server and syncs with the SIS via gRPC.
• apps/generator: Generates and sends realistic Caliper events for testing. Uses gRPC to fetch SIS data.
• packages/caliper: Contains Caliper type definitions and library files (CJS, ESM, UMD builds).
• packages/grpc: Contains gRPC client/server logic and TypeScript code generated from proto files.

---

Getting Started

Prerequisites

• Node.js (v18+ recommended)
• TypeScript
• Neo4j database (for pipeline app)

Install dependencies

npm install

Run the Pipeline App

npm run start:pipeline

Run the Generator App

npm run start:generator

Regenerate Async EduAPI bindings (if you change proto files)

The Async EduAPI prototype is built on gRPC. It uses protobuffers (.proto files) to define the messages.

npm run proto:generate

---

Development Notes

• Follows Clean Architecture: each app has interface, application, domain, and infrastructure layers.
• Shared code is in packages/ for easy reuse.
• Easily extensible for new event types, data sources, or analytics pipelines.

---

📚 Additional Documentation & Resources

• Code Structure Overview:
  docs/code-structure.md
  Explains the project's use of Domain-Driven Design (DDD) and Clean Architecture, and how to navigate the codebase.

• MCP Experiment:
  docs/mcp.md
  A playground/ example that shows how to integrate with the Model Context Protocol (MCP).

• Postman Collection:
  postman/caliper-events.postman_collection.json
  Use this collection to test Caliper event endpoints and explore the API.

---

🐳 Docker & Docker Compose

You can run the entire stack (pipeline, generator, Neo4j) using Docker Compose. This is the recommended way to get started quickly, especially for development or demo purposes.

1. Build the Docker Images

From the project root, run:

docker-compose build

2. Start the Stack (Pipeline + Neo4j, Generator Stopped)

To start only the pipeline and Neo4j (with the generator not running):

docker-compose up 

• This will:
  • Start the pipeline service (Caliper event collector/consumer)
  • Start the db service (Neo4j database)
  • The generator service will not start automatically

3. Start the Generator Separately

In a new terminal window, you can start the generator service when you're ready:

docker compose run --rm generator

• This will start the generator (Caliper event producer) and connect it to the running pipeline and Neo4j services.

4. Stopping Services

To stop all running services:

docker-compose down

---

📝 Notes on Docker Compose Configuration

• The docker-compose.yml is set up so that the generator service does not start by default (it uses deploy.replicas: 0 and can be started on demand).
• All services use the same Docker image, with the service type controlled by the SERVICE_TYPE environment variable.
• Neo4j data is persisted in a Docker volume (neo4j_data).

---

License

MIT