Quickstart

This guide gets you from zero to a running LiteJoin pipeline in under 5 minutes. You’ll poll a public API, detect changes, and forward them to a local webhook.

Prerequisites

Go 1.24+ with CGO enabled (for SQLite)
A C compiler (gcc or clang)

Install

# Clone the repository
git clone https://github.com/litejoin/litejoin.git
cd litejoin

# Build the binary
CGO_ENABLED=1 go build -o litejoin ./cmd/litejoin

Create a Pipeline Config

Create a file called litejoin.yaml:

sources:
  # Poll the USGS earthquake API every 30 seconds
  - name: earthquakes
    type: api
    topic: quakes
    api:
      url: "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson"
      interval: 30s
      key_path: "id"
      response_path: "features"
      change_detection: diff

sinks:
  # Stream results via Server-Sent Events
  - type: sse
    name: dashboard
    config:
      addr: ":9100"

joins:
  - name: significant-quakes
    query: |
      SELECT
        key as quake_id,
        json_extract(payload, '$.properties.mag') as magnitude,
        json_extract(payload, '$.properties.place') as location,
        json_extract(payload, '$.properties.time') as event_time
      FROM quakes
      WHERE json_extract(payload, '$.properties.mag') >= 2.5
        AND timestamp > (strftime('%s', 'now') - 3600)
    sink: dashboard

storage:
  shard_count: 4
  data_dir: ./data

retention:
  duration: 24h

Run

./litejoin run -c litejoin.yaml

You’ll see output like:

litejoin: starting engine with 1 sources, 1 sinks, 1 joins
api-source earthquakes: first poll — 12 records ingested
sse-sink dashboard: listening on :9100

See Results

Open a terminal and subscribe to the SSE stream:

curl -N http://localhost:9100/subscribe?join=significant-quakes

Or fetch a snapshot of current results:

curl http://localhost:9100/snapshot?join=significant-quakes | jq

Every time the USGS data changes — new earthquake, updated magnitude — LiteJoin emits the enriched result.

Next Steps

LiteJoin Studio

Build pipelines visually with the desktop app.

API Source Guide

Configure watermarks, pagination, and change detection.

SQL Joins

Learn how to join data across multiple sources.

Deployment

Run LiteJoin in Docker, systemd, or Kubernetes.

Introduction

Getting Started

Sources

Sinks

Building Pipelines

Operations

Quickstart

Quickstart

Prerequisites

Install

Create a Pipeline Config

Run

See Results

Next Steps

LiteJoin Studio

API Source Guide

SQL Joins

Deployment

​Quickstart

​Prerequisites

​Install

​Create a Pipeline Config

​Run

​See Results

​Next Steps

LiteJoin Studio

API Source Guide

SQL Joins

Deployment

Quickstart

Prerequisites

Install

Create a Pipeline Config

Run

See Results

Next Steps