Skip to main content

​
Quickstart

This guide gets you from zero to a running LiteJoin pipeline in under 5 minutes. You’ll poll a public API, detect changes, and forward them to a local webhook.

​
Prerequisites

  • Go 1.24+ with CGO enabled (for SQLite)
  • A C compiler (gcc or clang)

​
Install

# Clone the repository
git clone https://github.com/litejoin/litejoin.git
cd litejoin

# Build the binary
CGO_ENABLED=1 go build -o litejoin ./cmd/litejoin

​
Create a Pipeline Config

Create a file called litejoin.yaml: 
sources:
  # Poll the USGS earthquake API every 30 seconds
  - name: earthquakes
    type: api
    topic: quakes
    api:
      url: "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson"
      interval: 30s
      key_path: "id"
      response_path: "features"
      change_detection: diff

sinks:
  # Stream results via Server-Sent Events
  - type: sse
    name: dashboard
    config:
      addr: ":9100"

joins:
  - name: significant-quakes
    query: |
      SELECT
        key as quake_id,
        json_extract(payload, '$.properties.mag') as magnitude,
        json_extract(payload, '$.properties.place') as location,
        json_extract(payload, '$.properties.time') as event_time
      FROM quakes
      WHERE json_extract(payload, '$.properties.mag') >= 2.5
        AND timestamp > (strftime('%s', 'now') - 3600)
    sink: dashboard

storage:
  shard_count: 4
  data_dir: ./data

retention:
  duration: 24h

​
Run

./litejoin run -c litejoin.yaml
You’ll see output like: 
litejoin: starting engine with 1 sources, 1 sinks, 1 joins
api-source earthquakes: first poll — 12 records ingested
sse-sink dashboard: listening on :9100

​
See Results

Open a terminal and subscribe to the SSE stream: 
curl -N http://localhost:9100/subscribe?join=significant-quakes
Or fetch a snapshot of current results: 
curl http://localhost:9100/snapshot?join=significant-quakes | jq
Every time the USGS data changes — new earthquake, updated magnitude — LiteJoin emits the enriched result.

​
Next Steps

LiteJoin Studio

Build pipelines visually with the desktop app.

API Source Guide

Configure watermarks, pagination, and change detection.

SQL Joins

Learn how to join data across multiple sources.

Deployment

Run LiteJoin in Docker, systemd, or Kubernetes.