Deployment Guide LiteJoin is a single binary with no external dependencies. This makes deployment simple, but there are operational considerations for production use.
Resource Planning CPU LiteJoin is lightweight. A 2-vCPU machine handles most workloads. CPU usage scales with:
Number of sources (each poll cycle parses JSON, diffs, and writes)
Join complexity (each write triggers join re-evaluation)
Window evaluation frequency
Memory Default memory usage is ~50–100 MB. SQLite’s page cache is the main consumer. If tiered storage is enabled with DuckDB, allocate additional memory (default DuckDB limit: 256 MB).
Disk Disk usage depends on:
Shard count × retention duration × ingest rate Each shard is a separate SQLite database file
WAL files can temporarily double the on-disk size during heavy writes
Rule of thumb: Allocate 2–3× your expected data size to account for WAL files and temporary compaction during archival.Network API source polling generates outbound HTTP requests. Calculate your request budget:
A source polling every 10s that fetches 5 pages = 30 requests/minute.
Monitoring Health Check Log Levels LiteJoin logs to stdout. Key patterns to monitor:
Pipeline Monitoring LiteJoin logs pipeline queue depths periodically to help diagnose bottlenecks:
Graceful Shutdown LiteJoin handles SIGTERM and SIGINT gracefully:
Sources stop polling / consuming
In-flight messages are flushed to storage
Pending DLQ retries complete
Sink connections are closed
SQLite databases are checkpointed and closed
Allow 10–30 seconds for graceful shutdown in your orchestrator’s termination grace period.
Multiple Pipelines Run multiple LiteJoin instances with separate configs and data directories:
Never point two LiteJoin instances at the same data_dir. SQLite requires exclusive write access.
