Deployment Guide
LiteJoin is a single binary with no external dependencies. This makes deployment simple, but there are operational considerations for production use.
Resource Planning
CPU
LiteJoin is lightweight. A 2-vCPU machine handles most workloads. CPU usage scales with:
- Number of sources (each poll cycle parses JSON, diffs, and writes)
- Join complexity (each write triggers join re-evaluation)
- Window evaluation frequency
Memory
Default memory usage is ~50–100 MB. SQLite’s page cache is the main consumer. If tiered storage is enabled with DuckDB, allocate additional memory (default DuckDB limit: 256 MB).
Disk
Disk usage depends on:
- Shard count × retention duration × ingest rate
- Each shard is a separate SQLite database file
- WAL files can temporarily double the on-disk size during heavy writes
Rule of thumb: Allocate 2–3× your expected data size to account for WAL files and temporary compaction during archival.
Network
API source polling generates outbound HTTP requests. Calculate your request budget:
A source polling every 10s that fetches 5 pages = 30 requests/minute.
Monitoring
Health Check
Log Levels
LiteJoin logs to stdout. Key patterns to monitor:
Pipeline Monitoring
LiteJoin logs pipeline queue depths periodically to help diagnose bottlenecks:
Graceful Shutdown
LiteJoin handles SIGTERM and SIGINT gracefully:
- Sources stop polling / consuming
- In-flight messages are flushed to storage
- Pending DLQ retries complete
- Sink connections are closed
- SQLite databases are checkpointed and closed
Allow 10–30 seconds for graceful shutdown in your orchestrator’s termination grace period.
Multiple Pipelines
Run multiple LiteJoin instances with separate configs and data directories:
Never point two LiteJoin instances at the same data_dir. SQLite requires exclusive write access.
