Replication
LiteJoin integrates with Litestream for continuous backup of SQLite shards to cloud object storage. Litestream ships WAL frames to S3, GCS, or Azure Blob in near-real-time, providing disaster recovery with minimal data loss.Why Replicate?
- Data loss on crash — LiteJoin targets containerized deployments where local disk is ephemeral. A pod restart means data is gone.
- Restore-on-startup — Containers can boot, restore shards from S3, and resume processing automatically.
- Point-in-time recovery — Roll back to any point within the retention window after a bad deployment.
Sidecar Setup (Recommended)
The simplest approach: run Litestream alongside LiteJoin as a sidecar process.1. Create litestream.yml
2. Startup Script
-exec flag makes Litestream the parent process: it starts replication, launches LiteJoin as a subprocess, and gracefully flushes WAL frames on shutdown.
3. Dockerfile
Configuration
Replica Layout
Restore Flow
On startup withrestore_on_boot: true:
- For each shard, check if the local file exists.
- If missing, restore from the latest snapshot + replay WAL frames from the replica.
- If present, skip restore and start replication.
Performance Impact
Litestream reads WAL frames asynchronously from disk — it does not sit in the write path. Benchmarks show < 1% write latency impact in sidecar mode.Supported Backends
Litestream uses standard cloud SDK credentials (AWS_ACCESS_KEY_ID, GOOGLE_APPLICATION_CREDENTIALS, etc.). See the Litestream documentation for details.
Important Considerations
- Shard count changes require re-sharding. The FNV hash distribution changes when you modify
shard_count, invalidating existing replicas. - DLQ replication — the
dlq.dbdatabase should also be replicated for complete disaster recovery. - Encryption — Litestream does not encrypt WAL frames. Use server-side encryption (SSE-S3, SSE-KMS) for encryption at rest.