Skip to main content

Replication

LiteJoin integrates with Litestream for continuous backup of SQLite shards to cloud object storage. Litestream ships WAL frames to S3, GCS, or Azure Blob in near-real-time, providing disaster recovery with minimal data loss.

Why Replicate?

  • Data loss on crash — LiteJoin targets containerized deployments where local disk is ephemeral. A pod restart means data is gone.
  • Restore-on-startup — Containers can boot, restore shards from S3, and resume processing automatically.
  • Point-in-time recovery — Roll back to any point within the retention window after a bad deployment.
The simplest approach: run Litestream alongside LiteJoin as a sidecar process.

1. Create litestream.yml

2. Startup Script

The -exec flag makes Litestream the parent process: it starts replication, launches LiteJoin as a subprocess, and gracefully flushes WAL frames on shutdown.

3. Dockerfile

Configuration

Replica Layout

Restore Flow

On startup with restore_on_boot: true:
  1. For each shard, check if the local file exists.
  2. If missing, restore from the latest snapshot + replay WAL frames from the replica.
  3. If present, skip restore and start replication.

Performance Impact

Litestream reads WAL frames asynchronously from disk — it does not sit in the write path. Benchmarks show < 1% write latency impact in sidecar mode.

Supported Backends

Litestream uses standard cloud SDK credentials (AWS_ACCESS_KEY_ID, GOOGLE_APPLICATION_CREDENTIALS, etc.). See the Litestream documentation for details.

Important Considerations

Litestream is a backup/recovery tool, not a high-availability solution. There is no automatic failover. If LiteJoin crashes, it must restart and restore — there is a recovery window bounded by sync_interval.
  • Shard count changes require re-sharding. The FNV hash distribution changes when you modify shard_count, invalidating existing replicas.
  • DLQ replication — the dlq.db database should also be replicated for complete disaster recovery.
  • Encryption — Litestream does not encrypt WAL frames. Use server-side encryption (SSE-S3, SSE-KMS) for encryption at rest.