> ## Documentation Index
> Fetch the complete documentation index at: https://docs.litejoin.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Replication

> Continuous backup and disaster recovery with Litestream.

# Replication

LiteJoin integrates with [Litestream](https://litestream.io/) for continuous backup of SQLite shards to cloud object storage. Litestream ships WAL frames to S3, GCS, or Azure Blob in near-real-time, providing disaster recovery with minimal data loss.

## Why Replicate?

* **Data loss on crash** — LiteJoin targets containerized deployments where local disk is ephemeral. A pod restart means data is gone.
* **Restore-on-startup** — Containers can boot, restore shards from S3, and resume processing automatically.
* **Point-in-time recovery** — Roll back to any point within the retention window after a bad deployment.

## Sidecar Setup (Recommended)

The simplest approach: run Litestream alongside LiteJoin as a sidecar process.

### 1. Create `litestream.yml`

```yaml theme={null}
dbs:
  - path: ./data/shard_0.db
    replicas:
      - type: s3
        bucket: my-litejoin-backups
        path: shard_0
        region: us-east-1
        sync-interval: 1s

  - path: ./data/shard_1.db
    replicas:
      - type: s3
        bucket: my-litejoin-backups
        path: shard_1
        region: us-east-1
        sync-interval: 1s

  # ... one entry per shard (match your shard_count)
```

### 2. Startup Script

```bash theme={null}
#!/bin/bash
# run.sh — restore shards, then start Litestream with LiteJoin

# Restore all shards from replica (skips if local file exists)
for i in $(seq 0 7); do
  litestream restore -if-db-not-exists \
    -o ./data/shard_$i.db \
    s3://my-litejoin-backups/shard_$i
done

# Start Litestream replicate, exec LiteJoin as child process
exec litestream replicate -exec "litejoin run -c litejoin.yaml"
```

The `-exec` flag makes Litestream the parent process: it starts replication, launches LiteJoin as a subprocess, and gracefully flushes WAL frames on shutdown.

### 3. Dockerfile

```dockerfile theme={null}
FROM golang:1.24-alpine AS build
RUN apk add --no-cache gcc musl-dev
WORKDIR /app
COPY . .
RUN CGO_ENABLED=1 go build -o litejoin ./cmd/litejoin

FROM alpine:3.21
RUN apk add --no-cache litestream ca-certificates
COPY --from=build /app/litejoin /usr/local/bin/
COPY litestream.yml /etc/litestream.yml
COPY run.sh /run.sh
RUN chmod +x /run.sh
ENTRYPOINT ["/run.sh"]
```

## Configuration

```yaml theme={null}
replication:
  enabled: true
  mode: "sidecar"
  replica_url: "s3://my-bucket/litejoin"
  sync_interval: 1s
  restore_on_boot: true
  snapshot_interval: 1h
  retention: 72h
```

| Field               | Type     | Default   | Description                             |
| ------------------- | -------- | --------- | --------------------------------------- |
| `enabled`           | bool     | `false`   | Enable replication.                     |
| `mode`              | string   | `sidecar` | `"sidecar"` or `"embedded"`.            |
| `replica_url`       | string   | —         | S3/GCS/Azure URL for replicas.          |
| `sync_interval`     | duration | `1s`      | How often WAL frames are shipped.       |
| `restore_on_boot`   | bool     | `true`    | Restore shards from replica on startup. |
| `snapshot_interval` | duration | `1h`      | How often full snapshots are created.   |
| `retention`         | duration | `72h`     | How long to keep replica history.       |

## Replica Layout

```
s3://my-bucket/litejoin/
├── shard_0/
│   └── generations/
│       └── a1b2c3d4/
│           ├── snapshots/
│           └── wal/
├── shard_1/
│   └── ...
├── ...
└── dlq/
    └── ...   # DLQ database also replicated
```

## Restore Flow

On startup with `restore_on_boot: true`:

1. For each shard, check if the local file exists.
2. If missing, restore from the latest snapshot + replay WAL frames from the replica.
3. If present, skip restore and start replication.

## Performance Impact

Litestream reads WAL frames asynchronously from disk — it does not sit in the write path. Benchmarks show **\< 1% write latency impact** in sidecar mode.

## Supported Backends

| Backend              | URL Format                    |
| -------------------- | ----------------------------- |
| Amazon S3            | `s3://bucket-name/prefix`     |
| Google Cloud Storage | `gcs://bucket-name/prefix`    |
| Azure Blob Storage   | `abs://container-name/prefix` |
| SFTP                 | `sftp://user@host/path`       |
| Local file           | `file:///path/to/backup`      |

<Note>
  Litestream uses standard cloud SDK credentials (AWS\_ACCESS\_KEY\_ID, GOOGLE\_APPLICATION\_CREDENTIALS, etc.). See the [Litestream documentation](https://litestream.io/reference/config/) for details.
</Note>

## Important Considerations

<Warning>
  Litestream is a **backup/recovery** tool, not a high-availability solution. There is no automatic failover. If LiteJoin crashes, it must restart and restore — there is a recovery window bounded by `sync_interval`.
</Warning>

* **Shard count changes** require re-sharding. The FNV hash distribution changes when you modify `shard_count`, invalidating existing replicas.
* **DLQ replication** — the `dlq.db` database should also be replicated for complete disaster recovery.
* **Encryption** — Litestream does not encrypt WAL frames. Use server-side encryption (SSE-S3, SSE-KMS) for encryption at rest.