> ## Documentation Index
> Fetch the complete documentation index at: https://docs.litejoin.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage

> SQLite sharding, tiered storage with Parquet + DuckDB, and cloud archival.

# Storage

LiteJoin uses SQLite as its primary data store, with optional tiered storage for long-term retention.

## SQLite Sharding

Data is distributed across multiple SQLite databases using FNV hashing on the message key:

```yaml theme={null}
storage:
  shard_count: 8       # Number of SQLite shards
  data_dir: "./data"   # Directory for shard files
  reader_pool_size: 4  # Reader connections per shard
```

Each shard is a separate `.db` file in WAL mode with a single writer connection and a configurable pool of reader connections.

| Setting            | Default  | Description                                                               |
| ------------------ | -------- | ------------------------------------------------------------------------- |
| `shard_count`      | `8`      | Number of SQLite database shards. More shards = better write parallelism. |
| `data_dir`         | `./data` | Directory for all data files.                                             |
| `reader_pool_size` | `4`      | Read connections per shard for concurrent queries.                        |

## Retention

Data older than the retention duration is periodically deleted:

```yaml theme={null}
retention:
  duration: 24h        # Keep data for 24 hours
  clean_interval: 1m   # Check every minute
```

<Warning>
  When tiered storage is disabled, deleted data is **permanently lost**. Set retention based on your downstream query needs.
</Warning>

## Tiered Storage (Optional)

When enabled, LiteJoin compacts expired data into **Parquet** files before deleting from SQLite. These files are queryable via an embedded **DuckDB** instance and can optionally be uploaded to cloud storage.

### How It Works

```
SQLite (hot) → Compactor → Parquet (warm) → Uploader → Cloud Storage (cold)
```

1. **Retention fires** — rows older than the TTL are eligible for compaction.
2. **Compactor** reads rows from SQLite, writes them to Parquet files with Snappy compression.
3. **Rows are deleted** from SQLite, reclaiming space.
4. **DuckDB** queries Parquet files for historical data.
5. **Uploader** (optional) copies Parquet files to S3/GCS/Azure Blob Storage.

### Configuration

```yaml theme={null}
storage:
  archive:
    enabled: true
    compaction_interval: 1m
    target_file_size: 128MB
    compression: snappy          # snappy | zstd | none
    duckdb_memory_limit: 256MB
    local_retention: 168h        # Keep local Parquet for 7 days

    cloud:
      enabled: false
      provider: s3               # s3 | gcs | azure
      bucket: my-litejoin-archive
      prefix: litejoin/
      region: us-east-1
      upload_concurrency: 4
```

### Archive Config Reference

| Field                 | Type     | Default  | Description                             |
| --------------------- | -------- | -------- | --------------------------------------- |
| `enabled`             | bool     | `false`  | Enable tiered storage.                  |
| `compaction_interval` | duration | `1m`     | How often compaction runs.              |
| `target_file_size`    | string   | `128MB`  | Target Parquet file size.               |
| `compression`         | string   | `snappy` | Parquet compression codec.              |
| `duckdb_memory_limit` | string   | `256MB`  | Max memory for DuckDB queries.          |
| `duckdb_threads`      | int      | `0`      | DuckDB threads. `0` = match GOMAXPROCS. |
| `local_retention`     | duration | `168h`   | How long to keep local Parquet files.   |

### Cloud Config Reference

| Field                      | Type     | Default | Description               |
| -------------------------- | -------- | ------- | ------------------------- |
| `cloud.enabled`            | bool     | `false` | Enable cloud upload.      |
| `cloud.provider`           | string   | —       | `s3`, `gcs`, or `azure`.  |
| `cloud.bucket`             | string   | —       | Bucket name.              |
| `cloud.prefix`             | string   | —       | Key prefix within bucket. |
| `cloud.region`             | string   | —       | Cloud region.             |
| `cloud.upload_concurrency` | int      | `4`     | Parallel upload workers.  |
| `cloud.upload_timeout`     | duration | `5m`    | Per-file upload timeout.  |

### Data Lifecycle Example

Given `retention: 1h` and `archive.local_retention: 168h`:

| Time     | Tier            | State                                       |
| -------- | --------------- | ------------------------------------------- |
| t=0      | SQLite          | Written, available for real-time joins      |
| t=1h     | Parquet (local) | Compacted from SQLite, queryable via DuckDB |
| t=1h+30s | Parquet + Cloud | Uploaded to S3 (if enabled)                 |
| t=7d     | Cloud only      | Local Parquet evicted                       |
| t=∞      | Cloud           | Retained indefinitely                       |

### Querying Historical Data

Historical data is queryable via the [Snapshot API](/pipelines/snapshot-api). When a `from` parameter extends beyond the retention window, the snapshot handler automatically queries Parquet files via DuckDB.

<Note>
  The hot path (real-time joins) has **zero overhead** from tiered storage. DuckDB is only used for historical queries.
</Note>