Kafka Source

The kafka source consumes messages from one or more Kafka topics using a consumer group.

Configuration

sources:
  - type: kafka
    name: kafka-main
    topics:
      - orders
      - users
    config:
      brokers: "localhost:9092"
      group_id: "litejoin-consumer"

Field	Type	Required	Description
`brokers`	string	yes	Comma-separated list of Kafka broker addresses.
`group_id`	string	yes	Consumer group ID.
`topics`	string[]	yes	List of Kafka topics to consume from.

Message Mapping

Kafka messages are mapped to LiteJoin messages as follows:

Kafka Field	LiteJoin Field
Topic	`topic`
Key	`key`
Value	`payload`
Timestamp	`timestamp`

Example

Consume orders and users from Kafka, join them, and forward to a webhook:

sources:
  - type: kafka
    name: kafka-main
    topics:
      - orders
      - users
    config:
      brokers: "broker1:9092,broker2:9092"
      group_id: "litejoin-prod"

joins:
  - name: order-user-join
    query: |
      SELECT o.key, o.payload, u.payload
      FROM orders o
      INNER JOIN users u ON json_extract(o.payload, '$.user_id') = u.key
      WHERE o.timestamp > (strftime('%s', 'now') - 3600)
    sink: webhook-out

sinks:
  - type: http
    name: webhook-out
    config:
      url: "http://localhost:9000/webhook"

When to Use

Use the Kafka source when:

You already have a Kafka cluster producing events
You want LiteJoin to enrich or join Kafka streams with other data sources
You need consumer group semantics (offset management, partition assignment)

For systems without Kafka, use the API Source or HTTP Source.