Skip to content

[connectors] Lateness-aware Kafka ingest #5539

@ryzhyk

Description

@ryzhyk

It is currently impossible to guarantee lateness when reading from a multi-partition Kafka topic, since ingest from different partitions can be out of sync by unbounded time range. One solution is to artificially delay reading from a partition if it gets too far ahead of other partitions.

No concrete design yet. One complication is "idle partitions" that stop producing outputs for a long time (or forever).

Related to: #4854

Metadata

Metadata

Assignees

Labels

connectorsIssues related to the adapters/connectors crateuser-reportedReported by a user or customer

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions