Hard20 minDistributed Systems
UpdatedAug 6, 2026
Edit

Kafka: Choosing a Partition Key

Question Variations

  • "Why use `accountId` rather than `eventId` as a Kafka key?"
  • "How would you detect a hot Kafka partition?"
  • "Can adding partitions fix a single hot account?"

Why This Is Asked

An event stream has strict ordering requirements per account, but a small number of accounts generate most traffic. This tests whether a candidate can choose a key that preserves the needed order while detecting and mitigating hot partitions.

Key Concepts

  • Ordering boundary: Records sharing a key map to the same partition and retain order there.
  • Key selection: Choose the aggregate whose state transition requires sequence, not an arbitrary or random identifier.
  • Skew: Hot keys cannot be parallelized within one ordered partition and must be handled by domain-level strategies.
  • Evolution: Increasing partitions changes key-to-partition mapping and must not be mistaken for a repair of existing ordering.

Question Variations

  • “Why use accountId rather than eventId as a Kafka key?”
  • “How would you detect a hot Kafka partition?”
  • “Can adding partitions fix a single hot account?”

Answers by Technology

+ Add Variant
Apache KafkaImprove this answer ✏️

Expected Answer

Use accountId as the key when account state transitions must be ordered. Kafka maps equal keys to the same partition, which supplies sequence for that account while allowing other accounts to process in parallel. An event ID is a poor key because related events scatter across partitions. Monitor records, bytes, and processing lag per partition to detect skew. A single hot account cannot be split across partitions without relaxing its ordering requirement; address it with domain changes such as serializing that account, batching, or a redesigned sub-key with an explicit merge strategy. Adding partitions improves average capacity but remaps future keys and does not parallelize one existing ordered key.

Why It Matters

The key encodes an ordering contract. A random key gives throughput but can apply cancellation before creation; a hot key can cap the entire consumer group’s throughput.

Example Code

await producer.send({ topic: "account-events", messages: [{ key: event.accountId, value: JSON.stringify(event) }] });

Common Mistakes

  • Keying by event ID: Related state changes lose partition order.
  • Assuming more partitions split one hot account: One key still maps to one partition.

Follow-up Questions

  • What ordering does Kafka provide? (Answer: Total order within one partition, not across a topic.)
  • How is skew observed? (Answer: Compare per-partition traffic, consumer lag, and processing latency.)

Related Questions

References