Blog · Search
Searching for a value across millions of Kafka messages: how long it really takes
"Can you find the order for customer RSSMRA80A01H501U?" In a database that is a query on an index. In Apache Kafka it is not: a topic is a log, and the broker does not know what is inside the messages. To find a value inside a JSON, someone has to read the messages and look inside them.
How long does it take? Less than you fear, if you do the maths right.
Bytes matter, not messages
The temptation is to think in number of messages. But the real work is downloading the bytes from the broker to your PC. Checking the text costs little for a well-written program.
A simple model holds up surprisingly well:
time ≈ the longer of (bytes to download ÷ bandwidth) and (messages ÷ messages per second the program can check)
With messages of a few hundred bytes, bandwidth almost always wins. A few orders of magnitude, for uncompressed topics and 500-byte messages:
| Messages | Data | Network at about 35 MB/s | 1 Gbit network |
|---|---|---|---|
| 1 million | 0.5 GB | about 14 s | about 4 s |
| 10 million | 5 GB | about 2 min | about 43 s |
| 50 million | 25 GB | about 11 min | about 4 min |
These are estimates, not promises: network, brokers and disks change everything. But they give the right order of magnitude and, above all, they can be computed before you start, from the number of messages (end minus start of each partition) and a small sample for the average size.
A real measurement for comparison: 12 million JSON messages of about 500 bytes, 5.8 GB uncompressed, read end to end in 2 minutes 2 seconds on a local Redpanda in Docker, with the search engine never going above 30 MB of RAM.
Compression helps
If the topic is compressed with lz4 or zstd, fewer bytes cross the network and the time is often halved. Decompressing costs little compared with the network. If you often search the same topics, ask whoever produces them whether compression is on: it is a producer setting, compression.type.
Three ways to read less
1. By key: one partition only
If the value you are looking for is the message key, you do not need to read the whole topic. The key decides the partition: compute it again and read only that one. On a 12-partition topic, about 12 times less data.
Watch out for the detail that trips people up: the partitioner is not the same everywhere. The Java producer uses murmur2 on the key bytes. librdkafka, by default, uses CRC32 (consistent_random). Compute it with the wrong one and you search the wrong partition and find nothing. A good tool tries both, or lets you choose.
And if the producer assigned partitions by hand, the key tells you nothing: then you need to search the whole topic.
2. By time range: from time to offsets
If you know the order is from yesterday, you do not need to read a month. Kafka can translate an instant into an offset: the ListOffsets request with a timestamp (offsetsForTimes in the Java client) returns, for each partition, the first offset with a timestamp equal to or later than it. Do it for the start and the end of the range, and read only what lies in between.
3. Stop early
If you only need to know the value is there, stop at the first match. If you want the first hundred, stop at a hundred. It sounds obvious, but many tools read everything anyway.
What to expect from a tool
- An estimate before it starts, so you can decide whether to narrow the time range.
- Time left while it runs, corrected with the real speed.
- A count of all matches, even if it keeps only the first thousand in memory.
- Flat memory: reading 10 million messages must not mean keeping them all in RAM.
- A speed limit, when you search a production cluster and do not want to load the brokers.