Kafka consumer has some time offset

Description

I’ve set up Kafka and Memgraph under the same docker environment. Have only one open topic. Currently, when I start my producer, messages are immediately consumed by the command-line consumer, while the Memgraph consumer has some unexpected time offset till messages are consumed. Also, need to state that the Memgraph instance is using an internal Kafka stream while the command line is using an external stream.

Expected behavior

Consuming messages at the same time as the command-line Kafka tool.

Your environment

  • Memgraph 1.6.1 (Docker)
  • Ubuntu 20.04.2 LTS

How long is the delay? Is it a couple of seconds or just slightly?

Memgraph has a BATCH_INTERVAL that defines how long should Memgraph wait for a batch to fill up to BATCH_SIZE. (Streams | Memgraph Docs)
Maybe that can help?

Wasn’t aware of that interval, thank you, and will try configuring it! Also, could be useful to state the time unit in the documentation.

There is a time unit in the docs, it’s miliseconds :slight_smile:

CREATE STREAM <stream name>
  TOPICS <topic1> [, <topic2>, ...]
  TRANSFORM <transform procedure>
  [CONSUMER_GROUP <consumer group name>]
  [BATCH_INTERVAL <milliseconds>]
  [BATCH_SIZE <size>];
1 Like

Was looking at the description, sorry haha

  • BATCH_INTERVAL denotes the maximum wait time interval for consuming message(s) before calling the transformation procedure with the already received message(s). This value must be greater than zero and is defaulted to 100.
1 Like

I’ll make a PR for that whole page. I dislike it greatly.

Documentation has been improved by fitting query options in a table view.