This looks like a version of Java library, so I used Kafka server 2.11-0.9.0.1. However, I wasn’t able to reproduce your error even after 20 minutes of reading from local Kafka. Could you please completely reset the package code on your machine? On Linux, you can do it like this:
cd ~/.julia/v0.5/Kafka
git reset --hard HEAD
git pull origin master
Also, note that the code you have posted is very inefficient:
for each in messages
...
offset = produce(kc, "my_second_topic", 0, messages2)
end
You read the whole batch of messages in a single request to Kafka, but make a produce request for each of these messages. In my test, I’ve got 25k messages in a batch, so I had to do 25k produce requests. Kafka can handle it, but a much more performant option would be to do something like this:
messages = take!(fetch(kc, "my_topic", 0, offset))
new_messages = process_consumed_messages(messages)
take!(produce(kc, "my_second_topic", 0, new_messages))
In this case each fetch request will be followed by only one produce request.