Hey everyone! I have a function makebatch!(::IOStream, kwargs...) that creates one batch of data to feed to a model (from a file). For this the function takes an IOStream and reads as many lines as it needs to fill the batch (e.g. after every readline from the IOStream it checks if it has to read another line, so the number of lines read per batch is arbitrary).
To train the model I want to do multiple epochs of this procedure (it may be better to store the data after the first read, but it’s pretty big and I want to tackle this problem later on). Ideally I would want something like this:
for epoch in 1:10
io = open("data.txt")
while isopen(io)
batch = makebatch!(io)
train!(batch)
end
end
Is there a correct condition I can use here (because while isopen(io) does not work. Basically I want to tell my code to keep making batches and train the model with them until the IOStream is exhausted, but I could not find a function to track the state of the IOStream since it does not seem to get closed automatically when it reached its end.