I thought that this would end up in several some buffer-size chunks, but actually the 9GB file seems to be downloaded at once:
data_url = "https://zenodo.org/records/11549846/files/U2018_CLC2018_V2020_20u1.gpkg?download=1"
chunk_counter = 1
HTTP.open("GET", data_url) do io # Note the SSL support
while !eof(io)
global chunk_counter
println(chunk_counter)
data = String(read(io))
chunk_counter += 1
end
end
In this case it’s a binary data, but is there a way to stream a remote resource in chunks that are guaranteed to ends with a newline, so that I can process them with some online algorithm (i.e. train a ML model that supports multiple fitting ) ?