I’m trying to load a multifile dataset using NCDatasets, and to speed things up I am trying to use dagger in this way
include("test/test_catarrays.jl") # this creates some small test files mfds = Dataset(fnames); X = Distribute(Blocks(2,3,1), variable(mfds,"var")) collect(sum(X,dims = 3)) # same results as sum(variable(mfds,"var")[:,:,:],dims = 3)
This is something experimental in NCDatasets and I saw it in here Now, I’ve tried to use it to load my own variables but I don’t really get the meaning of
Blocks and how should I select that. So far I have only discovered that if I pass
Blocks a different number of arguments than the dimensions in the array to be read from the file, it throws an error. Other than that I don’t seem to notice how to make it work, sometimes it just stays there forever without completing the task.
Does anyone have any experience using Dagger that can throw a light on this?