Say I have a big LazyArray
with size 10^5
and the underlying structure is that every 5000
entries are in a “chunk” and one has to read and decompress a chunk together.
I have implemented getindex()
for LazyArray
and it caches the “chunk” that contains my entry such that when you iterate over it, it only does slow I/O once per chunk. In fact this works so well I can even wrap multiple array into a dataframe-like lazy object with TypedTables.Table
and have super fast type-stable loop.
But I now run into problem with @threads for ... in LazyTable
, it tries to materialize the table ahead of the for loop (by design of threads macro I guess?). My question is 1) is there a way to support @threads
? if not, how can I error and tell user to use @spawn
inside a normal for loop