Hello All,
I am working on large machine learning project, where data are stored in S3. My plan is to asynchronously download data to workers that would learn the Neural Network or do other magic on them.
So far, I have encountered two limitations.
Firstly,
AWS.S3.get_bkt(env,bucket_name,options=AWS.S3.GetBucketOptions(max_keys=2000))
returns at most 1000 elements of the bucket, even though I requested more.
Secondly, when I download the file stored as jld, I cannot parse it. After downloading the file issuing
r = AWS.S3.get_object(env,bkt,file_name);
r.obj is of type Array{UInt8,1} but I cannot load it using load(IOBuffer(r.obj)).
Has anyone tried to overcome these issues?
Thanks for suggestions.
Tomas