Hi, could you please tell me how to read data chunk by chunk? For example, read 100 columns at one time, like: sub_data=read_data(file, start_column=1,end_column=100) After n-iteration, I can process all of my 100n columns and get all partial result. The main problem is that I cannot find which …

[image] Carol: For example, if I have a 50k by 50k array, and I have 2 process in MPI, then each process should read a 50k by 25k array. Do we have method to achieve that? HDF5 is designed for precisely this sort of thing.

How to read big data chunk by chunk(column-wise chunking)?

General Usage

jpsamaroo June 8, 2019, 11:03pm 2

Please read the following informative post and then update your post accordingly:

1 Like