Hi, could you please tell me how to read data chunk by chunk? For example, read 100 columns at one time, like: sub_data=read_data(file, start_column=1,end_column=100) After n-iteration, I can process all of my 100n columns and get all partial result. The main problem is that I cannot find which …

[image] Carol: For example, if I have a 50k by 50k array, and I have 2 process in MPI, then each process should read a 50k by 25k array. Do we have method to achieve that? HDF5 is designed for precisely this sort of thing.

How to read big data chunk by chunk(column-wise chunking)?

General Usage

jpsamaroo June 8, 2019, 11:03pm 2

Please read the following informative post and then update your post accordingly:

1 Like

Topic		Replies	Views
Need example of processing csv file in chunks New to Julia package , csv	3	1235	January 4, 2021
How can I split large data using a faster and more efficient function (data science)? New to Julia csv	9	851	October 27, 2022
What's the best way to work with millions of rows of data? Performance	7	2126	February 24, 2020
Partition a large CSV file into smaller files without loading into memory General Usage question , csv , io	6	3721	March 10, 2019
Reading and processing Data files concurrently Data parallel	18	3859	September 20, 2017

How to read big data chunk by chunk(column-wise chunking)?

Related topics