Hi, could you please tell me how to read data chunk by chunk? For example, read 100 columns at one time, like: sub_data=read_data(file, start_column=1,end_column=100) After n-iteration, I can process all of my 100n columns and get all partial result. The main problem is that I cannot find which …

[image] Carol: For example, if I have a 50k by 50k array, and I have 2 process in MPI, then each process should read a 50k by 25k array. Do we have method to achieve that? HDF5 is designed for precisely this sort of thing.

How to read big data chunk by chunk(column-wise chunking)?

General Usage

stevengj June 9, 2019, 4:53pm 6

HDF5 is designed for precisely this sort of thing.

4 Likes

Topic		Replies	Views
Need example of processing csv file in chunks New to Julia package , csv	3	1177	January 4, 2021
Importing only specific lines from large csv file Data csv	6	897	January 11, 2024
Reading (big) ascii files Data	11	2696	April 5, 2019
Reading Multiple Files in a Folder Into a tensor in Julia Data question	8	2431	May 24, 2021
How can I split large data using a faster and more efficient function (data science)? New to Julia csv	9	782	October 27, 2022

How to read big data chunk by chunk(column-wise chunking)?

Related topics