Hi, could you please tell me how to read data chunk by chunk? For example, read 100 columns at one time, like: sub_data=read_data(file, start_column=1,end_column=100) After n-iteration, I can process all of my 100n columns and get all partial result. The main problem is that I cannot find which …

[image] Carol: For example, if I have a 50k by 50k array, and I have 2 process in MPI, then each process should read a 50k by 25k array. Do we have method to achieve that? HDF5 is designed for precisely this sort of thing.

How to read big data chunk by chunk(column-wise chunking)?

General Usage

stevengj June 9, 2019, 4:53pm 6

HDF5 is designed for precisely this sort of thing.

4 Likes

Topic		Replies	Views
How can I split large data using a faster and more efficient function (data science)? New to Julia csv	9	876	October 27, 2022
Need example of processing csv file in chunks New to Julia package , csv	3	1248	January 4, 2021
What's the best way to work with millions of rows of data? Performance	7	2161	February 24, 2020
Skipping a lot of lines in CSV.read() allocates too much memory Performance csv , io	77	2498	February 23, 2024
Partition a large CSV file into smaller files without loading into memory General Usage question , csv , io	6	3731	March 10, 2019

How to read big data chunk by chunk(column-wise chunking)?

Related topics