Hi, could you please tell me how to read data chunk by chunk? For example, read 100 columns at one time, like: sub_data=read_data(file, start_column=1,end_column=100) After n-iteration, I can process all of my 100n columns and get all partial result. The main problem is that I cannot find which …

[image] Carol: For example, if I have a 50k by 50k array, and I have 2 process in MPI, then each process should read a 50k by 25k array. Do we have method to achieve that? HDF5 is designed for precisely this sort of thing.

Please read the following informative post and then update your post accordingly: [image] Please read: make it easier to help you Meta Discussion Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner and experienced. This public…

Yes, it’d be helpful if you provided some more details here: what kind of format is your data in? csv? feather? excel? something else? Why is processing > 100 columns at a time too big? I ask because on my 5-year-old laptop, I can process certain csv files with 20,000 columns without much trouble. I…

The package JuliaDB is supposed to be useful to perform some operations with large datasets. https://github.com/JuliaComputing/JuliaDB.jl

Thanks for your reply. The data have both .txt file and .jld file. I’m using MPI to achieve parallel computing, and I want each process to read part of the matrix (Columnwise block-striped matrix). For example, if I have a 50k by 50k array, and I have 2 process in MPI, then each process should rea…

How to read big data chunk by chunk(column-wise chunking)?

General Usage

jpsamaroo June 8, 2019, 11:03pm 2

Please read the following informative post and then update your post accordingly:

1 Like

Topic		Replies	Views
How can I split large data using a faster and more efficient function (data science)? New to Julia csv	9	887	October 27, 2022
Need example of processing csv file in chunks New to Julia package , csv	3	1253	January 4, 2021
What's the best way to work with millions of rows of data? Performance	7	2182	February 24, 2020
Skipping a lot of lines in CSV.read() allocates too much memory Performance csv , io	77	2603	February 23, 2024
Partition a large CSV file into smaller files without loading into memory General Usage question , csv , io	6	3738	March 10, 2019

How to read big data chunk by chunk(column-wise chunking)?

Related topics