# How to read big data chunk by chunk(column-wise chunking)?
**URL:** https://discourse.julialang.org/t/how-to-read-big-data-chunk-by-chunk-column-wise-chunking/25084
**Category:** General Usage
**Tags:** question
**Created:** [June 8, 2019, 10:04pm UTC](https://discourse.julialang.org/t/how-to-read-big-data-chunk-by-chunk-column-wise-chunking/25084 "2019-06-08T22:04:14Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Carol](https://discourse.julialang.org/u/Carol)
#### Post date: [June 8, 2019, 10:04pm UTC](https://discourse.julialang.org/t/how-to-read-big-data-chunk-by-chunk-column-wise-chunking/25084/1 "2019-06-08T22:04:14Z")
Hi, could you please tell me how to read data chunk by chunk?
For example, read 100 columns at one time, like:
```julia
sub_data=read_data(file, start_column=1,end_column=100)
```
After n-iteration, I can process all of my 100n columns and get all partial result.
The main problem is that I cannot find which package I can use for column-wise chunking. Can someone give me some directions?
* * *
Background: I’m using MPI to achieve parallel computing, and I want each process to read part of the matrix.(Columnwise block-striped matrix).
For example, if I have a 50k by 50k array, and I have 2 processes in MPI, then each process should read a 50k by 25k array.
Thanks
---
### Author:  [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)
#### Post date: [June 8, 2019, 11:03pm UTC](https://discourse.julialang.org/t/how-to-read-big-data-chunk-by-chunk-column-wise-chunking/25084/2 "2019-06-08T23:03:42Z")
Please read the following informative post and then update your post accordingly:
> [@Please read: make it easier to help you](https://discourse.julialang.org/t/psa-make-it-easier-to-help-you/14757):
>
> Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner and experienced. This public service announcement (PSA) outlines best practices when asking for help. Following these points makes it easier for us to help you and more likely you’ll get a prompt, useful answer. Keywords are highlighted to make it easier to refer to specific points. Choose a descriptive title that captures the key part of your question, eg “plots with multiple axes” instead of …
---
### Author:  [@quinnj](https://discourse.julialang.org/u/quinnj)
#### Post date: [June 9, 2019, 12:19am UTC](https://discourse.julialang.org/t/how-to-read-big-data-chunk-by-chunk-column-wise-chunking/25084/3 "2019-06-09T00:19:59Z")
Yes, it’d be helpful if you provided some more details here: what kind of format is your data in? csv? feather? excel? something else? Why is processing \> 100 columns at a time too big? I ask because on my 5-year-old laptop, I can process certain csv files with 20,000 columns without much trouble.
In the CSV.jl package, a recent addition is the [`CSV.Rows`](https://github.com/JuliaData/CSV.jl/blob/master/src/rows.jl#L22) type, which allows efficient iteration, row-by-row, over the values in a csv file. It even allows a `reusebuffer=true` keyword argument that will allocate a single buffer for the entire file to be re-used while iterating. So you could process an entire file by doing something like:
```julia
for row in CSV.Rows(filename; reusebuffer=true)
# do things with row values: row.col1, row.col2, etc. where `col1` is a column name in the csv file
end
```
Hope that helps?
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [June 9, 2019, 12:38am UTC](https://discourse.julialang.org/t/how-to-read-big-data-chunk-by-chunk-column-wise-chunking/25084/4 "2019-06-09T00:38:52Z")
The package JuliaDB is supposed to be useful to perform some operations with large datasets.
[https://github.com/JuliaComputing/JuliaDB.jl](https://github.com/JuliaComputing/JuliaDB.jl)
---
### Author:  [@Carol](https://discourse.julialang.org/u/Carol)
#### Post date: [June 9, 2019, 1:20am UTC](https://discourse.julialang.org/t/how-to-read-big-data-chunk-by-chunk-column-wise-chunking/25084/5 "2019-06-09T01:20:04Z")
Thanks for your reply. The data have both .txt file and .jld file.
I’m using MPI to achieve parallel computing, and I want each process to read part of the matrix (Columnwise block-striped matrix).
For example, if I have a 50k by 50k array, and I have 2 process in MPI, then each process should read a 50k by 25k array.
Do we have method to achieve that?
Thanks
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [June 9, 2019, 4:53pm UTC](https://discourse.julialang.org/t/how-to-read-big-data-chunk-by-chunk-column-wise-chunking/25084/6 "2019-06-09T16:53:43Z")
> [@Carol](#):
>
> For example, if I have a 50k by 50k array, and I have 2 process in MPI, then each process should read a 50k by 25k array.
> Do we have method to achieve that?
HDF5 is [designed for precisely](https://www.hdfgroup.org/2015/08/parallel-io-with-hdf5/) this sort of thing.