Hello everyone, Consider the following example: using CSV, DataFrames number_of_lines = 10^4 CSV.write("data.csv", DataFrame(rand(number_of_lines,10), :auto)) steps = 20 chunk = round(Int,number_of_lines/steps) for i in 1:steps @time a=CSV.read("data.csv",DataFrame,skipto=i*chunk ,limit=chu…

Here is a summary of some of the solutions presented so far in this thread. Most refer to the lammps “dump file” format, so here is a code that generates an identical file, for reproducibility. ▶ Creating file Here are some ways to read the data from the file. ▶ CSV skiplines loop ▶ Numpy.loadtxt ▶ r…

It’s not clear precisely what the OP wants, but hopefully my MWE demonstrates the general approach that they will need to take.

The best approach will be some combination of my MWE with the suggestion that @SteffenPL gave.

You can specify the schema you’re interested in and skip the errors. Or there are methods to retain errors as well. DuckDB.execute(db, "SELECT * FROM read_csv('~/LAMMPS.txt', header = false, columns = {'id': 'UINTEGER', 'type': 'UINTEGER', 'xs': 'DOUBLE', 'ys': 'DOUBLE', 'zs': 'DOUBLE'}, ignore_…

[image] aris: As for the general, well-behaved case in the MRE above (for which the topic was created), I’m not sure whether we got a answer or not. I don’t have formal proof of what was hypothesized in the previous message, but the following could be a strong clue ▶ Chunk affects the file si…

Thank you for the suggestion @CameronBieganek , that creates a very tidy file indeed, but I’m not sure whether this is relevant here. First of all, it discards the BOX BOUNDS section, which I would like to keep/utilize **, and also it writes everything into a new file, which I would have to import …

Update: I reckon this is the fastest solution I found so far; using CSV, DataFrames number_of_lines = 10^6 CSV.write("data.csv", DataFrame(rand(number_of_lines, 10), :auto)) steps = 10 chunk = round(Int, number_of_lines / steps) # Preallocate arrays raw_strings::Vector{String} = [" " for _ in 1…

Just as a note, the raw_strings are not actually preallocating anything, since the line raw_strings[i] = readline(io) will just replace the ith entry of the vector with the new string from readline(io). Out of curiosity, is this similar to the speed of the numpy approach? Or is it just the fastest …

Numpy.loadtxt suffers from the same drawback as CSV.read; That is, for really big files, the first few iterations are lightning fast, but when you dig deeper into the file, and skip a lot of lines, you have to wait more and more (for some pointer to reach the desired line, I suppose?). That being …

[image] aris: numy is A LOT faster than csv Does readline compare to numpy? PS have you thought about using readuntil instead of readline. If you have a block delimiter in your file, you could easily read many lines at a time.

Skipping a lot of lines in CSV.read() allocates too much memory

General Usage Performance

rocco_sprmnt21 February 13, 2024, 9:41pm 33

Here how to use arrow format instead of csv for better performance is discussed

2 Likes

Topic		Replies	Views
Reading a few rows from a BIG CSV file General Usage dataframes , csv , big-data	39	4904	January 18, 2024
CSV Reading (rewrite in C?) Internals & Design	50	5374	October 1, 2018
Reading a file from line x to line y General Usage csv	28	929	May 22, 2024
CSV vs DelimitedFiles vs Numpy Performance	15	1116	January 20, 2024
How can I split large data using a faster and more efficient function (data science)? New to Julia csv	9	885	October 27, 2022

Skipping a lot of lines in CSV.read() allocates too much memory

Related topics