Hi, I would like to read a portion of a lot of big files, as I know that the lines that interest me are located within line x and line y Currently, I load the files with CSV.File and read parts, but the loading overhead is way too much. The same process in shell (with sed/grep…) takes 27 minutes,…

Did you try these? skipto limit

Unfortunately, as far as I’ve tested it, skipto/limit has poor performance: @time csv=CSV.File("data/big_file.txt"; header=false, skipto=1501245, limit=450) 5.636217 seconds (137.98 M allocations: 2.146 GiB, 0.36% gc time, 0.08% compilation time) 450-element CSV.File: ... In shell: time sed -n …

You could try using eachline, Iterators.drop and Iterators.take to assemble the lines you need into an in-memory stream, and then read the data from there. Depending on the amount of data you need to scan, you can either join the lines in question into a single string in memory, or use an IOBuffer a…

Looks like very complicated for such a simple operation. For various reasons, I can’t use SQL… I’m currently experimenting with: block=readchomp(`sed -n "$start,$end p;$end_plus_one q" $filename`) It’s fast but it doesn’t return an array.

Only ever used Windows so take all of this with a grain of salt. Looking it up, sed is a stream editor working on one line at a time, wc -l counts lines, and piped processes execute concurrently (maybe like broadcasting fusing into 1 loop), so my guess is that shell command increments a count for ea…

If the file contains a big dataset, usually the content of the lines has data identifying the data. In this case, random access and binary search to find the initial position to read and stop condition could blow the other methods out of the water. For example, if each line contains a sequential ID…

[image] Ju_ska: For various reasons, I can’t use SQL… Well, that’s a bit vague. I understand if you can’t create intermediate SQL files, but it’s odd if you can’t use SQL at all… DuckDB, for instance, supports querying directly from a CSV file. I don’t know the performance of this operation,…

I’d like to keep things simple for something as simple as this… For now I crafted a run command with sed inside, and write the extract in a ram folder, then I read the extract as a csv. I now run under 18 minutes \o/ .

Can you provide a few lines from the big_file to have as a sample, and what is the size in MB and in lines of the file? (for each problem there are specific optimizations)

Reading a file from line x to line y

General Usage

rafael.guerra May 17, 2024, 10:58am 6

Please check also this thread.

Topic		Replies	Views
Skipping a lot of lines in CSV.read() allocates too much memory Performance csv , io	77	2603	February 23, 2024
Reading a few rows from a BIG CSV file General Usage dataframes , csv , big-data	39	4918	January 18, 2024
How can I split large data using a faster and more efficient function (data science)? New to Julia csv	9	887	October 27, 2022
Importing only specific lines from large csv file Data csv	6	1089	January 11, 2024
Read text file containing proper CSV data chunks Data dataframes , csv	11	644	December 20, 2023

Reading a file from line x to line y

Related topics