# Shuffle lines in a big file, and trim it
**URL:** https://discourse.julialang.org/t/shuffle-lines-in-a-big-file-and-trim-it/114247
**Category:** General Usage
**Tags:** io
**Created:** [May 14, 2024, 12:55pm UTC](https://discourse.julialang.org/t/shuffle-lines-in-a-big-file-and-trim-it/114247 "2024-05-14T12:55:12Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Ju\_ska](https://discourse.julialang.org/u/Ju_ska)
#### Post date: [May 14, 2024, 12:55pm UTC](https://discourse.julialang.org/t/shuffle-lines-in-a-big-file-and-trim-it/114247/1 "2024-05-14T12:55:12Z")
Hi,
In shell, whatever the size of the file, I do:
```julia
cat file.txt | shuf | head -n 5000 > shuffled_trimmed_file.txt
```
With Julia, I tried to load the file with readdlm(), but it is too big and the process stops.
I looked at ways to randomly read lines from a file, but found nothing convenient/simple.
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [May 14, 2024, 9:50pm UTC](https://discourse.julialang.org/t/shuffle-lines-in-a-big-file-and-trim-it/114247/2 "2024-05-14T21:50:58Z")
A simple way, in case it helps:
```julia
# 1 - Create input file
using Random
open("in.txt", "w") do io
foreach(_ -> println(io, randstring(rand(1:9))), 1:1_000_000)
end
# 2 - Read n random lines
using StatsBase
Nlines = countlines("in.txt")
n = 5_000
ix = sort(sample(1:Nlines, n; replace=false))
str = Vector{String}(undef, n)
i = j = 1
for line in eachline("in.txt")
(i in ix) && begin str[j] = line; j += 1; end
(j==n+1) && break
i += 1
end
# 3 - shuffle the n lines and output to file
shuffle!(str)
open("out.txt", "w") do io
for i in eachindex(str)
println(io, str[i])
end
end
```
---
### Author:  [@Dan](https://discourse.julialang.org/u/Dan)
#### Post date: [May 14, 2024, 10:55pm UTC](https://discourse.julialang.org/t/shuffle-lines-in-a-big-file-and-trim-it/114247/3 "2024-05-14T22:55:08Z")
Another more low-level method:
```julia
using Mmap
function process_file2(in_fn, out_fn, n)
f = open(in_fn, "r")
fout = open(out_fn, "w")
mm = Mmap.mmap(f, Vector{UInt8})
L = length(mm)
i = 0
lines = Set{UInt}()
while i < n
ix = rand(1:L)
ix2 = ix
while ix2 < L && mm[ix2] != UInt8('\n')
ix2 += 1
end
ix2 += 1
ix2 < L || continue
ix2 in lines && continue
push!(lines, ix2)
while ix2 < L && mm[ix2] != UInt8('\n')
write(fout, mm[ix2])
ix2 += 1
end
write(fout, '\n')
i += 1
end
close(fout)
close(f)
end
n = 5_000
process_file2("infile.txt", "outfile.txt", n)
```
This tries to avoid reading the whole file, and thus is o(`Nlines`). In practice, it was 1000x faster than previous post.
A somewhat annoying cost, is non-uniformity if lines are of different lengths and subsampled number of lines can’t be too close to original number of lines.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [May 15, 2024, 12:48am UTC](https://discourse.julialang.org/t/shuffle-lines-in-a-big-file-and-trim-it/114247/5 "2024-05-15T00:48:54Z")
In addition to the suggestions here, note that `readdlm` is untyped and thus very slow and bad for large files—work with lines instead. I would generally just avoid `readdlm` actually, I regret it being a stdlib.
The simplest version of what you want would be this:
```julia
using Random
foreach(println, shuffle!(readlines())[1:5000])
```
This should be reasonably efficient and is actually shorter than the shell commands and does the equivalent work. If you run this from the command line it works like this:
```sh
julia -e 'using Random; foreach(println, shuffle!(readlines())[1:5000])' < file.txt > shuffled_trimmed_file.txt
```
Or if you want to open named files it gets a bit more verbose:
```julia
using Random
open("file.txt", read=true) do in
open("shuffled_trimmed_file.txt", write=true) do out
lines = shuffle!(readlines(in))
for i = 1:5000
println(out, line[i])
end
end
end
```
This can definitely be golfed to be shorter, but you get the point.
It would, however, be much more efficient to use reservoir sampling and only keep at most 5000 lines in memory at a time. There’s a very cool package called [StreamSampling](https://github.com/JuliaDynamics/StreamSampling.jl) that implements this for you:
```julia
using StreamSampling
lines = itsample(eachline("file.txt"), 5000)
```
That’s it and it’s _wildly_ efficient since it never needs to hold more than 5000 lines in memory.
---
### Author:  [@Ju\_ska](https://discourse.julialang.org/u/Ju_ska)
#### Post date: [May 15, 2024, 8:17am UTC](https://discourse.julialang.org/t/shuffle-lines-in-a-big-file-and-trim-it/114247/6 "2024-05-15T08:17:36Z")
Thanks everyone for you help !
---
### Author:  [@rdavis120](https://discourse.julialang.org/u/rdavis120)
#### Post date: [May 15, 2024, 9:46pm UTC](https://discourse.julialang.org/t/shuffle-lines-in-a-big-file-and-trim-it/114247/7 "2024-05-15T21:46:44Z")
Here is another solution if you need a solution to work across languages:
```julia
db = DuckDB.DB()
DuckDB.query(db, "COPY (SELECT * FROM 'input.csv' USING SAMPLE reservoir(5000 ROWS)) TO 'output.csv'")
```